Max level shown:
1 starting March 17, 2025 at 10:18:47 PM UTC
2 Creating 3 downstairs regions
3 Creating region directory at: /var/tmp/dsc/region
4 Creating dsc directory at: /tmp/dsc
5 Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.26930743
6 Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.020971946
7 Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.023120591
8 Created 3 regions
9 Starting 3 downstairs
10 Using existing output directory "/tmp/dsc"
11 Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 })
12 start ds: 8810
13 start ds: 8820
14 start ds: 8830
15 start access at:127.0.0.1:9998
16 Starting downstairs at port 8810
17 Starting downstairs at port 8820
18 Make output file at "/tmp/dsc/downstairs-8820.txt"
19 Make output file at "/tmp/dsc/downstairs-8810.txt"
20 Starting downstairs at port 8830
21 Make output file at "/tmp/dsc/downstairs-8830.txt"
22 [8810][0] initial start wait reports Starting
23 [8820][1] initial start wait reports Starting
24 [8830][2] initial start wait reports Starting
25 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1201)
26 [8830][2] initial start wait reports Running
27 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1202)
28 [8820][1] initial start wait reports Running
29 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1203)
30 [8810][0] initial start wait reports Running
31 All downstairs are running
32 Mar 17 22:18:48.289 INFO listening, local_addr: 127.0.0.1:9998
33 Control access at:127.0.0.1:9998
34 Running initial fill
35 Mar 17 22:18:53.388 INFO accepted connection, remote_addr: 127.0.0.1:54934, local_addr: 127.0.0.1:9998
36 Mar 17 22:18:53.391 INFO request completed, latency_us: 695, response_code: 200, uri: /regioninfo, method: GET, req_id: b9470b8b-ad83-47ba-934c-f89eb6ca6bee, remote_addr: 127.0.0.1:54934, local_addr: 127.0.0.1:9998
37 Mar 17 22:18:53.393 INFO request completed, latency_us: 120, response_code: 200, uri: /regioncount, method: GET, req_id: fe1586a1-2697-47f0-acb4-aa7d5dd56777, remote_addr: 127.0.0.1:54934, local_addr: 127.0.0.1:9998
382025-03-17T22:18:53.393ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
392025-03-17T22:18:53.395ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
40 Mar 17 22:18:53.395 INFO request completed, latency_us: 127, response_code: 200, uri: /port/cid/0, method: GET, req_id: 5c2bd33c-5aad-4a5f-9ef9-fbb30a6cd898, remote_addr: 127.0.0.1:54934, local_addr: 127.0.0.1:9998
41 Mar 17 22:18:53.395 INFO request completed, latency_us: 88, response_code: 200, uri: /port/cid/1, method: GET, req_id: d3dcb9ee-c22d-4196-9d5a-400ceeaa7407, remote_addr: 127.0.0.1:54934, local_addr: 127.0.0.1:9998
42 Mar 17 22:18:53.396 INFO request completed, latency_us: 87, response_code: 200, uri: /port/cid/2, method: GET, req_id: 5a4b7446-9c64-4a11-b828-f7253ca9d224, remote_addr: 127.0.0.1:54934, local_addr: 127.0.0.1:9998
432025-03-17T22:18:53.396ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
442025-03-17T22:18:53.398ZINFOcrucible: Upstairs starts
452025-03-17T22:18:53.398ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "acc6b11fa639390710fd9a0449e05c901b048ed2", git_commit_timestamp: "2025-03-17T22:02:02.000000000Z", git_branch: "renovate/itertools-0.x", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
462025-03-17T22:18:53.398ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
472025-03-17T22:18:53.398ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
482025-03-17T22:18:53.399ZINFOcrucible: Crucible efb80f3a-2eb0-42b4-8b64-66199dfd1fc5 has session id: a414dda4-ba6f-4553-95d1-378f576de1ad session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
492025-03-17T22:18:53.399ZINFOcrucible: Upstairs opts: Upstairs UUID: efb80f3a-2eb0-42b4-8b64-66199dfd1fc5, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
502025-03-17T22:18:53.399ZINFOcrucible: Crucible stats registered with UUID: efb80f3a-2eb0-42b4-8b64-66199dfd1fc5 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
512025-03-17T22:18:53.399ZWARNcrucible: could not find Downstairs address for Nexus session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
522025-03-17T22:18:53.401ZINFOcrucible: Downstairs in volume = 3 task = crutest
532025-03-17T22:18:53.401ZINFOcrucible: The guest has requested activation with gen:1
542025-03-17T22:18:53.401ZINFOcrucible: efb80f3a-2eb0-42b4-8b64-66199dfd1fc5 active request set session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
552025-03-17T22:18:53.401ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
562025-03-17T22:18:53.402ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
572025-03-17T22:18:53.402ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
582025-03-17T22:18:53.402ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
592025-03-17T22:18:53.402ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
602025-03-17T22:18:53.402ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
612025-03-17T22:18:53.402ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
622025-03-17T22:18:53.402ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
632025-03-17T22:18:53.402ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
642025-03-17T22:18:53.402ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:62383) = io task client = 2 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
652025-03-17T22:18:53.402ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:53354) = io task client = 1 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
662025-03-17T22:18:53.402ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:64586) = io task client = 0 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
672025-03-17T22:18:53.404ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
682025-03-17T22:18:53.404ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
692025-03-17T22:18:53.404ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
702025-03-17T22:18:53.405ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
712025-03-17T22:18:53.405ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
722025-03-17T22:18:53.405ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
732025-03-17T22:18:53.407ZINFOcrucible: Waiting for more clients to be ready session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
742025-03-17T22:18:53.407ZINFOcrucible: Waiting for more clients to be ready session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
752025-03-17T22:18:53.407ZINFOcrucible: [0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
762025-03-17T22:18:53.407ZINFOcrucible: [0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
772025-03-17T22:18:53.407ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
782025-03-17T22:18:53.407ZINFOcrucible: [1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
792025-03-17T22:18:53.407ZINFOcrucible: [1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
802025-03-17T22:18:53.407ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
812025-03-17T22:18:53.407ZINFOcrucible: [2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
822025-03-17T22:18:53.407ZINFOcrucible: [2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
832025-03-17T22:18:53.407ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
842025-03-17T22:18:53.407ZINFOcrucible: Max found gen is 1 = downstairs session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
852025-03-17T22:18:53.407ZINFOcrucible: Generation requested: 1 >= found:1 = downstairs session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
862025-03-17T22:18:53.407ZINFOcrucible: Next flush: 1 = downstairs session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
872025-03-17T22:18:53.407ZINFOcrucible: All extents match = downstairs session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
882025-03-17T22:18:53.407ZINFOcrucible: No downstairs reconciliation required session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
892025-03-17T22:18:53.407ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
902025-03-17T22:18:53.407ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
912025-03-17T22:18:53.407ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
922025-03-17T22:18:53.407ZINFOcrucible: All required reconciliation work is completed session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
932025-03-17T22:18:53.407ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
942025-03-17T22:18:53.407ZINFOcrucible: efb80f3a-2eb0-42b4-8b64-66199dfd1fc5 is now active with session: a414dda4-ba6f-4553-95d1-378f576de1ad session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
952025-03-17T22:18:53.407ZINFOcrucible: Set Active after no reconciliation session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
962025-03-17T22:18:53.407ZINFOcrucible: The guest has finished waiting for activation with:1
97 {"msg":"Activated sub_volume efb80f3a-2eb0-42b4-8b64-66199dfd1fc5","v":0,"name":"crucible","level":30Activate successful
98 ,"Wait for a query_work_queue command to finish before sending IO
99 time":"2025-03-17T22:18:53.407899895Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1207}
100 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
101 Fill test
102 Read and Verify all blocks (0..5000 range:false)
103 Wrote out file "/tmp/test_replay_verify.log"
104 CLIENT: Tests done. All submitted work has been ACK'd
105 ----------------------------------------------------------------
106 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:20
107 States: Active Active Active
108 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
109 1021 Acked Read 256 Done Done Done false
110 1022 Acked Read 256 Done Done Done false
111 1023 Acked Read 256 Done Done Done false
112 1024 Acked Read 256 Done Done Done false
113 1025 Acked Read 256 Done Done Done false
114 1026 Acked Read 256 Done Done Done false
115 1027 Acked Read 256 Done Done Done false
116 1028 Acked Read 256 Done Done Done false
117 1029 Acked Read 256 Done Done Done false
118 1030 Acked Read 256 Done Done Done false
119 1031 Acked Read 256 Done Done Done false
120 1032 Acked Read 256 Done Done Done false
121 1033 Acked Read 256 Done Done Done false
122 1034 Acked Read 256 Done Done Done false
123 1035 Acked Read 256 Done Done Done false
124 1036 Acked Read 256 Done Done Done false
125 1037 Acked Read 256 Done Done Done false
126 1038 Acked Read 136 Done Done Done false
127 1039 Acked Read 256 Done Done Done false
128 1040 Acked Read 256 Done Done Done false
129 STATES DS:0 DS:1 DS:2 TOTAL
130 Sent 0 0 0 0
131 Done 20 20 20 60
132 Skipped 0 0 0 0
133 Error 0 0 0 0
134 Last Flush: Some(JobId(1020)) Some(JobId(1020)) Some(JobId(1020))
135 Downstairs last five completed: 1020 1019 1018 1017 1016
136 Upstairs last five completed: 1040 1039 1038 1037 1036
137 CLIENT: Up:0 ds:20 act:3
138 ----------------------------------------------------------------
139 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0
140 Downstairs last five completed: 1041 1040 1039 1038 1037
141 Upstairs last five completed: 1041 1040 1039 1038 1037
142 CLIENT: Up:0 ds:0 act:3
143 CLIENT: All crucible jobs finished, exiting program
1442025-03-17T22:18:57.976ZWARNcrucible: Guest handle has been dropped
1452025-03-17T22:18:57.976ZWARNcrucible: client request queue closed unexpectedly; is the program exiting? = io task client = 0 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
1462025-03-17T22:18:57.977ZWARNcrucible: client task is sending Done(QueueClosed) = io task client = 0 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
1472025-03-17T22:18:57.977ZWARNcrucible: client task could not reply to main task; shutting down? = io task client = 0 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
1482025-03-17T22:18:57.977ZINFOcrucible: client task is exiting = io task client = 0 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
1492025-03-17T22:18:57.977ZINFOcrucible: client stopping due to Err(RecvError(())) = io task client = 2 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
1502025-03-17T22:18:57.977ZWARNcrucible: client task is sending Done(QueueClosed) = io task client = 2 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
1512025-03-17T22:18:57.977ZWARNcrucible: client task could not reply to main task; shutting down? = io task client = 2 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
1522025-03-17T22:18:57.977ZINFOcrucible: client task is exiting = io task client = 2 session_id = a414dda4-ba6f-4553-95d1-378f576de1ad
153 Replay loop starts now March 17, 2025 at 10:18:57 PM UTC
154 Mar 17 22:18:58.046 INFO accepted connection, remote_addr: 127.0.0.1:44577, local_addr: 127.0.0.1:9998
155 Mar 17 22:18:58.046 INFO request completed, latency_us: 170, response_code: 200, uri: /regioninfo, method: GET, req_id: 70f504c9-7f85-4724-b6e0-c3d15d4719a1, remote_addr: 127.0.0.1:44577, local_addr: 127.0.0.1:9998
1562025-03-17T22:18:58.047ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
157 Mar 17 22:18:58.047 INFO request completed, latency_us: 90, response_code: 200, uri: /regioncount, method: GET, req_id: 73bb775a-ecda-4640-8963-7f7e67df4a48, remote_addr: 127.0.0.1:44577, local_addr: 127.0.0.1:9998
1582025-03-17T22:18:58.047ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
159 Mar 17 22:18:58.048 INFO request completed, latency_us: 138, response_code: 200, uri: /port/cid/0, method: GET, req_id: 50c8d495-a0f4-4b43-a295-25d7f9df05de, remote_addr: 127.0.0.1:44577, local_addr: 127.0.0.1:9998
160 Mar 17 22:18:58.049 INFO request completed, latency_us: 113, response_code: 200, uri: /port/cid/1, method: GET, req_id: 97b33e79-6a6a-44a3-bce3-946618202d95, remote_addr: 127.0.0.1:44577, local_addr: 127.0.0.1:9998
161 Mar 17 22:18:58.049 INFO request completed, latency_us: 91, response_code: 200, uri: /port/cid/2, method: GET, req_id: 2cb96a5d-8c1a-4b3e-bf70-50c8623f2556, remote_addr: 127.0.0.1:44577, local_addr: 127.0.0.1:9998
1622025-03-17T22:18:58.049ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
1632025-03-17T22:18:58.049ZINFOcrucible: Upstairs starts
1642025-03-17T22:18:58.050ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "acc6b11fa639390710fd9a0449e05c901b048ed2", git_commit_timestamp: "2025-03-17T22:02:02.000000000Z", git_branch: "renovate/itertools-0.x", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
1652025-03-17T22:18:58.050ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
1662025-03-17T22:18:58.050ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
1672025-03-17T22:18:58.050ZINFOcrucible: Crucible ff114f14-d00f-420e-966b-bf2151b7c457 has session id: 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1682025-03-17T22:18:58.050ZINFOcrucible: Upstairs opts: Upstairs UUID: ff114f14-d00f-420e-966b-bf2151b7c457, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1692025-03-17T22:18:58.050ZINFOcrucible: Crucible stats registered with UUID: ff114f14-d00f-420e-966b-bf2151b7c457 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1702025-03-17T22:18:58.050ZWARNcrucible: could not find Downstairs address for Nexus session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1712025-03-17T22:18:58.051ZINFOcrucible: Downstairs in volume = 3 task = crutest
1722025-03-17T22:18:58.051ZINFOcrucible: The guest has requested activation with gen:2
1732025-03-17T22:18:58.051ZINFOcrucible: ff114f14-d00f-420e-966b-bf2151b7c457 active request set session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1742025-03-17T22:18:58.051ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1752025-03-17T22:18:58.051ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1762025-03-17T22:18:58.051ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1772025-03-17T22:18:58.051ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1782025-03-17T22:18:58.051ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1792025-03-17T22:18:58.051ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1802025-03-17T22:18:58.051ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1812025-03-17T22:18:58.051ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1822025-03-17T22:18:58.051ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1832025-03-17T22:18:58.051ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:60948) = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1842025-03-17T22:18:58.051ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:49458) = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1852025-03-17T22:18:58.051ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:40871) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1862025-03-17T22:18:58.052ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1872025-03-17T22:18:58.052ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1882025-03-17T22:18:58.052ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1892025-03-17T22:18:58.053ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1902025-03-17T22:18:58.053ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1912025-03-17T22:18:58.053ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1922025-03-17T22:18:58.053ZINFOcrucible: Waiting for more clients to be ready session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1932025-03-17T22:18:58.054ZINFOcrucible: Waiting for more clients to be ready session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1942025-03-17T22:18:58.054ZINFOcrucible: [0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1952025-03-17T22:18:58.054ZINFOcrucible: [0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1962025-03-17T22:18:58.054ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1972025-03-17T22:18:58.054ZINFOcrucible: [1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1982025-03-17T22:18:58.054ZINFOcrucible: [1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1992025-03-17T22:18:58.054ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
2002025-03-17T22:18:58.054ZINFOcrucible: [2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
2012025-03-17T22:18:58.054ZINFOcrucible: [2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
2022025-03-17T22:18:58.054ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
2032025-03-17T22:18:58.054ZINFOcrucible: Max found gen is 2 = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
2042025-03-17T22:18:58.054ZINFOcrucible: Generation requested: 2 >= found:2 = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
2052025-03-17T22:18:58.054ZINFOcrucible: Next flush: 2 = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
2062025-03-17T22:18:58.054ZINFOcrucible: All extents match = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
2072025-03-17T22:18:58.054ZINFOcrucible: No downstairs reconciliation required session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
2082025-03-17T22:18:58.054ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
2092025-03-17T22:18:58.054ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
210 {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:18:58.054376637Z","hostname"Activate successful
211 :"Wait for a query_work_queue command to finish before sending IO
212 w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"client":"2","":"downstairs","session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
2132025-03-17T22:18:58.054ZINFOcrucible: All required reconciliation work is completed session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
2142025-03-17T22:18:58.054ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
2152025-03-17T22:18:58.054ZINFOcrucible: ff114f14-d00f-420e-966b-bf2151b7c457 is now active with session: 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
2162025-03-17T22:18:58.054ZINFOcrucible: Set Active after no reconciliation session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
2172025-03-17T22:18:58.054ZINFOcrucible: The guest has finished waiting for activation with:2
2182025-03-17T22:18:58.054ZINFOcrucible: Activated sub_volume ff114f14-d00f-420e-966b-bf2151b7c457
219 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
220 Loading write count information from file "/tmp/test_replay_verify.log"
221 Mar 17 22:18:58.088 INFO accepted connection, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
222 Main task has work to do, go find it
223 got dsc Stop(1)
224 stop 1
225 Mar 17 22:18:58.089 INFO request completed, latency_us: 180, response_code: 204, uri: /stop/cid/1, method: POST, req_id: fe5ee89d-00b6-479c-927e-c6c60a6c3a7b, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
226 [8820] Got stop action so:false kr:false
2272025-03-17T22:18:58.089ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
2282025-03-17T22:18:58.090ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
2292025-03-17T22:18:58.090ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
230 Mar 17 22:18:58.090 INFO request completed, latency_us: 179, response_code: 200, uri: /state/cid/1, method: GET, req_id: 28481a21-a13b-498a-b0d6-6aa1dd2697fe, remote_addr: 127.0.0.1:58317, local_addr: 127.{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:18:58.090169068Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
231 0.0.1:9998
232 [8820] Exited with: ExitStatus(unix_wait_status(9))
233 [8820][1] reports Exit
234 Mar 17 22:19:02.092 INFO request completed, latency_us: 299, response_code: 200, uri: /state/cid/1, method: GET, req_id: 569968a2-8b61-4c2e-aa60-ab2257b29507, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
235 001/300 Write block 3001 len 4096 data: 2
236 002/300 Flush
237 003/300 Write block 4500 len 16384 data: 2 2 2 2
238 004/300 Read block 1740 len 36864
239 005/300 Read block 1772 len 40960
240 006/300 Read block 4900 len 16384
241 007/300 Write block 1546 len 40960 data: 2 2 2 2 2 2 2 2 2 2
242 008/300 Write block 4220 len 32768 data: 2 2 2 2 2 2 2 2
243 009/300 Write block 521 len 8192 data: 2 2
244 010/300 Read block 4908 len 40960
245 011/300 Write block 240 len 12288 data: 2 2 2
246 012/300 Read block 680 len 36864
247 013/300 Write block 2705 len 16384 data: 2 2 2 2
248 014/300 Write block 143 len 32768 data: 2 2 2 2 2 2 2 2
249 015/300 Read block 721 len 24576
250 016/300 Read block 3105 len 20480
251 017/300 Read block 1359 len 8192
252 018/300 Flush
253 019/300 Write block 2389 len 8192 data: 2 2
254 020/300 Flush
255 021/300 Write block 3683 len 20480 data: 2 2 2 2 2
256 022/300 Write block 1494 len 24576 data: 2 2 2 2 2 2
257 023/300 Write block 3938 len 40960 data: 2 2 2 2 2 2 2 2 2 2
258 024/300 Write block 3418 len 24576 data: 2 2 2 2 2 2
259 025/300 Read block 523 len 36864
260 026/300 Flush
261 027/300 Flush
262 028/300 Read block 2976 len 16384
263 029/300 Read block 861 len 28672
264 030/300 Flush
265 031/300 Read block 2974 len 16384
266 032/300 Read block 2410 len 8192
267 033/300 Read block 2611 len 28672
268 034/300 Flush
269 035/300 Read block 2909 len 24576
270 036/300 Write block 3034 len 36864 data: 2 2 2 2 2 2 2 2 2
271 037/300 Write block 1255 len 12288 data: 2 2 2
272 038/300 Write block 3013 len 20480 data: 2 2 2 2 2
273 039/300 Read block 2808 len 12288
274 040/300 Write block 2831 len 32768 data: 2 2 2 2 2 2 2 2
275 041/300 Write block 3299 len 36864 data: 2 2 2 2 2 2 2 2 2
276 042/300 Read block 434 len 32768
277 043/300 Write block 1285 len 24576 data: 2 2 2 2 2 2
278 044/300 Flush
279 045/300 Read block 1116 len 16384
280 046/300 Read block 316 len 28672
281 047/300 Read block 645 len 8192
282 048/300 Write block 1592 len 36864 data: 2 2 2 2 2 2 2 2 2
283 049/300 Write block 4448 len 8192 data: 2 2
284 050/300 Read block 1491 len 16384
285 051/300 Read block 3469 len 16384
286 052/300 Read block 3463 len 32768
287 053/300 Read block 636 len 40960
288 054/300 Flush
289 055/300 Read block 4770 len 36864
290 056/300 Write block 894 len 12288 data: 2 2 2
291 057/300 Flush
292 058/300 Write block 4766 len 40960 data: 2 2 2 2 2 2 2 2 2 2
293 059/300 Read block 3451 len 20480
294 060/300 Write block 1895 len 36864 data: 2 2 2 2 2 2 2 2 2
295 061/300 Read block 4365 len 20480
296 062/300 Read block 4763 len 24576
297 063/300 Read block 2381 len 16384
298 064/300 Read block 354 len 24576
299 065/300 Read block 3486 len 24576
300 066/300 Write block 4801 len 20480 data: 2 2 2 2 2
301 067/300 Read block 3101 len 20480
302 068/300 Write block 280 len 40960 data: 2 2 2 2 2 2 2 2 2 2
303 069/300 Read block 2670 len 36864
304 070/300 Flush
305 071/300 Read block 4228 len 28672
306 072/300 Read block 850 len 28672
307 073/300 Read block 319 len 8192
308 074/300 Read block 1692 len 8192
309 075/300 Read block 2165 len 16384
310 076/300 Read block 2882 len 40960
311 077/300 Read block 3847 len 8192
312 078/300 Write block 4176 len 16384 data: 2 2 2 2
313 079/300 Write block 3465 len 4096 data: 2
314 080/300 Write block 3406 len 4096 data: 2
315 081/300 Read block 4656 len 12288
316 082/300 Read block 1564 len 16384
317 083/300 Read block 2350 len 16384
318 084/300 Read block 2063 len 36864
319 085/300 Read block 2379 len 12288
320 086/300 Read block 3836 len 40960
321 087/300 Read block 2162 len 36864
322 088/300 Write block 4697 len 32768 data: 2 2 2 2 2 2 2 2
323 089/300 Read block 3384 len 40960
324 090/300 Read block 956 len 32768
325 091/300 Read block 2137 len 4096
326 092/300 Read block 2665 len 24576
327 093/300 Write block 4631 len 32768 data: 2 2 2 2 2 2 2 2
328 094/300 Write block 97 len 12288 data: 2 2 2
329 095/300 Read block 4842 len 40960
330 096/300 Write block 2140 len 12288 data: 2 2 2
331 097/300 Read block 612 len 40960
332 098/300 Write block 1648 len 4096 data: 2
333 099/300 Read block 594 len 8192
334 100/300 Read block 4847 len 28672
335 101/300 Read block 1987 len 8192
336 102/300 Read block 2222 len 16384
337 103/300 Write block 2146 len 8192 data: 2 2
338 104/300 Write block 2314 len 8192 data: 2 2
339 105/300 Write block 738 len 28672 data: 2 2 2 2 2 2 2
340 106/300 Write block 3345 len 12288 data: 2 2 2
341 107/300 Write block 3662 len 40960 data: 2 2 2 2 2 2 2 2 2 2
342 108/300 Write block 4378 len 40960 data: 2 2 2 2 2 2 2 2 2 2
343 109/300 Write block 3133 len 24576 data: 2 2 2 2 2 2
344 110/300 Read block 1621 len 12288
345 111/300 Flush
346 112/300 Write block 4383 len 28672 data: 3 3 3 3 3 2 2
347 113/300 Read block 2117 len 24576
348 114/300 Write block 4143 len 16384 data: 2 2 2 2
349 115/300 Write block 1815 len 12288 data: 2 2 2
350 116/300 Flush
351 117/300 Read block 631 len 40960
352 118/300 Read block 2203 len 12288
353 119/300 Read block 3335 len 8192
354 120/300 Write block 1 len 24576 data: 2 2 2 2 2 2
355 121/300 Read block 3215 len 40960
356 122/300 Read block 418 len 40960
357 123/300 Write block 3078 len 20480 data: 2 2 2 2 2
358 124/300 Write block 120 len 24576 data: 2 2 2 2 2 2
359 125/300 Write block 4082 len 16384 data: 2 2 2 2
360 126/300 Read block 2859 len 24576
361 127/300 Write block 4145 len 12288 data: 3 3 2
362 128/300 Write block 2545 len 24576 data: 2 2 2 2 2 2
363 129/300 Read block 1618 len 40960
364 130/300 Read block 670 len 4096
365 131/300 Write block 923 len 8192 data: 2 2
366 132/300 Write block 3165 len 12288 data: 2 2 2
367 133/300 Write block 1875 len 28672 data: 2 2 2 2 2 2 2
368 134/300 Write block 2967 len 8192 data: 2 2
369 135/300 Write block 371 len 24576 data: 2 2 2 2 2 2
370 136/300 Write block 3640 len 36864 data: 2 2 2 2 2 2 2 2 2
371 137/300 Read block 1897 len 36864
372 138/300 Write block 3054 len 16384 data: 2 2 2 2
373 139/300 Write block 2523 len 4096 data: 2
374 140/300 Write block 420 len 28672 data: 2 2 2 2 2 2 2
375 141/300 Read block 1681 len 4096
376 142/300 Read block 3360 len 20480
377 143/300 Flush
378 144/300 Read block 4668 len 32768
379 145/300 Flush
380 146/300 Write block 1789 len 4096 data: 2
381 147/300 Read block 1063 len 4096
382 148/300 Write block 2788 len 32768 data: 2 2 2 2 2 2 2 2
383 149/300 Flush
384 150/300 Write block 1114 len 24576 data: 2 2 2 2 2 2
385 151/300 Write block 1987 len 36864 data: 2 2 2 2 2 2 2 2 2
386 152/300 Flush
387 153/300 Write block 2390 len 36864 data: 3 2 2 2 2 2 2 2 2
388 154/300 Read block 4571 len 32768
389 155/300 Read block 184 len 28672
390 156/300 Read block 1719 len 8192
391 157/300 Read block 4240 len 12288
392 158/300 Write block 624 len 16384 data: 2 2 2 2
393 159/300 Read block 2314 len 16384
394 160/300 Read block 3640 len 16384
395 161/300 Read block 2967 len 32768
396 162/300 Read block 1793 len 32768
397 163/300 Write block 4511 len 40960 data: 2 2 2 2 2 2 2 2 2 2
398 164/300 Write block 957 len 4096 data: 2
399 165/300 Read block 4800 len 20480
400 166/300 Read block 3378 len 12288
401 167/300 Read block 1639 len 32768
402 168/300 Flush
403 169/300 Read block 2113 len 40960
404 170/300 Write block 1011 len 32768 data: 2 2 2 2 2 2 2 2
405 171/300 Write block 910 len 4096 data: 2
406 172/300 Write block 3248 len 12288 data: 2 2 2
407 173/300 Flush
408 174/300 Read block 3342 len 8192
409 175/300 Write block 3251 len 16384 data: 2 2 2 2
410 176/300 Flush
411 177/300 Read block 2285 len 40960
412 178/300 Write block 137 len 12288 data: 2 2 2
413 179/300 Write block 1566 len 28672 data: 2 2 2 2 2 2 2
414 180/300 Write block 4299 len 12288 data: 2 2 2
415 181/300 Read block 1252 len 20480
416 182/300 Write block 3172 len 24576 data: 2 2 2 2 2 2
417 183/300 Read block 4297 len 4096
418 184/300 Read block 694 len 28672
419 185/300 Flush
420 186/300 Read block 2292 len 24576
421 187/300 Read block 1396 len 12288
422 188/300 Write block 3502 len 28672 data: 2 2 2 2 2 2 2
423 189/300 Read block 80 len 16384
424 190/300 Write block 1660 len 32768 data: 2 2 2 2 2 2 2 2
425 191/300 Read block 4483 len 12288
426 192/300 Read block 2089 len 20480
427 193/300 Read block 1370 len 32768
428 194/300 Write block 1602 len 36864 data: 2 2 2 2 2 2 2 2 2
429 195/300 Read block 3835 len 20480
430 196/300 Read block 2423 len 16384
431 197/300 Read block 1007 len 28672
432 198/300 Flush
433 199/300 Read block 3625 len 36864
434 200/300 Read block 3199 len 8192
435 201/300 Read block 3087 len 12288
436 202/300 Flush
437 203/300 Read block 1542 len 8192
438 204/300 Write block 4669 len 12288 data: 2 2 2
439 205/300 Write block 1251 len 40960 data: 2 2 2 2 3 3 3 2 2 2
440 206/300 Read block 4613 len 36864
441 207/300 Write block 1123 len 20480 data: 2 2 2 2 2
442 208/300 Write block 1508 len 24576 data: 2 2 2 2 2 2
443 209/300 Read block 3405 len 36864
444 210/300 Write block 3972 len 36864 data: 2 2 2 2 2 2 2 2 2
445 211/300 Write block 843 len 20480 data: 2 2 2 2 2
446 212/300 Read block 2686 len 4096
447 213/300 Read block 388 len 24576
448 214/300 Write block 1037 len 12288 data: 2 2 2
449 215/300 Write block 2324 len 8192 data: 2 2
450 216/300 Flush
451 217/300 Read block 2661 len 4096
452 218/300 Read block 4045 len 32768
453 219/300 Write block 945 len 40960 data: 2 2 2 2 2 2 2 2 2 2
454 220/300 Write block 4496 len 32768 data: 2 2 2 2 3 3 3 3
455 221/300 Read block 2327 len 24576
456 222/300 Read block 958 len 8192
457 223/300 Flush
458 224/300 Read block 4141 len 32768
459 225/300 Read block 3396 len 36864
460 226/300 Read block 997 len 16384
461 227/300 Write block 882 len 36864 data: 2 2 2 2 2 2 2 2 2
462 228/300 Write block 4479 len 24576 data: 2 2 2 2 2 2
463 229/300 Flush
464 230/300 Flush
465 231/300 Write block 1347 len 40960 data: 2 2 2 2 2 2 2 2 2 2
466 232/300 Write block 1374 len 4096 data: 2
467 233/300 Read block 423 len 24576
468 234/300 Write block 3528 len 16384 data: 2 2 2 2
469 235/300 Flush
470 236/300 Read block 4814 len 36864
471 237/300 Write block 3578 len 8192 data: 2 2
472 238/300 Write block 823 len 8192 data: 2 2
473 239/300 Read block 440 len 4096
474 240/300 Flush
475 241/300 Read block 986 len 32768
476 242/300 Read block 2361 len 40960
477 243/300 Write block 2132 len 4096 data: 2
478 244/300 Read block 1804 len 16384
479 245/300 Write block 56 len 36864 data: 2 2 2 2 2 2 2 2 2
480 246/300 Write block 3271 len 20480 data: 2 2 2 2 2
481 247/300 Flush
482 248/300 Read block 2449 len 12288
483 249/300 Read block 1556 len 8192
484 250/300 Write block 2863 len 12288 data: 2 2 2
485 251/300 Read block 993 len 16384
486 252/300 Write block 2707 len 12288 data: 3 3 2
487 253/300 Read block 1508 len 28672
488 254/300 Read block 2296 len 16384
489 255/300 Read block 990 len 24576
490 256/300 Read block 212 len 4096
491 257/300 Read block 1893 len 36864
492 258/300 Write block 2311 len 16384 data: 2 2 2 3
493 259/300 Write block 1836 len 12288 data: 2 2 2
494 260/300 Flush
495 261/300 Write block 4013 len 28672 data: 2 2 2 2 2 2 2
496 262/300 Flush
497 263/300 Read block 2210 len 20480
498 264/300 Read block 3248 len 24576
499 265/300 Write block 3605 len 36864 data: 2 2 2 2 2 2 2 2 2
500 266/300 Read block 2076 len 32768
501 267/300 Read block 561 len 32768
502 268/300 Read block 1099 len 20480
503 269/300 Read block 3243 len 32768
504 270/300 Read block 2933 len 24576
505 271/300 Write block 676 len 12288 data: 2 2 2
506 272/300 Flush
507 273/300 Read block 4299 len 32768
508 274/300 Read block 1795 len 24576
509 275/300 Write block 3516 len 20480 data: 2 2 2 2 2
510 276/300 Write block 1847 len 20480 data: 2 2 2 2 2
511 277/300 Write block 1286 len 40960 data: 3 3 3 3 3 2 2 2 2 2
512 278/300 Write block 3178 len 16384 data: 2 2 2 2
513 279/300 Read block 4831 len 12288
514 280/300 Read block 2948 len 20480
515 281/300 Read block 509 len 4096
516 282/300 Read block 4747 len 36864
517 283/300 Read block 4508 len 24576
518 284/300 Read block 1369 len 40960
519 285/300 Read block 2000 len 12288
520 286/300 Read block 2462 len 24576
521 287/300 Read block 122 len 36864
522 288/300 Write block 1917 len 40960 data: 2 2 2 2 2 2 2 2 2 2
523 289/300 Write block 2496 len 8192 data: 2 2
524 290/300 Flush
525 291/300 Read block 1866 len 12288
526 292/300 Write block 534 len 8192 data: 2 2
527 293/300 Flush
528 294/300 Write block 857 len 20480 data: 2 2 2 2 2
529 295/300 Read block 2759 len 20480
530 296/300 Write block 2390 len 20480 data: 4 3 3 3 3
531 297/300 Flush
532 298/300 Read block 3508 len 20480
533 299/300 Write block 1809 len 36864 data: 2 2 2 2 2 2 3 3 3
534 300/300 Write block 1728 len 4096 data: 2
535 301/300 Read block 3550 len 32768
536 Main task has work to do, go find it
537 got dsc Start(1)
538 start 1
539 Mar 17 22:19:02.339 INFO request completed, latency_us: 226, response_code: 204, uri: /start/cid/1, method: POST, req_id: 5486feb6-d274-46d1-ae03-6e2477439a5f, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
540 Replay: started 1, returned:Ok(())
541 ----------------------------------------------------------------
542 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
543 States: Active Offline Active
544 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
545 1000 Acked Write 1 Done Sent Done false
546 1001 Acked Flush 0 Done Sent Done false
547 1002 Acked Write 4 Done Sent Done false
548 1003 Acked Read 9 Done Sent Done false
549 1004 Acked Read 10 Done Sent Done false
550 1005 Acked Read 4 Done Sent Done false
551 1006 Acked Write 10 Done Sent Done false
552 1007 Acked Write 8 Done Sent Done false
553 1008 Acked Write 2 Done Sent Done false
554 1009 Acked Read 10 Done Sent Done false
555 1010 Acked Write 3 Done Sent Done false
556 1011 Acked Read 9 Done Sent Done false
557 1012 Acked Write 4 Done Sent Done false
558 1013 Acked Write 8 Done Sent Done false
559 1014 Acked Read 6 Done Sent Done false
560 1015 Acked Read 5 Done Sent Done false
561 1016 Acked Read 2 Done Sent Done false
562 1017 Acked Flush 0 Done Sent Done false
563 1018 Acked Write 2 Done Sent Done false
564 1019 Acked Flush 0 Done Sent Done false
565 1020 Acked Write 5 Done Sent Done false
566 1021 Acked Write 6 Done Sent Done false
567 1022 Acked Write 10 Done Sent Done false
568 1023 Acked Write 6 Done Sent Done false
569 1024 Acked Read 9 Done Sent Done false
570 1025 Acked Flush 0 Done Sent Done false
571 1026 Acked Flush 0 Done Sent Done false
572 1027 Acked Read 4 Done Sent Done false
573 1028 Acked Read 7 Done Sent Done false
574 1029 Acked Flush 0 Done Sent Done false
575 1030 Acked Read 4 Done Sent Done false
576 1031 Acked Read 2 Done Sent Done false
577 1032 Acked Read 7 Done Sent Done false
578 1033 Acked Flush 0 Done Sent Done false
579 1034 Acked Read 6 Done Sent Done false
580 1035 Acked Write 9 Done Sent Done false
581 1036 Acked Write 3 Done Sent Done false
582 1037 Acked Write 5 Done Sent Done false
583 1038 Acked Read 3 Done Sent Done false
584 1039 Acked Write 8 Done Sent Done false
585 1040 Acked Write 9 Done Sent Done false
586 1041 Acked Read 8 Done Sent Done false
587 1042 Acked Write 6 Done Sent Done false
588 1043 Acked Flush 0 Done Sent Done false
589 1044 Acked Read 4 Done Sent Done false
590 1045 Acked Read 7 Done Sent Done false
591 1046 Acked Read 2 Done Sent Done false
592 1047 Acked Write 9 Done Sent Done false
593 1048 Acked Write 2 Done Sent Done false
594 1049 Acked Read 4 Done Sent Done false
595 1050 Acked Read 4 Done Sent Done false
596 1051 Acked Read 8 Done Sent Done false
597 1052 Acked Read 10 Done Sent Done false
598 1053 Acked Flush 0 Done Sent Done false
599 1054 Acked Read 9 Done Sent Done false
600 1055 Acked Write 3 Done Sent Done false
601 1056 Acked Flush 0 Done Sent Done false
602 1057 Acked Write 10 Done Sent Done false
603 1058 Acked Read 5 Done Sent Done false
604 1059 Acked Write 9 Done Sent Done false
605 1060 Acked Read 5 Done Sent Done false
606 1061 Acked Read 6 Done Sent Done false
607 1062 Acked Read 4 Done Sent Done false
608 1063 Acked Read 6 Done Sent Done false
609 1064 Acked Read 6 Done Sent Done false
610 1065 Acked Write 5 Done Sent Done false
611 1066 Acked Read 5 Done Sent Done false
612 1067 Acked Write 10 Done Sent Done false
613 1068 Acked Read 9 Done Sent Done false
614 1069 Acked Flush 0 Done Sent Done false
615 1070 Acked Read 7 Done Sent Done false
616 1071 Acked Read 7 Done Sent Done false
617 1072 Acked Read 2 Done Sent Done false
618 1073 Acked Read 2 Done Sent Done false
619 1074 Acked Read 4 Done Sent Done false
620 1075 Acked Read 10 Done Sent Done false
621 1076 Acked Read 2 Done Sent Done false
622 1077 Acked Write 4 Done Sent Done false
623 1078 Acked Write 1 Done Sent Done false
624 1079 Acked Write 1 Done Sent Done false
625 1080 Acked Read 3 Done Sent Done false
626 1081 Acked Read 4 Done Sent Done false
627 1082 Acked Read 4 Done Sent Done false
628 1083 Acked Read 9 Done Sent Done false
629 1084 Acked Read 3 Done Sent Done false
630 1085 Acked Read 10 Done Sent Done false
631 1086 Acked Read 9 Done Sent Done false
632 1087 Acked Write 8 Done Sent Done false
633 1088 Acked Read 10 Done Sent Done false
634 1089 Acked Read 8 Done Sent Done false
635 1090 Acked Read 1 Done Sent Done false
636 1091 Acked Read 6 Done Sent Done false
637 1092 Acked Write 8 Done Sent Done false
638 1093 Acked Write 3 Done Sent Done false
639 1094 Acked Read 10 Done Sent Done false
640 1095 Acked Write 3 Done Sent Done false
641 1096 Acked Read 10 Done Sent Done false
642 1097 Acked Write 1 Done Sent Done false
643 1098 Acked Read 2 Done Sent Done false
644 1099 Acked Read 7 Done Sent Done false
645 1100 Acked Read 2 Done Sent Done false
646 1101 Acked Read 4 Done Sent Done false
647 1102 Acked Write 2 Done Sent Done false
648 1103 Acked Write 2 Done Sent Done false
649 1104 Acked Write 7 Done Sent Done false
650 1105 Acked Write 3 Done Sent Done false
651 1106 Acked Write 10 Done Sent Done false
652 1107 Acked Write 10 Done Sent Done false
653 1108 Acked Write 6 Done Sent Done false
654 1109 Acked Read 3 Done Sent Done false
655 1110 Acked Flush 0 Done Sent Done false
656 1111 Acked Write 7 Done Sent Done false
657 1112 Acked Read 6 Done Sent Done false
658 1113 Acked Write 4 Done Sent Done false
659 1114 Acked Write 3 Done Sent Done false
660 1115 Acked Flush 0 Done Sent Done false
661 1116 Acked Read 10 Done Sent Done false
662 1117 Acked Read 3 Done Sent Done false
663 1118 Acked Read 2 Done Sent Done false
664 1119 Acked Write 6 Done Sent Done false
665 1120 Acked Read 10 Done Sent Done false
666 1121 Acked Read 10 Done Sent Done false
667 1122 Acked Write 5 Done Sent Done false
668 1123 Acked Write 6 Done Sent Done false
669 1124 Acked Write 4 Done Sent Done false
670 1125 Acked Read 6 Done Sent Done false
671 1126 Acked Write 3 Done Sent Done false
672 1127 Acked Write 6 Done Sent Done false
673 1128 Acked Read 10 Done Sent Done false
674 1129 Acked Read 1 Done Sent Done false
675 1130 Acked Write 2 Done Sent Done false
676 1131 Acked Write 3 Done Sent Done false
677 1132 Acked Write 7 Done Sent Done false
678 1133 Acked Write 2 Done Sent Done false
679 1134 Acked Write 6 Done Sent Done false
680 1135 Acked Write 9 Done Sent Done false
681 1136 Acked Read 9 Done Sent Done false
682 1137 Acked Write 4 Done Sent Done false
683 1138 Acked Write 1 Done Sent Done false
684 1139 Acked Write 7 Done Sent Done false
685 1140 Acked Read 1 Done Sent Done false
686 1141 Acked Read 5 Done Sent Done false
687 1142 Acked Flush 0 Done Sent Done false
688 1143 Acked Read 8 Done Sent Done false
689 1144 Acked Flush 0 Done Sent Done false
690 1145 Acked Write 1 Done Sent Done false
691 1146 Acked Read 1 Done Sent Done false
692 1147 Acked Write 8 Done Sent Done false
693 1148 Acked Flush 0 Done Sent Done false
694 1149 Acked Write 6 Done Sent Done false
695 1150 Acked Write 9 Done Sent Done false
696 1151 Acked Flush 0 Done Sent Done false
697 1152 Acked Write 9 Done Sent Done false
698 1153 Acked Read 8 Done Sent Done false
699 1154 Acked Read 7 Done Sent Done false
700 1155 Acked Read 2 Done Sent Done false
701 1156 Acked Read 3 Done Sent Done false
702 1157 Acked Write 4 Done Sent Done false
703 1158 Acked Read 4 Done Sent Done false
704 1159 Acked Read 4 Done Sent Done false
705 1160 Acked Read 8 Done Sent Done false
706 1161 Acked Read 8 Done Sent Done false
707 1162 Acked Write 10 Done Sent Done false
708 1163 Acked Write 1 Done Sent Done false
709 1164 Acked Read 5 Done Sent Done false
710 1165 Acked Read 3 Done Sent Done false
711 1166 Acked Read 8 Done Sent Done false
712 1167 Acked Flush 0 Done Sent Done false
713 1168 Acked Read 10 Done Sent Done false
714 1169 Acked Write 8 Done Sent Done false
715 1170 Acked Write 1 Done Sent Done false
716 1171 Acked Write 3 Done Sent Done false
717 1172 Acked Flush 0 Done Sent Done false
718 1173 Acked Read 2 Done Sent Done false
719 1174 Acked Write 4 Done Sent Done false
720 1175 Acked Flush 0 Done Sent Done false
721 1176 Acked Read 10 Done Sent Done false
722 1177 Acked Write 3 Done Sent Done false
723 1178 Acked Write 7 Done Sent Done false
724 1179 Acked Write 3 Done Sent Done false
725 1180 Acked Read 5 Done Sent Done false
726 1181 Acked Write 6 Done Sent Done false
727 1182 Acked Read 1 Done Sent Done false
728 1183 Acked Read 7 Done Sent Done false
729 1184 Acked Flush 0 Done Sent Done false
730 1185 Acked Read 6 Done Sent Done false
731 1186 Acked Read 3 Done Sent Done false
732 1187 Acked Write 7 Done Sent Done false
733 1188 Acked Read 4 Done Sent Done false
734 1189 Acked Write 8 Done Sent Done false
735 1190 Acked Read 3 Done Sent Done false
736 1191 Acked Read 5 Done Sent Done false
737 1192 Acked Read 8 Done Sent Done false
738 1193 Acked Write 9 Done Sent Done false
739 1194 Acked Read 5 Done Sent Done false
740 1195 Acked Read 4 Done Sent Done false
741 1196 Acked Read 7 Done Sent Done false
742 1197 Acked Flush 0 Done Sent Done false
743 1198 Acked Read 9 Done Sent Done false
744 1199 Acked Read 2 Done Sent Done false
745 1200 Acked Read 3 Done Sent Done false
746 1201 Acked Flush 0 Done Sent Done false
747 1202 Acked Read 2 Done Sent Done false
748 1203 Acked Write 3 Done Sent Done false
749 1204 Acked Write 10 Done Sent Done false
750 1205 Acked Read 9 Done Sent Done false
751 1206 Acked Write 5 Done Sent Done false
752 1207 Acked Write 6 Done Sent Done false
753 1208 Acked Read 9 Done Sent Done false
754 1209 Acked Write 9 Done Sent Done false
755 1210 Acked Write 5 Done Sent Done false
756 1211 Acked Read 1 Done Sent Done false
757 1212 Acked Read 6 Done Sent Done false
758 1213 Acked Write 3 Done Sent Done false
759 1214 Acked Write 2 Done Sent Done false
760 1215 Acked Flush 0 Done Sent Done false
761 1216 Acked Read 1 Done Sent Done false
762 1217 Acked Read 8 Done Sent Done false
763 1218 Acked Write 10 Done Sent Done false
764 1219 Acked Write 8 Done Sent Done false
765 1220 Acked Read 6 Done Sent Done false
766 1221 Acked Read 2 Done Sent Done false
767 1222 Acked Flush 0 Done Sent Done false
768 1223 Acked Read 8 Done Sent Done false
769 1224 Acked Read 9 Done Sent Done false
770 1225 Acked Read 4 Done Sent Done false
771 1226 Acked Write 9 Done Sent Done false
772 1227 Acked Write 6 Done Sent Done false
773 1228 Acked Flush 0 Done Sent Done false
774 1229 Acked Flush 0 Done Sent Done false
775 1230 Acked Write 10 Done Sent Done false
776 1231 Acked Write 1 Done Sent Done false
777 1232 Acked Read 6 Done Sent Done false
778 1233 Acked Write 4 Done Sent Done false
779 1234 Acked Flush 0 Done Sent Done false
780 1235 Acked Read 9 Done Sent Done false
781 1236 Acked Write 2 Done Sent Done false
782 1237 Acked Write 2 Done Sent Done false
783 1238 Acked Read 1 Done Sent Done false
784 1239 Acked Flush 0 Done Sent Done false
785 1240 Acked Read 8 Done Sent Done false
786 1241 Acked Read 10 Done Sent Done false
787 1242 Acked Write 1 Done Sent Done false
788 1243 Acked Read 4 Done Sent Done false
789 1244 Acked Write 9 Done Sent Done false
790 1245 Acked Write 5 Done Sent Done false
791 1246 Acked Flush 0 Done Sent Done false
792 1247 Acked Read 3 Done Sent Done false
793 1248 Acked Read 2 Done Sent Done false
794 1249 Acked Write 3 Done Sent Done false
795 1250 Acked Read 4 Done Sent Done false
796 1251 Acked Write 3 Done Sent Done false
797 1252 Acked Read 7 Done Sent Done false
798 1253 Acked Read 4 Done Sent Done false
799 1254 Acked Read 6 Done Sent Done false
800 1255 Acked Read 1 Done Sent Done false
801 1256 Acked Read 9 Done Sent Done false
802 1257 Acked Write 4 Done Sent Done false
803 1258 Acked Write 3 Done Sent Done false
804 1259 Acked Flush 0 Done Sent Done false
805 1260 Acked Write 7 Done Sent Done false
806 1261 Acked Flush 0 Done Sent Done false
807 1262 Acked Read 5 Done Sent Done false
808 1263 Acked Read 6 Done Sent Done false
809 1264 Acked Write 9 Done Sent Done false
810 1265 Acked Read 8 Done Sent Done false
811 1266 Acked Read 8 Done Sent Done false
812 1267 Acked Read 5 Done Sent Done false
813 1268 Acked Read 8 Done Sent Done false
814 1269 Acked Read 6 Done Sent Done false
815 1270 Acked Write 3 Done Sent Done false
816 1271 Acked Flush 0 Done Sent Done false
817 1272 Acked Read 8 Done Sent Done false
818 1273 Acked Read 6 Done Sent Done false
819 1274 Acked Write 5 Done Sent Done false
820 1275 Acked Write 5 Done Sent Done false
821 1276 Acked Write 10 Done Sent Done false
822 1277 Acked Write 4 Done Sent Done false
823 1278 Acked Read 3 Done Sent Done false
824 1279 Acked Read 5 Done Sent Done false
825 1280 Acked Read 1 Done Sent Done false
826 1281 Acked Read 9 Done Sent Done false
827 1282 Acked Read 6 Done Sent Done false
828 1283 Acked Read 10 Done Sent Done false
829 1284 Acked Read 3 Done Sent Done false
830 1285 Acked Read 6 Done Sent Done false
831 1286 Acked Read 9 Done Sent Done false
832 1287 Acked Write 10 Done Sent Done false
833 1288 Acked Write 2 Done Sent Done false
834 1289 Acked Flush 0 Done Sent Done false
835 1290 Acked Read 3 Done Sent Done false
836 1291 Acked Write 2 Done Sent Done false
837 1292 Acked Flush 0 Done Sent Done false
838 1293 Acked Write 5 Done Sent Done false
839 1294 Acked Read 5 Done Sent Done false
840 1295 Acked Write 5 Done Sent Done false
841 1296 Acked Flush 0 Done Sent Done false
842 1297 Acked Read 5 Done Sent Done false
843 1298 Acked Write 9 Done Sent Done false
844 1299 Acked Write 1 Done Sent Done false
845 1300 Acked Read 8 Done Sent Done false
846 STATES DS:0 DS:1 DS:2 TOTAL
847 Sent 0 301 0 301
848 Done 301 0 301 602
849 Skipped 0 0 0 0
850 Error 0 0 0 0
851 Last Flush: Some(JobId(1296)) None Some(JobId(1296))
852 Downstairs last five completed:
853 Upstairs last five completed: 1300 1299 1298 1297 1296
854 CLIENT: Up:0 ds:301 act:2
8552025-03-17T22:19:02.841ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
8562025-03-17T22:19:03.559ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
8572025-03-17T22:19:04.561ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
8582025-03-17T22:19:05.564ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
859 [8820] got start action
860 [8820] I am going to restart
861 Starting downstairs at port 8820
862 Make output file at "/tmp/dsc/downstairs-8820.txt"
863 [8820][1] reports Starting
864 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1212)
865 [8820][1] reports Running
866 ----------------------------------------------------------------
867 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
868 States: Active Offline Active
869 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
870 1000 Acked Write 1 Done Sent Done false
871 1001 Acked Flush 0 Done Sent Done false
872 1002 Acked Write 4 Done Sent Done false
873 1003 Acked Read 9 Done Sent Done false
874 1004 Acked Read 10 Done Sent Done false
875 1005 Acked Read 4 Done Sent Done false
876 1006 Acked Write 10 Done Sent Done false
877 1007 Acked Write 8 Done Sent Done false
878 1008 Acked Write 2 Done Sent Done false
879 1009 Acked Read 10 Done Sent Done false
880 1010 Acked Write 3 Done Sent Done false
881 1011 Acked Read 9 Done Sent Done false
882 1012 Acked Write 4 Done Sent Done false
883 1013 Acked Write 8 Done Sent Done false
884 1014 Acked Read 6 Done Sent Done false
885 1015 Acked Read 5 Done Sent Done false
886 1016 Acked Read 2 Done Sent Done false
887 1017 Acked Flush 0 Done Sent Done false
888 1018 Acked Write 2 Done Sent Done false
889 1019 Acked Flush 0 Done Sent Done false
890 1020 Acked Write 5 Done Sent Done false
891 1021 Acked Write 6 Done Sent Done false
892 1022 Acked Write 10 Done Sent Done false
893 1023 Acked Write 6 Done Sent Done false
894 1024 Acked Read 9 Done Sent Done false
895 1025 Acked Flush 0 Done Sent Done false
896 1026 Acked Flush 0 Done Sent Done false
897 1027 Acked Read 4 Done Sent Done false
898 1028 Acked Read 7 Done Sent Done false
899 1029 Acked Flush 0 Done Sent Done false
900 1030 Acked Read 4 Done Sent Done false
901 1031 Acked Read 2 Done Sent Done false
902 1032 Acked Read 7 Done Sent Done false
903 1033 Acked Flush 0 Done Sent Done false
904 1034 Acked Read 6 Done Sent Done false
905 1035 Acked Write 9 Done Sent Done false
906 1036 Acked Write 3 Done Sent Done false
907 1037 Acked Write 5 Done Sent Done false
908 1038 Acked Read 3 Done Sent Done false
909 1039 Acked Write 8 Done Sent Done false
910 1040 Acked Write 9 Done Sent Done false
911 1041 Acked Read 8 Done Sent Done false
912 1042 Acked Write 6 Done Sent Done false
913 1043 Acked Flush 0 Done Sent Done false
914 1044 Acked Read 4 Done Sent Done false
915 1045 Acked Read 7 Done Sent Done false
916 1046 Acked Read 2 Done Sent Done false
917 1047 Acked Write 9 Done Sent Done false
918 1048 Acked Write 2 Done Sent Done false
919 1049 Acked Read 4 Done Sent Done false
920 1050 Acked Read 4 Done Sent Done false
921 1051 Acked Read 8 Done Sent Done false
922 1052 Acked Read 10 Done Sent Done false
923 1053 Acked Flush 0 Done Sent Done false
924 1054 Acked Read 9 Done Sent Done false
925 1055 Acked Write 3 Done Sent Done false
926 1056 Acked Flush 0 Done Sent Done false
927 1057 Acked Write 10 Done Sent Done false
928 1058 Acked Read 5 Done Sent Done false
929 1059 Acked Write 9 Done Sent Done false
930 1060 Acked Read 5 Done Sent Done false
931 1061 Acked Read 6 Done Sent Done false
932 1062 Acked Read 4 Done Sent Done false
933 1063 Acked Read 6 Done Sent Done false
934 1064 Acked Read 6 Done Sent Done false
935 1065 Acked Write 5 Done Sent Done false
936 1066 Acked Read 5 Done Sent Done false
937 1067 Acked Write 10 Done Sent Done false
938 1068 Acked Read 9 Done Sent Done false
939 1069 Acked Flush 0 Done Sent Done false
940 1070 Acked Read 7 Done Sent Done false
941 1071 Acked Read 7 Done Sent Done false
942 1072 Acked Read 2 Done Sent Done false
943 1073 Acked Read 2 Done Sent Done false
944 1074 Acked Read 4 Done Sent Done false
945 1075 Acked Read 10 Done Sent Done false
946 1076 Acked Read 2 Done Sent Done false
947 1077 Acked Write 4 Done Sent Done false
948 1078 Acked Write 1 Done Sent Done false
949 1079 Acked Write 1 Done Sent Done false
950 1080 Acked Read 3 Done Sent Done false
951 1081 Acked Read 4 Done Sent Done false
952 1082 Acked Read 4 Done Sent Done false
953 1083 Acked Read 9 Done Sent Done false
954 1084 Acked Read 3 Done Sent Done false
955 1085 Acked Read 10 Done Sent Done false
956 1086 Acked Read 9 Done Sent Done false
957 1087 Acked Write 8 Done Sent Done false
958 1088 Acked Read 10 Done Sent Done false
959 1089 Acked Read 8 Done Sent Done false
960 1090 Acked Read 1 Done Sent Done false
961 1091 Acked Read 6 Done Sent Done false
962 1092 Acked Write 8 Done Sent Done false
963 1093 Acked Write 3 Done Sent Done false
964 1094 Acked Read 10 Done Sent Done false
965 1095 Acked Write 3 Done Sent Done false
966 1096 Acked Read 10 Done Sent Done false
967 1097 Acked Write 1 Done Sent Done false
968 1098 Acked Read 2 Done Sent Done false
969 1099 Acked Read 7 Done Sent Done false
970 1100 Acked Read 2 Done Sent Done false
971 1101 Acked Read 4 Done Sent Done false
972 1102 Acked Write 2 Done Sent Done false
973 1103 Acked Write 2 Done Sent Done false
974 1104 Acked Write 7 Done Sent Done false
975 1105 Acked Write 3 Done Sent Done false
976 1106 Acked Write 10 Done Sent Done false
977 1107 Acked Write 10 Done Sent Done false
978 1108 Acked Write 6 Done Sent Done false
979 1109 Acked Read 3 Done Sent Done false
980 1110 Acked Flush 0 Done Sent Done false
981 1111 Acked Write 7 Done Sent Done false
982 1112 Acked Read 6 Done Sent Done false
983 1113 Acked Write 4 Done Sent Done false
984 1114 Acked Write 3 Done Sent Done false
985 1115 Acked Flush 0 Done Sent Done false
986 1116 Acked Read 10 Done Sent Done false
987 1117 Acked Read 3 Done Sent Done false
988 1118 Acked Read 2 Done Sent Done false
989 1119 Acked Write 6 Done Sent Done false
990 1120 Acked Read 10 Done Sent Done false
991 1121 Acked Read 10 Done Sent Done false
992 1122 Acked Write 5 Done Sent Done false
993 1123 Acked Write 6 Done Sent Done false
994 1124 Acked Write 4 Done Sent Done false
995 1125 Acked Read 6 Done Sent Done false
996 1126 Acked Write 3 Done Sent Done false
997 1127 Acked Write 6 Done Sent Done false
998 1128 Acked Read 10 Done Sent Done false
999 1129 Acked Read 1 Done Sent Done false
1000 1130 Acked Write 2 Done Sent Done false
1001 1131 Acked Write 3 Done Sent Done false
1002 1132 Acked Write 7 Done Sent Done false
1003 1133 Acked Write 2 Done Sent Done false
1004 1134 Acked Write 6 Done Sent Done false
1005 1135 Acked Write 9 Done Sent Done false
1006 1136 Acked Read 9 Done Sent Done false
1007 1137 Acked Write 4 Done Sent Done false
1008 1138 Acked Write 1 Done Sent Done false
1009 1139 Acked Write 7 Done Sent Done false
1010 1140 Acked Read 1 Done Sent Done false
1011 1141 Acked Read 5 Done Sent Done false
1012 1142 Acked Flush 0 Done Sent Done false
1013 1143 Acked Read 8 Done Sent Done false
1014 1144 Acked Flush 0 Done Sent Done false
1015 1145 Acked Write 1 Done Sent Done false
1016 1146 Acked Read 1 Done Sent Done false
1017 1147 Acked Write 8 Done Sent Done false
1018 1148 Acked Flush 0 Done Sent Done false
1019 1149 Acked Write 6 Done Sent Done false
1020 1150 Acked Write 9 Done Sent Done false
1021 1151 Acked Flush 0 Done Sent Done false
1022 1152 Acked Write 9 Done Sent Done false
1023 1153 Acked Read 8 Done Sent Done false
1024 1154 Acked Read 7 Done Sent Done false
1025 1155 Acked Read 2 Done Sent Done false
1026 1156 Acked Read 3 Done Sent Done false
1027 1157 Acked Write 4 Done Sent Done false
1028 1158 Acked Read 4 Done Sent Done false
1029 1159 Acked Read 4 Done Sent Done false
1030 1160 Acked Read 8 Done Sent Done false
1031 1161 Acked Read 8 Done Sent Done false
1032 1162 Acked Write 10 Done Sent Done false
1033 1163 Acked Write 1 Done Sent Done false
1034 1164 Acked Read 5 Done Sent Done false
1035 1165 Acked Read 3 Done Sent Done false
1036 1166 Acked Read 8 Done Sent Done false
1037 1167 Acked Flush 0 Done Sent Done false
1038 1168 Acked Read 10 Done Sent Done false
1039 1169 Acked Write 8 Done Sent Done false
1040 1170 Acked Write 1 Done Sent Done false
1041 1171 Acked Write 3 Done Sent Done false
1042 1172 Acked Flush 0 Done Sent Done false
1043 1173 Acked Read 2 Done Sent Done false
1044 1174 Acked Write 4 Done Sent Done false
1045 1175 Acked Flush 0 Done Sent Done false
1046 1176 Acked Read 10 Done Sent Done false
1047 1177 Acked Write 3 Done Sent Done false
1048 1178 Acked Write 7 Done Sent Done false
1049 1179 Acked Write 3 Done Sent Done false
1050 1180 Acked Read 5 Done Sent Done false
1051 1181 Acked Write 6 Done Sent Done false
1052 1182 Acked Read 1 Done Sent Done false
1053 1183 Acked Read 7 Done Sent Done false
1054 1184 Acked Flush 0 Done Sent Done false
1055 1185 Acked Read 6 Done Sent Done false
1056 1186 Acked Read 3 Done Sent Done false
1057 1187 Acked Write 7 Done Sent Done false
1058 1188 Acked Read 4 Done Sent Done false
1059 1189 Acked Write 8 Done Sent Done false
1060 1190 Acked Read 3 Done Sent Done false
1061 1191 Acked Read 5 Done Sent Done false
1062 1192 Acked Read 8 Done Sent Done false
1063 1193 Acked Write 9 Done Sent Done false
1064 1194 Acked Read 5 Done Sent Done false
1065 1195 Acked Read 4 Done Sent Done false
1066 1196 Acked Read 7 Done Sent Done false
1067 1197 Acked Flush 0 Done Sent Done false
1068 1198 Acked Read 9 Done Sent Done false
1069 1199 Acked Read 2 Done Sent Done false
1070 1200 Acked Read 3 Done Sent Done false
1071 1201 Acked Flush 0 Done Sent Done false
1072 1202 Acked Read 2 Done Sent Done false
1073 1203 Acked Write 3 Done Sent Done false
1074 1204 Acked Write 10 Done Sent Done false
1075 1205 Acked Read 9 Done Sent Done false
1076 1206 Acked Write 5 Done Sent Done false
1077 1207 Acked Write 6 Done Sent Done false
1078 1208 Acked Read 9 Done Sent Done false
1079 1209 Acked Write 9 Done Sent Done false
1080 1210 Acked Write 5 Done Sent Done false
1081 1211 Acked Read 1 Done Sent Done false
1082 1212 Acked Read 6 Done Sent Done false
1083 1213 Acked Write 3 Done Sent Done false
1084 1214 Acked Write 2 Done Sent Done false
1085 1215 Acked Flush 0 Done Sent Done false
1086 1216 Acked Read 1 Done Sent Done false
1087 1217 Acked Read 8 Done Sent Done false
1088 1218 Acked Write 10 Done Sent Done false
1089 1219 Acked Write 8 Done Sent Done false
1090 1220 Acked Read 6 Done Sent Done false
1091 1221 Acked Read 2 Done Sent Done false
1092 1222 Acked Flush 0 Done Sent Done false
1093 1223 Acked Read 8 Done Sent Done false
1094 1224 Acked Read 9 Done Sent Done false
1095 1225 Acked Read 4 Done Sent Done false
1096 1226 Acked Write 9 Done Sent Done false
1097 1227 Acked Write 6 Done Sent Done false
1098 1228 Acked Flush 0 Done Sent Done false
1099 1229 Acked Flush 0 Done Sent Done false
1100 1230 Acked Write 10 Done Sent Done false
1101 1231 Acked Write 1 Done Sent Done false
1102 1232 Acked Read 6 Done Sent Done false
1103 1233 Acked Write 4 Done Sent Done false
1104 1234 Acked Flush 0 Done Sent Done false
1105 1235 Acked Read 9 Done Sent Done false
1106 1236 Acked Write 2 Done Sent Done false
1107 1237 Acked Write 2 Done Sent Done false
1108 1238 Acked Read 1 Done Sent Done false
1109 1239 Acked Flush 0 Done Sent Done false
1110 1240 Acked Read 8 Done Sent Done false
1111 1241 Acked Read 10 Done Sent Done false
1112 1242 Acked Write 1 Done Sent Done false
1113 1243 Acked Read 4 Done Sent Done false
1114 1244 Acked Write 9 Done Sent Done false
1115 1245 Acked Write 5 Done Sent Done false
1116 1246 Acked Flush 0 Done Sent Done false
1117 1247 Acked Read 3 Done Sent Done false
1118 1248 Acked Read 2 Done Sent Done false
1119 1249 Acked Write 3 Done Sent Done false
1120 1250 Acked Read 4 Done Sent Done false
1121 1251 Acked Write 3 Done Sent Done false
1122 1252 Acked Read 7 Done Sent Done false
1123 1253 Acked Read 4 Done Sent Done false
1124 1254 Acked Read 6 Done Sent Done false
1125 1255 Acked Read 1 Done Sent Done false
1126 1256 Acked Read 9 Done Sent Done false
1127 1257 Acked Write 4 Done Sent Done false
1128 1258 Acked Write 3 Done Sent Done false
1129 1259 Acked Flush 0 Done Sent Done false
1130 1260 Acked Write 7 Done Sent Done false
1131 1261 Acked Flush 0 Done Sent Done false
1132 1262 Acked Read 5 Done Sent Done false
1133 1263 Acked Read 6 Done Sent Done false
1134 1264 Acked Write 9 Done Sent Done false
1135 1265 Acked Read 8 Done Sent Done false
1136 1266 Acked Read 8 Done Sent Done false
1137 1267 Acked Read 5 Done Sent Done false
1138 1268 Acked Read 8 Done Sent Done false
1139 1269 Acked Read 6 Done Sent Done false
1140 1270 Acked Write 3 Done Sent Done false
1141 1271 Acked Flush 0 Done Sent Done false
1142 1272 Acked Read 8 Done Sent Done false
1143 1273 Acked Read 6 Done Sent Done false
1144 1274 Acked Write 5 Done Sent Done false
1145 1275 Acked Write 5 Done Sent Done false
1146 1276 Acked Write 10 Done Sent Done false
1147 1277 Acked Write 4 Done Sent Done false
1148 1278 Acked Read 3 Done Sent Done false
1149 1279 Acked Read 5 Done Sent Done false
1150 1280 Acked Read 1 Done Sent Done false
1151 1281 Acked Read 9 Done Sent Done false
1152 1282 Acked Read 6 Done Sent Done false
1153 1283 Acked Read 10 Done Sent Done false
1154 1284 Acked Read 3 Done Sent Done false
1155 1285 Acked Read 6 Done Sent Done false
1156 1286 Acked Read 9 Done Sent Done false
1157 1287 Acked Write 10 Done Sent Done false
1158 1288 Acked Write 2 Done Sent Done false
1159 1289 Acked Flush 0 Done Sent Done false
1160 1290 Acked Read 3 Done Sent Done false
1161 1291 Acked Write 2 Done Sent Done false
1162 1292 Acked Flush 0 Done Sent Done false
1163 1293 Acked Write 5 Done Sent Done false
1164 1294 Acked Read 5 Done Sent Done false
1165 1295 Acked Write 5 Done Sent Done false
1166 1296 Acked Flush 0 Done Sent Done false
1167 1297 Acked Read 5 Done Sent Done false
1168 1298 Acked Write 9 Done Sent Done false
1169 1299 Acked Write 1 Done Sent Done false
1170 1300 Acked Read 8 Done Sent Done false
1171 STATES DS:0 DS:1 DS:2 TOTAL
1172 Sent 0 301 0 301
1173 Done 301 0 301 602
1174 Skipped 0 0 0 0
1175 Error 0 0 0 0
1176 Last Flush: Some(JobId(1296)) None Some(JobId(1296))
1177 Downstairs last five completed:
1178 Upstairs last five completed: 1300 1299 1298 1297 1296
1179 CLIENT: Up:0 ds:301 act:2
11802025-03-17T22:19:06.843ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
11812025-03-17T22:19:07.566ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
11822025-03-17T22:19:08.092ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
11832025-03-17T22:19:08.092ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:65119) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
11842025-03-17T22:19:08.093ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
11852025-03-17T22:19:08.093ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
11862025-03-17T22:19:08.093ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
11872025-03-17T22:19:08.093ZINFOcrucible: send last flush ID to this DS: None = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
11882025-03-17T22:19:08.093ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
11892025-03-17T22:19:08.093ZINFOcrucible: [1] client re-new 301 jobs since flush None = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
11902025-03-17T22:19:08.093ZINFOcrucible: [1] Replayed 301 jobs since flush: None = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1191 ----------------------------------------------------------------
1192 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
1193 Downstairs last five completed: 1301 1300 1299 1298 1297
1194 Upstairs last five completed: 1301 1300 1299 1298 1297
1195 CLIENT: Up:0 ds:0 act:3
1196 Replay: All jobs finished
1197 Main task has work to do, go find it
1198 got dsc Stop(2)
1199 stop 2
1200 [8830] Got stop action so:false kr:false
1201 Mar 17 22:19:10.344 INFO request completed, latency_us: 502, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 15beec10-7b59-4bc3-b2ae-a8254569cbb2, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
12022025-03-17T22:19:10.345ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
12032025-03-17T22:19:10.345ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
12042025-03-17T22:19:10.345ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
12052025-03-17T22:19:10.345ZINFOcrucible: client task is exiting = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1206 Mar 17 22:19:10.345 INFO request completed, latency_us: 159, response_code: 200, uri: /state/cid/2, method: GET, req_id: c54348b1-8679-4fa8-b017-e62b495dcb35, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
1207 [8830] Exited with: ExitStatus(unix_wait_status(9))
1208 [8830][2] reports Exit
1209 Mar 17 22:19:14.349 INFO request completed, latency_us: 327, response_code: 200, uri: /state/cid/2, method: GET, req_id: aeabff9f-a26f-4fb8-8520-180d25ce2da3, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
1210 001/300 Read block 2557 len 16384
1211 002/300 Write block 4956 len 12288 data: 2 2 2
1212 003/300 Read block 648 len 16384
1213 004/300 Flush
1214 005/300 Read block 5 len 36864
1215 006/300 Write block 467 len 16384 data: 2 2 2 2
1216 007/300 Flush
1217 008/300 Read block 3102 len 24576
1218 009/300 Read block 2176 len 36864
1219 010/300 Read block 4238 len 32768
1220 011/300 Read block 3726 len 12288
1221 012/300 Read block 1964 len 28672
1222 013/300 Write block 1559 len 24576 data: 2 2 2 2 2 2
1223 014/300 Write block 4961 len 32768 data: 2 2 2 2 2 2 2 2
1224 015/300 Read block 962 len 4096
1225 016/300 Read block 2455 len 36864
1226 017/300 Write block 1298 len 12288 data: 2 2 2
1227 018/300 Read block 1724 len 12288
1228 019/300 Read block 1527 len 28672
1229 020/300 Write block 2696 len 28672 data: 2 2 2 2 2 2 2
1230 021/300 Write block 1863 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1231 022/300 Read block 751 len 12288
1232 023/300 Read block 2633 len 24576
1233 024/300 Read block 926 len 4096
1234 025/300 Write block 1563 len 12288 data: 3 3 2
1235 026/300 Flush
1236 027/300 Write block 1931 len 24576 data: 2 2 2 2 2 2
1237 028/300 Read block 3420 len 12288
1238 029/300 Write block 2258 len 28672 data: 2 2 2 2 2 2 2
1239 030/300 Write block 4148 len 24576 data: 2 2 2 2 2 2
1240 031/300 Flush
1241 032/300 Read block 407 len 4096
1242 033/300 Read block 1129 len 16384
1243 034/300 Read block 2794 len 12288
1244 035/300 Read block 4481 len 36864
1245 036/300 Flush
1246 037/300 Read block 4305 len 24576
1247 038/300 Read block 1235 len 20480
1248 039/300 Read block 1874 len 32768
1249 040/300 Read block 976 len 20480
1250 041/300 Write block 519 len 8192 data: 2 2
1251 042/300 Write block 3080 len 12288 data: 3 3 3
1252 043/300 Write block 1839 len 12288 data: 2 2 2
1253 044/300 Write block 3542 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1254 045/300 Write block 420 len 16384 data: 3 3 3 3
1255 046/300 Read block 1954 len 36864
1256 047/300 Write block 870 len 32768 data: 2 2 2 2 2 2 2 2
1257 048/300 Read block 2313 len 16384
1258 049/300 Write block 2469 len 12288 data: 2 2 2
1259 050/300 Write block 318 len 36864 data: 2 2 2 2 2 2 2 2 2
1260 051/300 Write block 4412 len 32768 data: 2 2 2 2 2 2 2 2
1261 052/300 Flush
1262 053/300 Read block 2686 len 36864
1263 054/300 Write block 2197 len 4096 data: 2
1264 055/300 Write block 3545 len 36864 data: 3 3 3 3 3 3 3 2 2
1265 056/300 Read block 1810 len 28672
1266 057/300 Read block 2195 len 20480
1267 058/300 Read block 3749 len 4096
1268 059/300 Read block 4879 len 32768
1269 060/300 Flush
1270 061/300 Read block 1450 len 32768
1271 062/300 Write block 3327 len 8192 data: 2 2
1272 063/300 Write block 781 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1273 064/300 Read block 3845 len 40960
1274 065/300 Write block 4531 len 32768 data: 2 2 2 2 2 2 2 2
1275 066/300 Read block 868 len 32768
1276 067/300 Write block 446 len 16384 data: 2 2 2 2
1277 068/300 Flush
1278 069/300 Read block 4094 len 24576
1279 070/300 Read block 368 len 8192
1280 071/300 Write block 1272 len 16384 data: 2 2 2 2
1281 072/300 Read block 1871 len 32768
1282 073/300 Read block 4312 len 36864
1283 074/300 Read block 1976 len 32768
1284 075/300 Write block 1986 len 28672 data: 2 3 3 3 3 3 3
1285 076/300 Write block 4764 len 16384 data: 2 2 3 3
1286 077/300 Write block 257 len 4096 data: 2
1287 078/300 Read block 4302 len 16384
1288 079/300 Write block 3834 len 16384 data: 2 2 2 2
1289 080/300 Read block 2884 len 4096
1290 081/300 Read block 734 len 12288
1291 082/300 Read block 4152 len 16384
1292 083/300 Write block 1200 len 16384 data: 2 2 2 2
1293 084/300 Read block 4601 len 36864
1294 085/300 Read block 2831 len 28672
1295 086/300 Read block 326 len 20480
1296 087/300 Read block 4546 len 40960
1297 088/300 Read block 4086 len 40960
1298 089/300 Read block 4090 len 20480
1299 090/300 Write block 3043 len 36864 data: 2 2 2 2 2 2 2 2 2
1300 091/300 Read block 3875 len 32768
1301 092/300 Write block 1473 len 12288 data: 2 2 2
1302 093/300 Write block 4547 len 20480 data: 2 2 2 2 2
1303 094/300 Write block 449 len 40960 data: 3 2 2 2 2 2 2 2 2 2
1304 095/300 Read block 4670 len 24576
1305 096/300 Flush
1306 097/300 Write block 1745 len 16384 data: 2 2 2 2
1307 098/300 Write block 3197 len 32768 data: 2 2 2 2 2 2 2 2
1308 099/300 Read block 2081 len 16384
1309 100/300 Read block 1119 len 4096
1310 101/300 Read block 2734 len 4096
1311 102/300 Write block 4366 len 4096 data: 2
1312 103/300 Read block 4810 len 8192
1313 104/300 Read block 227 len 16384
1314 105/300 Read block 1851 len 40960
1315 106/300 Read block 4101 len 16384
1316 107/300 Write block 1527 len 4096 data: 2
1317 108/300 Read block 1781 len 16384
1318 109/300 Write block 1710 len 4096 data: 2
1319 110/300 Flush
1320 111/300 Write block 4074 len 24576 data: 2 2 2 2 2 2
1321 112/300 Write block 2287 len 12288 data: 2 2 2
1322 113/300 Write block 4641 len 32768 data: 2 2 2 2 2 2 2 2
1323 114/300 Read block 3023 len 16384
1324 115/300 Flush
1325 116/300 Write block 2011 len 36864 data: 2 2 2 2 2 2 2 2 2
1326 117/300 Write block 3874 len 24576 data: 2 2 2 2 2 2
1327 118/300 Write block 100 len 28672 data: 2 2 2 2 2 2 2
1328 119/300 Read block 1654 len 40960
1329 120/300 Read block 2827 len 36864
1330 121/300 Read block 4777 len 40960
1331 122/300 Read block 1344 len 24576
1332 123/300 Write block 4312 len 28672 data: 2 2 2 2 2 2 2
1333 124/300 Write block 2397 len 24576 data: 3 3 2 2 2 2
1334 125/300 Flush
1335 126/300 Write block 3035 len 36864 data: 3 3 3 3 3 3 3 3 3
1336 127/300 Write block 4548 len 20480 data: 3 3 3 3 2
1337 128/300 Read block 4198 len 28672
1338 129/300 Read block 2661 len 20480
1339 130/300 Write block 4257 len 32768 data: 2 2 2 2 2 2 2 2
1340 131/300 Read block 3277 len 36864
1341 132/300 Write block 4328 len 36864 data: 2 2 2 2 2 2 2 2 2
1342 133/300 Write block 2441 len 36864 data: 2 2 2 2 2 2 2 2 2
1343 134/300 Read block 1609 len 8192
1344 135/300 Write block 4195 len 16384 data: 2 2 2 2
1345 136/300 Write block 1828 len 24576 data: 2 2 2 2 2 2
1346 137/300 Read block 1922 len 20480
1347 138/300 Write block 4367 len 16384 data: 2 2 2 2
1348 139/300 Flush
1349 140/300 Write block 266 len 32768 data: 2 2 2 2 2 2 2 2
1350 141/300 Read block 26 len 40960
1351 142/300 Read block 2355 len 8192
1352 143/300 Write block 3247 len 20480 data: 2 3 3 3 3
1353 144/300 Read block 1655 len 16384
1354 145/300 Write block 2189 len 40960 data: 2 2 2 2 2 2 2 2 3 2
1355 146/300 Read block 495 len 28672
1356 147/300 Read block 2466 len 16384
1357 148/300 Flush
1358 149/300 Write block 2275 len 28672 data: 2 2 2 2 2 2 2
1359 150/300 Write block 4858 len 4096 data: 2
1360 151/300 Write block 4711 len 4096 data: 2
1361 152/300 Flush
1362 153/300 Read block 113 len 40960
1363 154/300 Read block 4362 len 24576
1364 155/300 Read block 2101 len 40960
1365 156/300 Write block 4231 len 4096 data: 2
1366 157/300 Write block 915 len 40960 data: 2 2 2 2 2 2 2 2 3 3
1367 158/300 Read block 4318 len 20480
1368 159/300 Read block 306 len 8192
1369 160/300 Write block 580 len 16384 data: 2 2 2 2
1370 161/300 Read block 1078 len 36864
1371 162/300 Read block 2551 len 32768
1372 163/300 Flush
1373 164/300 Write block 2429 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1374 165/300 Write block 187 len 8192 data: 2 2
1375 166/300 Write block 2066 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1376 167/300 Read block 4203 len 12288
1377 168/300 Read block 912 len 12288
1378 169/300 Write block 4190 len 8192 data: 2 2
1379 170/300 Flush
1380 171/300 Read block 4299 len 32768
1381 172/300 Write block 895 len 4096 data: 3
1382 173/300 Read block 2747 len 36864
1383 174/300 Read block 820 len 4096
1384 175/300 Read block 494 len 32768
1385 176/300 Write block 2130 len 4096 data: 2
1386 177/300 Write block 2165 len 16384 data: 2 2 2 2
1387 178/300 Write block 4651 len 20480 data: 2 2 2 2 2
1388 179/300 Write block 725 len 24576 data: 2 2 2 2 2 2
1389 180/300 Read block 2997 len 24576
1390 181/300 Write block 4677 len 28672 data: 2 2 2 2 2 2 2
1391 182/300 Write block 2040 len 12288 data: 2 2 2
1392 183/300 Write block 1693 len 32768 data: 2 2 2 2 2 2 2 2
1393 184/300 Write block 3347 len 16384 data: 3 2 2 2
1394 185/300 Read block 2314 len 20480
1395 186/300 Flush
1396 187/300 Write block 772 len 36864 data: 2 2 2 2 2 2 2 2 2
1397 188/300 Flush
1398 189/300 Read block 3390 len 8192
1399 190/300 Read block 2998 len 4096
1400 191/300 Read block 2593 len 16384
1401 192/300 Read block 1946 len 12288
1402 193/300 Read block 331 len 12288
1403 194/300 Read block 4465 len 24576
1404 195/300 Write block 809 len 24576 data: 2 2 2 2 2 2
1405 196/300 Flush
1406 197/300 Write block 1633 len 16384 data: 2 2 2 2
1407 198/300 Read block 4464 len 4096
1408 199/300 Write block 37 len 36864 data: 2 2 2 2 2 2 2 2 2
1409 200/300 Write block 1883 len 16384 data: 2 2 2 2
1410 201/300 Read block 2018 len 4096
1411 202/300 Write block 1752 len 36864 data: 2 2 2 2 2 2 2 2 2
1412 203/300 Write block 1074 len 24576 data: 2 2 2 2 2 2
1413 204/300 Read block 1697 len 28672
1414 205/300 Write block 3193 len 36864 data: 2 2 2 2 3 3 3 3 3
1415 206/300 Write block 460 len 36864 data: 2 2 2 2 2 2 2 3 3
1416 207/300 Write block 1305 len 28672 data: 2 2 2 2 2 2 2
1417 208/300 Write block 3691 len 12288 data: 2 2 2
1418 209/300 Read block 4558 len 28672
1419 210/300 Write block 2064 len 32768 data: 2 2 3 3 3 3 3 3
1420 211/300 Read block 3953 len 4096
1421 212/300 Read block 4546 len 28672
1422 213/300 Read block 500 len 8192
1423 214/300 Flush
1424 215/300 Write block 1754 len 40960 data: 3 3 3 3 3 3 3 2 2 2
1425 216/300 Read block 1360 len 8192
1426 217/300 Read block 1758 len 16384
1427 218/300 Flush
1428 219/300 Write block 3535 len 36864 data: 2 2 2 2 2 2 2 3 3
1429 220/300 Write block 1994 len 16384 data: 3 3 2 2
1430 221/300 Read block 752 len 24576
1431 222/300 Read block 1713 len 28672
1432 223/300 Write block 659 len 4096 data: 2
1433 224/300 Read block 3979 len 4096
1434 225/300 Read block 3859 len 16384
1435 226/300 Read block 4269 len 8192
1436 227/300 Write block 2742 len 36864 data: 2 2 2 2 2 2 2 2 2
1437 228/300 Read block 4814 len 4096
1438 229/300 Write block 3258 len 20480 data: 2 2 2 2 2
1439 230/300 Write block 4591 len 8192 data: 2 2
1440 231/300 Read block 3575 len 20480
1441 232/300 Write block 446 len 24576 data: 3 3 3 4 3 3
1442 233/300 Read block 4067 len 32768
1443 234/300 Write block 1744 len 36864 data: 2 3 3 3 3 2 2 2 3
1444 235/300 Read block 1246 len 8192
1445 236/300 Read block 108 len 24576
1446 237/300 Flush
1447 238/300 Write block 3256 len 8192 data: 2 2
1448 239/300 Read block 1112 len 4096
1449 240/300 Write block 4413 len 8192 data: 3 3
1450 241/300 Read block 1732 len 20480
1451 242/300 Write block 1374 len 16384 data: 3 2 2 2
1452 243/300 Read block 193 len 32768
1453 244/300 Write block 2792 len 4096 data: 3
1454 245/300 Write block 1917 len 32768 data: 3 3 3 3 3 3 3 3
1455 246/300 Write block 1978 len 32768 data: 2 2 2 2 2 2 2 2
1456 247/300 Read block 3673 len 32768
1457 248/300 Write block 4305 len 4096 data: 2
1458 249/300 Read block 3177 len 16384
1459 250/300 Read block 2790 len 32768
1460 251/300 Read block 2069 len 4096
1461 252/300 Read block 4637 len 20480
1462 253/300 Read block 2500 len 40960
1463 254/300 Write block 4737 len 20480 data: 2 2 2 2 2
1464 255/300 Read block 4404 len 12288
1465 256/300 Read block 4048 len 24576
1466 257/300 Read block 1798 len 8192
1467 258/300 Write block 3741 len 36864 data: 2 2 2 2 2 2 2 2 2
1468 259/300 Write block 1158 len 24576 data: 2 2 2 2 2 2
1469 260/300 Write block 4591 len 4096 data: 3
1470 261/300 Flush
1471 262/300 Write block 2914 len 4096 data: 2
1472 263/300 Write block 2284 len 40960 data: 2 2 2 3 3 3 2 2 2 2
1473 264/300 Write block 131 len 16384 data: 2 2 2 2
1474 265/300 Read block 4412 len 32768
1475 266/300 Read block 3041 len 20480
1476 267/300 Write block 2449 len 16384 data: 3 2 2 2
1477 268/300 Write block 3531 len 32768 data: 3 2 2 2 3 3 3 3
1478 269/300 Write block 3097 len 28672 data: 2 2 2 2 2 2 2
1479 270/300 Write block 4995 len 20480 data: 2 2 2 2 2
1480 271/300 Read block 3749 len 36864
1481 272/300 Write block 3750 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1482 273/300 Write block 1710 len 28672 data: 3 2 2 2 2 2 2
1483 274/300 Read block 1514 len 20480
1484 275/300 Write block 4259 len 4096 data: 3
1485 276/300 Write block 3147 len 8192 data: 2 2
1486 277/300 Write block 1499 len 36864 data: 3 2 2 2 2 2 2 2 2
1487 278/300 Write block 2018 len 28672 data: 3 3 2 2 2 2 2
1488 279/300 Flush
1489 280/300 Read block 2386 len 40960
1490 281/300 Write block 430 len 28672 data: 2 2 2 2 2 2 2
1491 282/300 Write block 3308 len 36864 data: 2 2 2 2 2 2 2 2 2
1492 283/300 Write block 2146 len 32768 data: 3 3 2 2 2 2 2 2
1493 284/300 Write block 1278 len 16384 data: 2 2 2 2
1494 285/300 Read block 1846 len 36864
1495 286/300 Read block 2524 len 40960
1496 287/300 Flush
1497 288/300 Read block 815 len 4096
1498 289/300 Write block 2550 len 4096 data: 3
1499 290/300 Write block 2474 len 24576 data: 2 2 2 2 2 2
1500 291/300 Read block 3212 len 28672
1501 292/300 Read block 4416 len 12288
1502 293/300 Write block 1595 len 28672 data: 3 3 3 3 3 3 2
1503 294/300 Read block 3107 len 28672
1504 295/300 Read block 1551 len 16384
1505 296/300 Read block 2345 len 32768
1506 297/300 Read block 1948 len 36864
1507 298/300 Read block 2604 len 28672
1508 299/300 Read block 2486 len 16384
1509 300/300 Write block 507 len 12288 data: 2 2 2
1510 301/300 Read block 1752 len 16384
1511 Main task has work to do, go find it
1512 got dsc Start(2)
1513 start 2
1514 Mar 17 22:19:14.586 INFO request completed, latency_us: 218, response_code: 204, uri: /start/cid/2, method: POST, req_id: 3ccbec2e-eb7f-4596-8788-40ffe054343d, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
1515 Replay: started 2, returned:Ok(())
1516 ----------------------------------------------------------------
1517 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1518 States: Active Active Offline
1519 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1520 1302 Acked Read 4 Done Done Sent false
1521 1303 Acked Write 3 Done Done Sent false
1522 1304 Acked Read 4 Done Done Sent false
1523 1305 Acked Flush 0 Done Done Sent false
1524 1306 Acked Read 9 Done Done Sent false
1525 1307 Acked Write 4 Done Done Sent false
1526 1308 Acked Flush 0 Done Done Sent false
1527 1309 Acked Read 6 Done Done Sent false
1528 1310 Acked Read 9 Done Done Sent false
1529 1311 Acked Read 8 Done Done Sent false
1530 1312 Acked Read 3 Done Done Sent false
1531 1313 Acked Read 7 Done Done Sent false
1532 1314 Acked Write 6 Done Done Sent false
1533 1315 Acked Write 8 Done Done Sent false
1534 1316 Acked Read 1 Done Done Sent false
1535 1317 Acked Read 9 Done Done Sent false
1536 1318 Acked Write 3 Done Done Sent false
1537 1319 Acked Read 3 Done Done Sent false
1538 1320 Acked Read 7 Done Done Sent false
1539 1321 Acked Write 7 Done Done Sent false
1540 1322 Acked Write 10 Done Done Sent false
1541 1323 Acked Read 3 Done Done Sent false
1542 1324 Acked Read 6 Done Done Sent false
1543 1325 Acked Read 1 Done Done Sent false
1544 1326 Acked Write 3 Done Done Sent false
1545 1327 Acked Flush 0 Done Done Sent false
1546 1328 Acked Write 6 Done Done Sent false
1547 1329 Acked Read 3 Done Done Sent false
1548 1330 Acked Write 7 Done Done Sent false
1549 1331 Acked Write 6 Done Done Sent false
1550 1332 Acked Flush 0 Done Done Sent false
1551 1333 Acked Read 1 Done Done Sent false
1552 1334 Acked Read 4 Done Done Sent false
1553 1335 Acked Read 3 Done Done Sent false
1554 1336 Acked Read 9 Done Done Sent false
1555 1337 Acked Flush 0 Done Done Sent false
1556 1338 Acked Read 6 Done Done Sent false
1557 1339 Acked Read 5 Done Done Sent false
1558 1340 Acked Read 8 Done Done Sent false
1559 1341 Acked Read 5 Done Done Sent false
1560 1342 Acked Write 2 Done Done Sent false
1561 1343 Acked Write 3 Done Done Sent false
1562 1344 Acked Write 3 Done Done Sent false
1563 1345 Acked Write 10 Done Done Sent false
1564 1346 Acked Write 4 Done Done Sent false
1565 1347 Acked Read 9 Done Done Sent false
1566 1348 Acked Write 8 Done Done Sent false
1567 1349 Acked Read 4 Done Done Sent false
1568 1350 Acked Write 3 Done Done Sent false
1569 1351 Acked Write 9 Done Done Sent false
1570 1352 Acked Write 8 Done Done Sent false
1571 1353 Acked Flush 0 Done Done Sent false
1572 1354 Acked Read 9 Done Done Sent false
1573 1355 Acked Write 1 Done Done Sent false
1574 1356 Acked Write 9 Done Done Sent false
1575 1357 Acked Read 7 Done Done Sent false
1576 1358 Acked Read 5 Done Done Sent false
1577 1359 Acked Read 1 Done Done Sent false
1578 1360 Acked Read 8 Done Done Sent false
1579 1361 Acked Flush 0 Done Done Sent false
1580 1362 Acked Read 8 Done Done Sent false
1581 1363 Acked Write 2 Done Done Sent false
1582 1364 Acked Write 10 Done Done Sent false
1583 1365 Acked Read 10 Done Done Sent false
1584 1366 Acked Write 8 Done Done Sent false
1585 1367 Acked Read 8 Done Done Sent false
1586 1368 Acked Write 4 Done Done Sent false
1587 1369 Acked Flush 0 Done Done Sent false
1588 1370 Acked Read 6 Done Done Sent false
1589 1371 Acked Read 2 Done Done Sent false
1590 1372 Acked Write 4 Done Done Sent false
1591 1373 Acked Read 8 Done Done Sent false
1592 1374 Acked Read 9 Done Done Sent false
1593 1375 Acked Read 8 Done Done Sent false
1594 1376 Acked Write 7 Done Done Sent false
1595 1377 Acked Write 4 Done Done Sent false
1596 1378 Acked Write 1 Done Done Sent false
1597 1379 Acked Read 4 Done Done Sent false
1598 1380 Acked Write 4 Done Done Sent false
1599 1381 Acked Read 1 Done Done Sent false
1600 1382 Acked Read 3 Done Done Sent false
1601 1383 Acked Read 4 Done Done Sent false
1602 1384 Acked Write 4 Done Done Sent false
1603 1385 Acked Read 9 Done Done Sent false
1604 1386 Acked Read 7 Done Done Sent false
1605 1387 Acked Read 5 Done Done Sent false
1606 1388 Acked Read 10 Done Done Sent false
1607 1389 Acked Read 10 Done Done Sent false
1608 1390 Acked Read 5 Done Done Sent false
1609 1391 Acked Write 9 Done Done Sent false
1610 1392 Acked Read 8 Done Done Sent false
1611 1393 Acked Write 3 Done Done Sent false
1612 1394 Acked Write 5 Done Done Sent false
1613 1395 Acked Write 10 Done Done Sent false
1614 1396 Acked Read 6 Done Done Sent false
1615 1397 Acked Flush 0 Done Done Sent false
1616 1398 Acked Write 4 Done Done Sent false
1617 1399 Acked Write 8 Done Done Sent false
1618 1400 Acked Read 4 Done Done Sent false
1619 1401 Acked Read 1 Done Done Sent false
1620 1402 Acked Read 1 Done Done Sent false
1621 1403 Acked Write 1 Done Done Sent false
1622 1404 Acked Read 2 Done Done Sent false
1623 1405 Acked Read 4 Done Done Sent false
1624 1406 Acked Read 10 Done Done Sent false
1625 1407 Acked Read 4 Done Done Sent false
1626 1408 Acked Write 1 Done Done Sent false
1627 1409 Acked Read 4 Done Done Sent false
1628 1410 Acked Write 1 Done Done Sent false
1629 1411 Acked Flush 0 Done Done Sent false
1630 1412 Acked Write 6 Done Done Sent false
1631 1413 Acked Write 3 Done Done Sent false
1632 1414 Acked Write 8 Done Done Sent false
1633 1415 Acked Read 4 Done Done Sent false
1634 1416 Acked Flush 0 Done Done Sent false
1635 1417 Acked Write 9 Done Done Sent false
1636 1418 Acked Write 6 Done Done Sent false
1637 1419 Acked Write 7 Done Done Sent false
1638 1420 Acked Read 10 Done Done Sent false
1639 1421 Acked Read 9 Done Done Sent false
1640 1422 Acked Read 10 Done Done Sent false
1641 1423 Acked Read 6 Done Done Sent false
1642 1424 Acked Write 7 Done Done Sent false
1643 1425 Acked Write 6 Done Done Sent false
1644 1426 Acked Flush 0 Done Done Sent false
1645 1427 Acked Write 9 Done Done Sent false
1646 1428 Acked Write 5 Done Done Sent false
1647 1429 Acked Read 7 Done Done Sent false
1648 1430 Acked Read 5 Done Done Sent false
1649 1431 Acked Write 8 Done Done Sent false
1650 1432 Acked Read 9 Done Done Sent false
1651 1433 Acked Write 9 Done Done Sent false
1652 1434 Acked Write 9 Done Done Sent false
1653 1435 Acked Read 2 Done Done Sent false
1654 1436 Acked Write 4 Done Done Sent false
1655 1437 Acked Write 6 Done Done Sent false
1656 1438 Acked Read 5 Done Done Sent false
1657 1439 Acked Write 4 Done Done Sent false
1658 1440 Acked Flush 0 Done Done Sent false
1659 1441 Acked Write 8 Done Done Sent false
1660 1442 Acked Read 10 Done Done Sent false
1661 1443 Acked Read 2 Done Done Sent false
1662 1444 Acked Write 5 Done Done Sent false
1663 1445 Acked Read 4 Done Done Sent false
1664 1446 Acked Write 10 Done Done Sent false
1665 1447 Acked Read 7 Done Done Sent false
1666 1448 Acked Read 4 Done Done Sent false
1667 1449 Acked Flush 0 Done Done Sent false
1668 1450 Acked Write 7 Done Done Sent false
1669 1451 Acked Write 1 Done Done Sent false
1670 1452 Acked Write 1 Done Done Sent false
1671 1453 Acked Flush 0 Done Done Sent false
1672 1454 Acked Read 10 Done Done Sent false
1673 1455 Acked Read 6 Done Done Sent false
1674 1456 Acked Read 10 Done Done Sent false
1675 1457 Acked Write 1 Done Done Sent false
1676 1458 Acked Write 10 Done Done Sent false
1677 1459 Acked Read 5 Done Done Sent false
1678 1460 Acked Read 2 Done Done Sent false
1679 1461 Acked Write 4 Done Done Sent false
1680 1462 Acked Read 9 Done Done Sent false
1681 1463 Acked Read 8 Done Done Sent false
1682 1464 Acked Flush 0 Done Done Sent false
1683 1465 Acked Write 10 Done Done Sent false
1684 1466 Acked Write 2 Done Done Sent false
1685 1467 Acked Write 10 Done Done Sent false
1686 1468 Acked Read 3 Done Done Sent false
1687 1469 Acked Read 3 Done Done Sent false
1688 1470 Acked Write 2 Done Done Sent false
1689 1471 Acked Flush 0 Done Done Sent false
1690 1472 Acked Read 8 Done Done Sent false
1691 1473 Acked Write 1 Done Done Sent false
1692 1474 Acked Read 9 Done Done Sent false
1693 1475 Acked Read 1 Done Done Sent false
1694 1476 Acked Read 8 Done Done Sent false
1695 1477 Acked Write 1 Done Done Sent false
1696 1478 Acked Write 4 Done Done Sent false
1697 1479 Acked Write 5 Done Done Sent false
1698 1480 Acked Write 6 Done Done Sent false
1699 1481 Acked Read 6 Done Done Sent false
1700 1482 Acked Write 7 Done Done Sent false
1701 1483 Acked Write 3 Done Done Sent false
1702 1484 Acked Write 8 Done Done Sent false
1703 1485 Acked Write 4 Done Done Sent false
1704 1486 Acked Read 5 Done Done Sent false
1705 1487 Acked Flush 0 Done Done Sent false
1706 1488 Acked Write 9 Done Done Sent false
1707 1489 Acked Flush 0 Done Done Sent false
1708 1490 Acked Read 2 Done Done Sent false
1709 1491 Acked Read 1 Done Done Sent false
1710 1492 Acked Read 4 Done Done Sent false
1711 1493 Acked Read 3 Done Done Sent false
1712 1494 Acked Read 3 Done Done Sent false
1713 1495 Acked Read 6 Done Done Sent false
1714 1496 Acked Write 6 Done Done Sent false
1715 1497 Acked Flush 0 Done Done Sent false
1716 1498 Acked Write 4 Done Done Sent false
1717 1499 Acked Read 1 Done Done Sent false
1718 1500 Acked Write 9 Done Done Sent false
1719 1501 Acked Write 4 Done Done Sent false
1720 1502 Acked Read 1 Done Done Sent false
1721 1503 Acked Write 9 Done Done Sent false
1722 1504 Acked Write 6 Done Done Sent false
1723 1505 Acked Read 7 Done Done Sent false
1724 1506 Acked Write 9 Done Done Sent false
1725 1507 Acked Write 9 Done Done Sent false
1726 1508 Acked Write 7 Done Done Sent false
1727 1509 Acked Write 3 Done Done Sent false
1728 1510 Acked Read 7 Done Done Sent false
1729 1511 Acked Write 8 Done Done Sent false
1730 1512 Acked Read 1 Done Done Sent false
1731 1513 Acked Read 7 Done Done Sent false
1732 1514 Acked Read 2 Done Done Sent false
1733 1515 Acked Flush 0 Done Done Sent false
1734 1516 Acked Write 10 Done Done Sent false
1735 1517 Acked Read 2 Done Done Sent false
1736 1518 Acked Read 4 Done Done Sent false
1737 1519 Acked Flush 0 Done Done Sent false
1738 1520 Acked Write 9 Done Done Sent false
1739 1521 Acked Write 4 Done Done Sent false
1740 1522 Acked Read 6 Done Done Sent false
1741 1523 Acked Read 7 Done Done Sent false
1742 1524 Acked Write 1 Done Done Sent false
1743 1525 Acked Read 1 Done Done Sent false
1744 1526 Acked Read 4 Done Done Sent false
1745 1527 Acked Read 2 Done Done Sent false
1746 1528 Acked Write 9 Done Done Sent false
1747 1529 Acked Read 1 Done Done Sent false
1748 1530 Acked Write 5 Done Done Sent false
1749 1531 Acked Write 2 Done Done Sent false
1750 1532 Acked Read 5 Done Done Sent false
1751 1533 Acked Write 6 Done Done Sent false
1752 1534 Acked Read 8 Done Done Sent false
1753 1535 Acked Write 9 Done Done Sent false
1754 1536 Acked Read 2 Done Done Sent false
1755 1537 Acked Read 6 Done Done Sent false
1756 1538 Acked Flush 0 Done Done Sent false
1757 1539 Acked Write 2 Done Done Sent false
1758 1540 Acked Read 1 Done Done Sent false
1759 1541 Acked Write 2 Done Done Sent false
1760 1542 Acked Read 5 Done Done Sent false
1761 1543 Acked Write 4 Done Done Sent false
1762 1544 Acked Read 8 Done Done Sent false
1763 1545 Acked Write 1 Done Done Sent false
1764 1546 Acked Write 8 Done Done Sent false
1765 1547 Acked Write 8 Done Done Sent false
1766 1548 Acked Read 8 Done Done Sent false
1767 1549 Acked Write 1 Done Done Sent false
1768 1550 Acked Read 4 Done Done Sent false
1769 1551 Acked Read 8 Done Done Sent false
1770 1552 Acked Read 1 Done Done Sent false
1771 1553 Acked Read 5 Done Done Sent false
1772 1554 Acked Read 10 Done Done Sent false
1773 1555 Acked Write 5 Done Done Sent false
1774 1556 Acked Read 3 Done Done Sent false
1775 1557 Acked Read 6 Done Done Sent false
1776 1558 Acked Read 2 Done Done Sent false
1777 1559 Acked Write 9 Done Done Sent false
1778 1560 Acked Write 6 Done Done Sent false
1779 1561 Acked Write 1 Done Done Sent false
1780 1562 Acked Flush 0 Done Done Sent false
1781 1563 Acked Write 1 Done Done Sent false
1782 1564 Acked Write 10 Done Done Sent false
1783 1565 Acked Write 4 Done Done Sent false
1784 1566 Acked Read 8 Done Done Sent false
1785 1567 Acked Read 5 Done Done Sent false
1786 1568 Acked Write 4 Done Done Sent false
1787 1569 Acked Write 8 Done Done Sent false
1788 1570 Acked Write 7 Done Done Sent false
1789 1571 Acked Write 5 Done Done Sent false
1790 1572 Acked Read 9 Done Done Sent false
1791 1573 Acked Write 10 Done Done Sent false
1792 1574 Acked Write 7 Done Done Sent false
1793 1575 Acked Read 5 Done Done Sent false
1794 1576 Acked Write 1 Done Done Sent false
1795 1577 Acked Write 2 Done Done Sent false
1796 1578 Acked Write 9 Done Done Sent false
1797 1579 Acked Write 7 Done Done Sent false
1798 1580 Acked Flush 0 Done Done Sent false
1799 1581 Acked Read 10 Done Done Sent false
1800 1582 Acked Write 7 Done Done Sent false
1801 1583 Acked Write 9 Done Done Sent false
1802 1584 Acked Write 8 Done Done Sent false
1803 1585 Acked Write 4 Done Done Sent false
1804 1586 Acked Read 9 Done Done Sent false
1805 1587 Acked Read 10 Done Done Sent false
1806 1588 Acked Flush 0 Done Done Sent false
1807 1589 Acked Read 1 Done Done Sent false
1808 1590 Acked Write 1 Done Done Sent false
1809 1591 Acked Write 6 Done Done Sent false
1810 1592 Acked Read 7 Done Done Sent false
1811 1593 Acked Read 3 Done Done Sent false
1812 1594 Acked Write 7 Done Done Sent false
1813 1595 Acked Read 7 Done Done Sent false
1814 1596 Acked Read 4 Done Done Sent false
1815 1597 Acked Read 8 Done Done Sent false
1816 1598 Acked Read 9 Done Done Sent false
1817 1599 Acked Read 7 Done Done Sent false
1818 1600 Acked Read 4 Done Done Sent false
1819 1601 Acked Write 3 Done Done Sent false
1820 1602 Acked Read 4 Done Done Sent false
1821 STATES DS:0 DS:1 DS:2 TOTAL
1822 Sent 0 0 301 301
1823 Done 301 301 0 602
1824 Skipped 0 0 0 0
1825 Error 0 0 0 0
1826 Last Flush: Some(JobId(1588)) Some(JobId(1588)) Some(JobId(1301))
1827 Downstairs last five completed: 1301 1300 1299 1298 1297
1828 Upstairs last five completed: 1602 1601 1600 1599 1598
1829 CLIENT: Up:0 ds:301 act:2
18302025-03-17T22:19:15.579ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
18312025-03-17T22:19:16.581ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
18322025-03-17T22:19:17.583ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
1833 ----------------------------------------------------------------
1834 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1835 States: Active Active Offline
1836 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1837 1302 Acked Read 4 Done Done Sent false
1838 1303 Acked Write 3 Done Done Sent false
1839 1304 Acked Read 4 Done Done Sent false
1840 1305 Acked Flush 0 Done Done Sent false
1841 1306 Acked Read 9 Done Done Sent false
1842 1307 Acked Write 4 Done Done Sent false
1843 1308 Acked Flush 0 Done Done Sent false
1844 1309 Acked Read 6 Done Done Sent false
1845 1310 Acked Read 9 Done Done Sent false
1846 1311 Acked Read 8 Done Done Sent false
1847 1312 Acked Read 3 Done Done Sent false
1848 1313 Acked Read 7 Done Done Sent false
1849 1314 Acked Write 6 Done Done Sent false
1850 1315 Acked Write 8 Done Done Sent false
1851 1316 Acked Read 1 Done Done Sent false
1852 1317 Acked Read 9 Done Done Sent false
1853 1318 Acked Write 3 Done Done Sent false
1854 1319 Acked Read 3 Done Done Sent false
1855 1320 Acked Read 7 Done Done Sent false
1856 1321 Acked Write 7 Done Done Sent false
1857 1322 Acked Write 10 Done Done Sent false
1858 1323 Acked Read 3 Done Done Sent false
1859 1324 Acked Read 6 Done Done Sent false
1860 1325 Acked Read 1 Done Done Sent false
1861 1326 Acked Write 3 Done Done Sent false
1862 1327 Acked Flush 0 Done Done Sent false
1863 1328 Acked Write 6 Done Done Sent false
1864 1329 Acked Read 3 Done Done Sent false
1865 1330 Acked Write 7 Done Done Sent false
1866 1331 Acked Write 6 Done Done Sent false
1867 1332 Acked Flush 0 Done Done Sent false
1868 1333 Acked Read 1 Done Done Sent false
1869 1334 Acked Read 4 Done Done Sent false
1870 1335 Acked Read 3 Done Done Sent false
1871 1336 Acked Read 9 Done Done Sent false
1872 1337 Acked Flush 0 Done Done Sent false
1873 1338 Acked Read 6 Done Done Sent false
1874 1339 Acked Read 5 Done Done Sent false
1875 1340 Acked Read 8 Done Done Sent false
1876 1341 Acked Read 5 Done Done Sent false
1877 1342 Acked Write 2 Done Done Sent false
1878 1343 Acked Write 3 Done Done Sent false
1879 1344 Acked Write 3 Done Done Sent false
1880 1345 Acked Write 10 Done Done Sent false
1881 1346 Acked Write 4 Done Done Sent false
1882 1347 Acked Read 9 Done Done Sent false
1883 1348 Acked Write 8 Done Done Sent false
1884 1349 Acked Read 4 Done Done Sent false
1885 1350 Acked Write 3 Done Done Sent false
1886 1351 Acked Write 9 Done Done Sent false
1887 1352 Acked Write 8 Done Done Sent false
1888 1353 Acked Flush 0 Done Done Sent false
1889 1354 Acked Read 9 Done Done Sent false
1890 1355 Acked Write 1 Done Done Sent false
1891 1356 Acked Write 9 Done Done Sent false
1892 1357 Acked Read 7 Done Done Sent false
1893 1358 Acked Read 5 Done Done Sent false
1894 1359 Acked Read 1 Done Done Sent false
1895 1360 Acked Read 8 Done Done Sent false
1896 1361 Acked Flush 0 Done Done Sent false
1897 1362 Acked Read 8 Done Done Sent false
1898 1363 Acked Write 2 Done Done Sent false
1899 1364 Acked Write 10 Done Done Sent false
1900 1365 Acked Read 10 Done Done Sent false
1901 1366 Acked Write 8 Done Done Sent false
1902 1367 Acked Read 8 Done Done Sent false
1903 1368 Acked Write 4 Done Done Sent false
1904 1369 Acked Flush 0 Done Done Sent false
1905 1370 Acked Read 6 Done Done Sent false
1906 1371 Acked Read 2 Done Done Sent false
1907 1372 Acked Write 4 Done Done Sent false
1908 1373 Acked Read 8 Done Done Sent false
1909 1374 Acked Read 9 Done Done Sent false
1910 1375 Acked Read 8 Done Done Sent false
1911 1376 Acked Write 7 Done Done Sent false
1912 1377 Acked Write 4 Done Done Sent false
1913 1378 Acked Write 1 Done Done Sent false
1914 1379 Acked Read 4 Done Done Sent false
1915 1380 Acked Write 4 Done Done Sent false
1916 1381 Acked Read 1 Done Done Sent false
1917 1382 Acked Read 3 Done Done Sent false
1918 1383 Acked Read 4 Done Done Sent false
1919 1384 Acked Write 4 Done Done Sent false
1920 1385 Acked Read 9 Done Done Sent false
1921 1386 Acked Read 7 Done Done Sent false
1922 1387 Acked Read 5 Done Done Sent false
1923 1388 Acked Read 10 Done Done Sent false
1924 1389 Acked Read 10 Done Done Sent false
1925 1390 Acked Read 5 Done Done Sent false
1926 1391 Acked Write 9 Done Done Sent false
1927 1392 Acked Read 8 Done Done Sent false
1928 1393 Acked Write 3 Done Done Sent false
1929 1394 Acked Write 5 Done Done Sent false
1930 1395 Acked Write 10 Done Done Sent false
1931 1396 Acked Read 6 Done Done Sent false
1932 1397 Acked Flush 0 Done Done Sent false
1933 1398 Acked Write 4 Done Done Sent false
1934 1399 Acked Write 8 Done Done Sent false
1935 1400 Acked Read 4 Done Done Sent false
1936 1401 Acked Read 1 Done Done Sent false
1937 1402 Acked Read 1 Done Done Sent false
1938 1403 Acked Write 1 Done Done Sent false
1939 1404 Acked Read 2 Done Done Sent false
1940 1405 Acked Read 4 Done Done Sent false
1941 1406 Acked Read 10 Done Done Sent false
1942 1407 Acked Read 4 Done Done Sent false
1943 1408 Acked Write 1 Done Done Sent false
1944 1409 Acked Read 4 Done Done Sent false
1945 1410 Acked Write 1 Done Done Sent false
1946 1411 Acked Flush 0 Done Done Sent false
1947 1412 Acked Write 6 Done Done Sent false
1948 1413 Acked Write 3 Done Done Sent false
1949 1414 Acked Write 8 Done Done Sent false
1950 1415 Acked Read 4 Done Done Sent false
1951 1416 Acked Flush 0 Done Done Sent false
1952 1417 Acked Write 9 Done Done Sent false
1953 1418 Acked Write 6 Done Done Sent false
1954 1419 Acked Write 7 Done Done Sent false
1955 1420 Acked Read 10 Done Done Sent false
1956 1421 Acked Read 9 Done Done Sent false
1957 1422 Acked Read 10 Done Done Sent false
1958 1423 Acked Read 6 Done Done Sent false
1959 1424 Acked Write 7 Done Done Sent false
1960 1425 Acked Write 6 Done Done Sent false
1961 1426 Acked Flush 0 Done Done Sent false
1962 1427 Acked Write 9 Done Done Sent false
1963 1428 Acked Write 5 Done Done Sent false
1964 1429 Acked Read 7 Done Done Sent false
1965 1430 Acked Read 5 Done Done Sent false
1966 1431 Acked Write 8 Done Done Sent false
1967 1432 Acked Read 9 Done Done Sent false
1968 1433 Acked Write 9 Done Done Sent false
1969 1434 Acked Write 9 Done Done Sent false
1970 1435 Acked Read 2 Done Done Sent false
1971 1436 Acked Write 4 Done Done Sent false
1972 1437 Acked Write 6 Done Done Sent false
1973 1438 Acked Read 5 Done Done Sent false
1974 1439 Acked Write 4 Done Done Sent false
1975 1440 Acked Flush 0 Done Done Sent false
1976 1441 Acked Write 8 Done Done Sent false
1977 1442 Acked Read 10 Done Done Sent false
1978 1443 Acked Read 2 Done Done Sent false
1979 1444 Acked Write 5 Done Done Sent false
1980 1445 Acked Read 4 Done Done Sent false
1981 1446 Acked Write 10 Done Done Sent false
1982 1447 Acked Read 7 Done Done Sent false
1983 1448 Acked Read 4 Done Done Sent false
1984 1449 Acked Flush 0 Done Done Sent false
1985 1450 Acked Write 7 Done Done Sent false
1986 1451 Acked Write 1 Done Done Sent false
1987 1452 Acked Write 1 Done Done Sent false
1988 1453 Acked Flush 0 Done Done Sent false
1989 1454 Acked Read 10 Done Done Sent false
1990 1455 Acked Read 6 Done Done Sent false
1991 1456 Acked Read 10 Done Done Sent false
1992 1457 Acked Write 1 Done Done Sent false
1993 1458 Acked Write 10 Done Done Sent false
1994 1459 Acked Read 5 Done Done Sent false
1995 1460 Acked Read 2 Done Done Sent false
1996 1461 Acked Write 4 Done Done Sent false
1997 1462 Acked Read 9 Done Done Sent false
1998 1463 Acked Read 8 Done Done Sent false
1999 1464 Acked Flush 0 Done Done Sent false
2000 1465 Acked Write 10 Done Done Sent false
2001 1466 Acked Write 2 Done Done Sent false
2002 1467 Acked Write 10 Done Done Sent false
2003 1468 Acked Read 3 Done Done Sent false
2004 1469 Acked Read 3 Done Done Sent false
2005 1470 Acked Write 2 Done Done Sent false
2006 1471 Acked Flush 0 Done Done Sent false
2007 1472 Acked Read 8 Done Done Sent false
2008 1473 Acked Write 1 Done Done Sent false
2009 1474 Acked Read 9 Done Done Sent false
2010 1475 Acked Read 1 Done Done Sent false
2011 1476 Acked Read 8 Done Done Sent false
2012 1477 Acked Write 1 Done Done Sent false
2013 1478 Acked Write 4 Done Done Sent false
2014 1479 Acked Write 5 Done Done Sent false
2015 1480 Acked Write 6 Done Done Sent false
2016 1481 Acked Read 6 Done Done Sent false
2017 1482 Acked Write 7 Done Done Sent false
2018 1483 Acked Write 3 Done Done Sent false
2019 1484 Acked Write 8 Done Done Sent false
2020 1485 Acked Write 4 Done Done Sent false
2021 1486 Acked Read 5 Done Done Sent false
2022 1487 Acked Flush 0 Done Done Sent false
2023 1488 Acked Write 9 Done Done Sent false
2024 1489 Acked Flush 0 Done Done Sent false
2025 1490 Acked Read 2 Done Done Sent false
2026 1491 Acked Read 1 Done Done Sent false
2027 1492 Acked Read 4 Done Done Sent false
2028 1493 Acked Read 3 Done Done Sent false
2029 1494 Acked Read 3 Done Done Sent false
2030 1495 Acked Read 6 Done Done Sent false
2031 1496 Acked Write 6 Done Done Sent false
2032 1497 Acked Flush 0 Done Done Sent false
2033 1498 Acked Write 4 Done Done Sent false
2034 1499 Acked Read 1 Done Done Sent false
2035 1500 Acked Write 9 Done Done Sent false
2036 1501 Acked Write 4 Done Done Sent false
2037 1502 Acked Read 1 Done Done Sent false
2038 1503 Acked Write 9 Done Done Sent false
2039 1504 Acked Write 6 Done Done Sent false
2040 1505 Acked Read 7 Done Done Sent false
2041 1506 Acked Write 9 Done Done Sent false
2042 1507 Acked Write 9 Done Done Sent false
2043 1508 Acked Write 7 Done Done Sent false
2044 1509 Acked Write 3 Done Done Sent false
2045 1510 Acked Read 7 Done Done Sent false
2046 1511 Acked Write 8 Done Done Sent false
2047 1512 Acked Read 1 Done Done Sent false
2048 1513 Acked Read 7 Done Done Sent false
2049 1514 Acked Read 2 Done Done Sent false
2050 1515 Acked Flush 0 Done Done Sent false
2051 1516 Acked Write 10 Done Done Sent false
2052 1517 Acked Read 2 Done Done Sent false
2053 1518 Acked Read 4 Done Done Sent false
2054 1519 Acked Flush 0 Done Done Sent false
2055 1520 Acked Write 9 Done Done Sent false
2056 1521 Acked Write 4 Done Done Sent false
2057 1522 Acked Read 6 Done Done Sent false
2058 1523 Acked Read 7 Done Done Sent false
2059 1524 Acked Write 1 Done Done Sent false
2060 1525 Acked Read 1 Done Done Sent false
2061 1526 Acked Read 4 Done Done Sent false
2062 1527 Acked Read 2 Done Done Sent false
2063 1528 Acked Write 9 Done Done Sent false
2064 1529 Acked Read 1 Done Done Sent false
2065 1530 Acked Write 5 Done Done Sent false
2066 1531 Acked Write 2 Done Done Sent false
2067 1532 Acked Read 5 Done Done Sent false
2068 1533 Acked Write 6 Done Done Sent false
2069 1534 Acked Read 8 Done Done Sent false
2070 1535 Acked Write 9 Done Done Sent false
2071 1536 Acked Read 2 Done Done Sent false
2072 1537 Acked Read 6 Done Done Sent false
2073 1538 Acked Flush 0 Done Done Sent false
2074 1539 Acked Write 2 Done Done Sent false
2075 1540 Acked Read 1 Done Done Sent false
2076 1541 Acked Write 2 Done Done Sent false
2077 1542 Acked Read 5 Done Done Sent false
2078 1543 Acked Write 4 Done Done Sent false
2079 1544 Acked Read 8 Done Done Sent false
2080 1545 Acked Write 1 Done Done Sent false
2081 1546 Acked Write 8 Done Done Sent false
2082 1547 Acked Write 8 Done Done Sent false
2083 1548 Acked Read 8 Done Done Sent false
2084 1549 Acked Write 1 Done Done Sent false
2085 1550 Acked Read 4 Done Done Sent false
2086 1551 Acked Read 8 Done Done Sent false
2087 1552 Acked Read 1 Done Done Sent false
2088 1553 Acked Read 5 Done Done Sent false
2089 1554 Acked Read 10 Done Done Sent false
2090 1555 Acked Write 5 Done Done Sent false
2091 1556 Acked Read 3 Done Done Sent false
2092 1557 Acked Read 6 Done Done Sent false
2093 1558 Acked Read 2 Done Done Sent false
2094 1559 Acked Write 9 Done Done Sent false
2095 1560 Acked Write 6 Done Done Sent false
2096 1561 Acked Write 1 Done Done Sent false
2097 1562 Acked Flush 0 Done Done Sent false
2098 1563 Acked Write 1 Done Done Sent false
2099 1564 Acked Write 10 Done Done Sent false
2100 1565 Acked Write 4 Done Done Sent false
2101 1566 Acked Read 8 Done Done Sent false
2102 1567 Acked Read 5 Done Done Sent false
2103 1568 Acked Write 4 Done Done Sent false
2104 1569 Acked Write 8 Done Done Sent false
2105 1570 Acked Write 7 Done Done Sent false
2106 1571 Acked Write 5 Done Done Sent false
2107 1572 Acked Read 9 Done Done Sent false
2108 1573 Acked Write 10 Done Done Sent false
2109 1574 Acked Write 7 Done Done Sent false
2110 1575 Acked Read 5 Done Done Sent false
2111 1576 Acked Write 1 Done Done Sent false
2112 1577 Acked Write 2 Done Done Sent false
2113 1578 Acked Write 9 Done Done Sent false
2114 1579 Acked Write 7 Done Done Sent false
2115 1580 Acked Flush 0 Done Done Sent false
2116 1581 Acked Read 10 Done Done Sent false
2117 1582 Acked Write 7 Done Done Sent false
2118 1583 Acked Write 9 Done Done Sent false
2119 1584 Acked Write 8 Done Done Sent false
2120 1585 Acked Write 4 Done Done Sent false
2121 1586 Acked Read 9 Done Done Sent false
2122 1587 Acked Read 10 Done Done Sent false
2123 1588 Acked Flush 0 Done Done Sent false
2124 1589 Acked Read 1 Done Done Sent false
2125 1590 Acked Write 1 Done Done Sent false
2126 1591 Acked Write 6 Done Done Sent false
2127 1592 Acked Read 7 Done Done Sent false
2128 1593 Acked Read 3 Done Done Sent false
2129 1594 Acked Write 7 Done Done Sent false
2130 1595 Acked Read 7 Done Done Sent false
2131 1596 Acked Read 4 Done Done Sent false
2132 1597 Acked Read 8 Done Done Sent false
2133 1598 Acked Read 9 Done Done Sent false
2134 1599 Acked Read 7 Done Done Sent false
2135 1600 Acked Read 4 Done Done Sent false
2136 1601 Acked Write 3 Done Done Sent false
2137 1602 Acked Read 4 Done Done Sent false
2138 STATES DS:0 DS:1 DS:2 TOTAL
2139 Sent 0 0 301 301
2140 Done 301 301 0 602
2141 Skipped 0 0 0 0
2142 Error 0 0 0 0
2143 Last Flush: Some(JobId(1588)) Some(JobId(1588)) Some(JobId(1301))
2144 Downstairs last five completed: 1301 1300 1299 1298 1297
2145 Upstairs last five completed: 1602 1601 1600 1599 1598
2146 CLIENT: Up:0 ds:301 act:2
21472025-03-17T22:19:19.587ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
21482025-03-17T22:19:20.347ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
21492025-03-17T22:19:20.347ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
21502025-03-17T22:19:20.347ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
21512025-03-17T22:19:20.347ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
21522025-03-17T22:19:20.347ZINFOcrucible: client task is exiting = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
2153 [8830] got start action
2154 [8830] I am going to restart
2155 Starting downstairs at port 8830
2156 Make output file at "/tmp/dsc/downstairs-8830.txt"
2157 [8830][2] reports Starting
2158 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1213)
2159 [8830][2] reports Running
21602025-03-17T22:19:20.849ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
21612025-03-17T22:19:21.591ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
2162 ----------------------------------------------------------------
2163 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2164 States: Active Active Offline
2165 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2166 1302 Acked Read 4 Done Done Sent false
2167 1303 Acked Write 3 Done Done Sent false
2168 1304 Acked Read 4 Done Done Sent false
2169 1305 Acked Flush 0 Done Done Sent false
2170 1306 Acked Read 9 Done Done Sent false
2171 1307 Acked Write 4 Done Done Sent false
2172 1308 Acked Flush 0 Done Done Sent false
2173 1309 Acked Read 6 Done Done Sent false
2174 1310 Acked Read 9 Done Done Sent false
2175 1311 Acked Read 8 Done Done Sent false
2176 1312 Acked Read 3 Done Done Sent false
2177 1313 Acked Read 7 Done Done Sent false
2178 1314 Acked Write 6 Done Done Sent false
2179 1315 Acked Write 8 Done Done Sent false
2180 1316 Acked Read 1 Done Done Sent false
2181 1317 Acked Read 9 Done Done Sent false
2182 1318 Acked Write 3 Done Done Sent false
2183 1319 Acked Read 3 Done Done Sent false
2184 1320 Acked Read 7 Done Done Sent false
2185 1321 Acked Write 7 Done Done Sent false
2186 1322 Acked Write 10 Done Done Sent false
2187 1323 Acked Read 3 Done Done Sent false
2188 1324 Acked Read 6 Done Done Sent false
2189 1325 Acked Read 1 Done Done Sent false
2190 1326 Acked Write 3 Done Done Sent false
2191 1327 Acked Flush 0 Done Done Sent false
2192 1328 Acked Write 6 Done Done Sent false
2193 1329 Acked Read 3 Done Done Sent false
2194 1330 Acked Write 7 Done Done Sent false
2195 1331 Acked Write 6 Done Done Sent false
2196 1332 Acked Flush 0 Done Done Sent false
2197 1333 Acked Read 1 Done Done Sent false
2198 1334 Acked Read 4 Done Done Sent false
2199 1335 Acked Read 3 Done Done Sent false
2200 1336 Acked Read 9 Done Done Sent false
2201 1337 Acked Flush 0 Done Done Sent false
2202 1338 Acked Read 6 Done Done Sent false
2203 1339 Acked Read 5 Done Done Sent false
2204 1340 Acked Read 8 Done Done Sent false
2205 1341 Acked Read 5 Done Done Sent false
2206 1342 Acked Write 2 Done Done Sent false
2207 1343 Acked Write 3 Done Done Sent false
2208 1344 Acked Write 3 Done Done Sent false
2209 1345 Acked Write 10 Done Done Sent false
2210 1346 Acked Write 4 Done Done Sent false
2211 1347 Acked Read 9 Done Done Sent false
2212 1348 Acked Write 8 Done Done Sent false
2213 1349 Acked Read 4 Done Done Sent false
2214 1350 Acked Write 3 Done Done Sent false
2215 1351 Acked Write 9 Done Done Sent false
2216 1352 Acked Write 8 Done Done Sent false
2217 1353 Acked Flush 0 Done Done Sent false
2218 1354 Acked Read 9 Done Done Sent false
2219 1355 Acked Write 1 Done Done Sent false
2220 1356 Acked Write 9 Done Done Sent false
2221 1357 Acked Read 7 Done Done Sent false
2222 1358 Acked Read 5 Done Done Sent false
2223 1359 Acked Read 1 Done Done Sent false
2224 1360 Acked Read 8 Done Done Sent false
2225 1361 Acked Flush 0 Done Done Sent false
2226 1362 Acked Read 8 Done Done Sent false
2227 1363 Acked Write 2 Done Done Sent false
2228 1364 Acked Write 10 Done Done Sent false
2229 1365 Acked Read 10 Done Done Sent false
2230 1366 Acked Write 8 Done Done Sent false
2231 1367 Acked Read 8 Done Done Sent false
2232 1368 Acked Write 4 Done Done Sent false
2233 1369 Acked Flush 0 Done Done Sent false
2234 1370 Acked Read 6 Done Done Sent false
2235 1371 Acked Read 2 Done Done Sent false
2236 1372 Acked Write 4 Done Done Sent false
2237 1373 Acked Read 8 Done Done Sent false
2238 1374 Acked Read 9 Done Done Sent false
2239 1375 Acked Read 8 Done Done Sent false
2240 1376 Acked Write 7 Done Done Sent false
2241 1377 Acked Write 4 Done Done Sent false
2242 1378 Acked Write 1 Done Done Sent false
2243 1379 Acked Read 4 Done Done Sent false
2244 1380 Acked Write 4 Done Done Sent false
2245 1381 Acked Read 1 Done Done Sent false
2246 1382 Acked Read 3 Done Done Sent false
2247 1383 Acked Read 4 Done Done Sent false
2248 1384 Acked Write 4 Done Done Sent false
2249 1385 Acked Read 9 Done Done Sent false
2250 1386 Acked Read 7 Done Done Sent false
2251 1387 Acked Read 5 Done Done Sent false
2252 1388 Acked Read 10 Done Done Sent false
2253 1389 Acked Read 10 Done Done Sent false
2254 1390 Acked Read 5 Done Done Sent false
2255 1391 Acked Write 9 Done Done Sent false
2256 1392 Acked Read 8 Done Done Sent false
2257 1393 Acked Write 3 Done Done Sent false
2258 1394 Acked Write 5 Done Done Sent false
2259 1395 Acked Write 10 Done Done Sent false
2260 1396 Acked Read 6 Done Done Sent false
2261 1397 Acked Flush 0 Done Done Sent false
2262 1398 Acked Write 4 Done Done Sent false
2263 1399 Acked Write 8 Done Done Sent false
2264 1400 Acked Read 4 Done Done Sent false
2265 1401 Acked Read 1 Done Done Sent false
2266 1402 Acked Read 1 Done Done Sent false
2267 1403 Acked Write 1 Done Done Sent false
2268 1404 Acked Read 2 Done Done Sent false
2269 1405 Acked Read 4 Done Done Sent false
2270 1406 Acked Read 10 Done Done Sent false
2271 1407 Acked Read 4 Done Done Sent false
2272 1408 Acked Write 1 Done Done Sent false
2273 1409 Acked Read 4 Done Done Sent false
2274 1410 Acked Write 1 Done Done Sent false
2275 1411 Acked Flush 0 Done Done Sent false
2276 1412 Acked Write 6 Done Done Sent false
2277 1413 Acked Write 3 Done Done Sent false
2278 1414 Acked Write 8 Done Done Sent false
2279 1415 Acked Read 4 Done Done Sent false
2280 1416 Acked Flush 0 Done Done Sent false
2281 1417 Acked Write 9 Done Done Sent false
2282 1418 Acked Write 6 Done Done Sent false
2283 1419 Acked Write 7 Done Done Sent false
2284 1420 Acked Read 10 Done Done Sent false
2285 1421 Acked Read 9 Done Done Sent false
2286 1422 Acked Read 10 Done Done Sent false
2287 1423 Acked Read 6 Done Done Sent false
2288 1424 Acked Write 7 Done Done Sent false
2289 1425 Acked Write 6 Done Done Sent false
2290 1426 Acked Flush 0 Done Done Sent false
2291 1427 Acked Write 9 Done Done Sent false
2292 1428 Acked Write 5 Done Done Sent false
2293 1429 Acked Read 7 Done Done Sent false
2294 1430 Acked Read 5 Done Done Sent false
2295 1431 Acked Write 8 Done Done Sent false
2296 1432 Acked Read 9 Done Done Sent false
2297 1433 Acked Write 9 Done Done Sent false
2298 1434 Acked Write 9 Done Done Sent false
2299 1435 Acked Read 2 Done Done Sent false
2300 1436 Acked Write 4 Done Done Sent false
2301 1437 Acked Write 6 Done Done Sent false
2302 1438 Acked Read 5 Done Done Sent false
2303 1439 Acked Write 4 Done Done Sent false
2304 1440 Acked Flush 0 Done Done Sent false
2305 1441 Acked Write 8 Done Done Sent false
2306 1442 Acked Read 10 Done Done Sent false
2307 1443 Acked Read 2 Done Done Sent false
2308 1444 Acked Write 5 Done Done Sent false
2309 1445 Acked Read 4 Done Done Sent false
2310 1446 Acked Write 10 Done Done Sent false
2311 1447 Acked Read 7 Done Done Sent false
2312 1448 Acked Read 4 Done Done Sent false
2313 1449 Acked Flush 0 Done Done Sent false
2314 1450 Acked Write 7 Done Done Sent false
2315 1451 Acked Write 1 Done Done Sent false
2316 1452 Acked Write 1 Done Done Sent false
2317 1453 Acked Flush 0 Done Done Sent false
2318 1454 Acked Read 10 Done Done Sent false
2319 1455 Acked Read 6 Done Done Sent false
2320 1456 Acked Read 10 Done Done Sent false
2321 1457 Acked Write 1 Done Done Sent false
2322 1458 Acked Write 10 Done Done Sent false
2323 1459 Acked Read 5 Done Done Sent false
2324 1460 Acked Read 2 Done Done Sent false
2325 1461 Acked Write 4 Done Done Sent false
2326 1462 Acked Read 9 Done Done Sent false
2327 1463 Acked Read 8 Done Done Sent false
2328 1464 Acked Flush 0 Done Done Sent false
2329 1465 Acked Write 10 Done Done Sent false
2330 1466 Acked Write 2 Done Done Sent false
2331 1467 Acked Write 10 Done Done Sent false
2332 1468 Acked Read 3 Done Done Sent false
2333 1469 Acked Read 3 Done Done Sent false
2334 1470 Acked Write 2 Done Done Sent false
2335 1471 Acked Flush 0 Done Done Sent false
2336 1472 Acked Read 8 Done Done Sent false
2337 1473 Acked Write 1 Done Done Sent false
2338 1474 Acked Read 9 Done Done Sent false
2339 1475 Acked Read 1 Done Done Sent false
2340 1476 Acked Read 8 Done Done Sent false
2341 1477 Acked Write 1 Done Done Sent false
2342 1478 Acked Write 4 Done Done Sent false
2343 1479 Acked Write 5 Done Done Sent false
2344 1480 Acked Write 6 Done Done Sent false
2345 1481 Acked Read 6 Done Done Sent false
2346 1482 Acked Write 7 Done Done Sent false
2347 1483 Acked Write 3 Done Done Sent false
2348 1484 Acked Write 8 Done Done Sent false
2349 1485 Acked Write 4 Done Done Sent false
2350 1486 Acked Read 5 Done Done Sent false
2351 1487 Acked Flush 0 Done Done Sent false
2352 1488 Acked Write 9 Done Done Sent false
2353 1489 Acked Flush 0 Done Done Sent false
2354 1490 Acked Read 2 Done Done Sent false
2355 1491 Acked Read 1 Done Done Sent false
2356 1492 Acked Read 4 Done Done Sent false
2357 1493 Acked Read 3 Done Done Sent false
2358 1494 Acked Read 3 Done Done Sent false
2359 1495 Acked Read 6 Done Done Sent false
2360 1496 Acked Write 6 Done Done Sent false
2361 1497 Acked Flush 0 Done Done Sent false
2362 1498 Acked Write 4 Done Done Sent false
2363 1499 Acked Read 1 Done Done Sent false
2364 1500 Acked Write 9 Done Done Sent false
2365 1501 Acked Write 4 Done Done Sent false
2366 1502 Acked Read 1 Done Done Sent false
2367 1503 Acked Write 9 Done Done Sent false
2368 1504 Acked Write 6 Done Done Sent false
2369 1505 Acked Read 7 Done Done Sent false
2370 1506 Acked Write 9 Done Done Sent false
2371 1507 Acked Write 9 Done Done Sent false
2372 1508 Acked Write 7 Done Done Sent false
2373 1509 Acked Write 3 Done Done Sent false
2374 1510 Acked Read 7 Done Done Sent false
2375 1511 Acked Write 8 Done Done Sent false
2376 1512 Acked Read 1 Done Done Sent false
2377 1513 Acked Read 7 Done Done Sent false
2378 1514 Acked Read 2 Done Done Sent false
2379 1515 Acked Flush 0 Done Done Sent false
2380 1516 Acked Write 10 Done Done Sent false
2381 1517 Acked Read 2 Done Done Sent false
2382 1518 Acked Read 4 Done Done Sent false
2383 1519 Acked Flush 0 Done Done Sent false
2384 1520 Acked Write 9 Done Done Sent false
2385 1521 Acked Write 4 Done Done Sent false
2386 1522 Acked Read 6 Done Done Sent false
2387 1523 Acked Read 7 Done Done Sent false
2388 1524 Acked Write 1 Done Done Sent false
2389 1525 Acked Read 1 Done Done Sent false
2390 1526 Acked Read 4 Done Done Sent false
2391 1527 Acked Read 2 Done Done Sent false
2392 1528 Acked Write 9 Done Done Sent false
2393 1529 Acked Read 1 Done Done Sent false
2394 1530 Acked Write 5 Done Done Sent false
2395 1531 Acked Write 2 Done Done Sent false
2396 1532 Acked Read 5 Done Done Sent false
2397 1533 Acked Write 6 Done Done Sent false
2398 1534 Acked Read 8 Done Done Sent false
2399 1535 Acked Write 9 Done Done Sent false
2400 1536 Acked Read 2 Done Done Sent false
2401 1537 Acked Read 6 Done Done Sent false
2402 1538 Acked Flush 0 Done Done Sent false
2403 1539 Acked Write 2 Done Done Sent false
2404 1540 Acked Read 1 Done Done Sent false
2405 1541 Acked Write 2 Done Done Sent false
2406 1542 Acked Read 5 Done Done Sent false
2407 1543 Acked Write 4 Done Done Sent false
2408 1544 Acked Read 8 Done Done Sent false
2409 1545 Acked Write 1 Done Done Sent false
2410 1546 Acked Write 8 Done Done Sent false
2411 1547 Acked Write 8 Done Done Sent false
2412 1548 Acked Read 8 Done Done Sent false
2413 1549 Acked Write 1 Done Done Sent false
2414 1550 Acked Read 4 Done Done Sent false
2415 1551 Acked Read 8 Done Done Sent false
2416 1552 Acked Read 1 Done Done Sent false
2417 1553 Acked Read 5 Done Done Sent false
2418 1554 Acked Read 10 Done Done Sent false
2419 1555 Acked Write 5 Done Done Sent false
2420 1556 Acked Read 3 Done Done Sent false
2421 1557 Acked Read 6 Done Done Sent false
2422 1558 Acked Read 2 Done Done Sent false
2423 1559 Acked Write 9 Done Done Sent false
2424 1560 Acked Write 6 Done Done Sent false
2425 1561 Acked Write 1 Done Done Sent false
2426 1562 Acked Flush 0 Done Done Sent false
2427 1563 Acked Write 1 Done Done Sent false
2428 1564 Acked Write 10 Done Done Sent false
2429 1565 Acked Write 4 Done Done Sent false
2430 1566 Acked Read 8 Done Done Sent false
2431 1567 Acked Read 5 Done Done Sent false
2432 1568 Acked Write 4 Done Done Sent false
2433 1569 Acked Write 8 Done Done Sent false
2434 1570 Acked Write 7 Done Done Sent false
2435 1571 Acked Write 5 Done Done Sent false
2436 1572 Acked Read 9 Done Done Sent false
2437 1573 Acked Write 10 Done Done Sent false
2438 1574 Acked Write 7 Done Done Sent false
2439 1575 Acked Read 5 Done Done Sent false
2440 1576 Acked Write 1 Done Done Sent false
2441 1577 Acked Write 2 Done Done Sent false
2442 1578 Acked Write 9 Done Done Sent false
2443 1579 Acked Write 7 Done Done Sent false
2444 1580 Acked Flush 0 Done Done Sent false
2445 1581 Acked Read 10 Done Done Sent false
2446 1582 Acked Write 7 Done Done Sent false
2447 1583 Acked Write 9 Done Done Sent false
2448 1584 Acked Write 8 Done Done Sent false
2449 1585 Acked Write 4 Done Done Sent false
2450 1586 Acked Read 9 Done Done Sent false
2451 1587 Acked Read 10 Done Done Sent false
2452 1588 Acked Flush 0 Done Done Sent false
2453 1589 Acked Read 1 Done Done Sent false
2454 1590 Acked Write 1 Done Done Sent false
2455 1591 Acked Write 6 Done Done Sent false
2456 1592 Acked Read 7 Done Done Sent false
2457 1593 Acked Read 3 Done Done Sent false
2458 1594 Acked Write 7 Done Done Sent false
2459 1595 Acked Read 7 Done Done Sent false
2460 1596 Acked Read 4 Done Done Sent false
2461 1597 Acked Read 8 Done Done Sent false
2462 1598 Acked Read 9 Done Done Sent false
2463 1599 Acked Read 7 Done Done Sent false
2464 1600 Acked Read 4 Done Done Sent false
2465 1601 Acked Write 3 Done Done Sent false
2466 1602 Acked Read 4 Done Done Sent false
2467 STATES DS:0 DS:1 DS:2 TOTAL
2468 Sent 0 0 301 301
2469 Done 301 301 0 602
2470 Skipped 0 0 0 0
2471 Error 0 0 0 0
2472 Last Flush: Some(JobId(1588)) Some(JobId(1588)) Some(JobId(1301))
2473 Downstairs last five completed: 1301 1300 1299 1298 1297
2474 Upstairs last five completed: 1602 1601 1600 1599 1598
2475 CLIENT: Up:0 ds:301 act:2
24762025-03-17T22:19:23.596ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
24772025-03-17T22:19:24.594ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
24782025-03-17T22:19:25.596ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
2479 ----------------------------------------------------------------
2480 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2481 States: Active Active Offline
2482 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2483 1302 Acked Read 4 Done Done Sent false
2484 1303 Acked Write 3 Done Done Sent false
2485 1304 Acked Read 4 Done Done Sent false
2486 1305 Acked Flush 0 Done Done Sent false
2487 1306 Acked Read 9 Done Done Sent false
2488 1307 Acked Write 4 Done Done Sent false
2489 1308 Acked Flush 0 Done Done Sent false
2490 1309 Acked Read 6 Done Done Sent false
2491 1310 Acked Read 9 Done Done Sent false
2492 1311 Acked Read 8 Done Done Sent false
2493 1312 Acked Read 3 Done Done Sent false
2494 1313 Acked Read 7 Done Done Sent false
2495 1314 Acked Write 6 Done Done Sent false
2496 1315 Acked Write 8 Done Done Sent false
2497 1316 Acked Read 1 Done Done Sent false
2498 1317 Acked Read 9 Done Done Sent false
2499 1318 Acked Write 3 Done Done Sent false
2500 1319 Acked Read 3 Done Done Sent false
2501 1320 Acked Read 7 Done Done Sent false
2502 1321 Acked Write 7 Done Done Sent false
2503 1322 Acked Write 10 Done Done Sent false
2504 1323 Acked Read 3 Done Done Sent false
2505 1324 Acked Read 6 Done Done Sent false
2506 1325 Acked Read 1 Done Done Sent false
2507 1326 Acked Write 3 Done Done Sent false
2508 1327 Acked Flush 0 Done Done Sent false
2509 1328 Acked Write 6 Done Done Sent false
2510 1329 Acked Read 3 Done Done Sent false
2511 1330 Acked Write 7 Done Done Sent false
2512 1331 Acked Write 6 Done Done Sent false
2513 1332 Acked Flush 0 Done Done Sent false
2514 1333 Acked Read 1 Done Done Sent false
2515 1334 Acked Read 4 Done Done Sent false
2516 1335 Acked Read 3 Done Done Sent false
2517 1336 Acked Read 9 Done Done Sent false
2518 1337 Acked Flush 0 Done Done Sent false
2519 1338 Acked Read 6 Done Done Sent false
2520 1339 Acked Read 5 Done Done Sent false
2521 1340 Acked Read 8 Done Done Sent false
2522 1341 Acked Read 5 Done Done Sent false
2523 1342 Acked Write 2 Done Done Sent false
2524 1343 Acked Write 3 Done Done Sent false
2525 1344 Acked Write 3 Done Done Sent false
2526 1345 Acked Write 10 Done Done Sent false
2527 1346 Acked Write 4 Done Done Sent false
2528 1347 Acked Read 9 Done Done Sent false
2529 1348 Acked Write 8 Done Done Sent false
2530 1349 Acked Read 4 Done Done Sent false
2531 1350 Acked Write 3 Done Done Sent false
2532 1351 Acked Write 9 Done Done Sent false
2533 1352 Acked Write 8 Done Done Sent false
2534 1353 Acked Flush 0 Done Done Sent false
2535 1354 Acked Read 9 Done Done Sent false
2536 1355 Acked Write 1 Done Done Sent false
2537 1356 Acked Write 9 Done Done Sent false
2538 1357 Acked Read 7 Done Done Sent false
2539 1358 Acked Read 5 Done Done Sent false
2540 1359 Acked Read 1 Done Done Sent false
2541 1360 Acked Read 8 Done Done Sent false
2542 1361 Acked Flush 0 Done Done Sent false
2543 1362 Acked Read 8 Done Done Sent false
2544 1363 Acked Write 2 Done Done Sent false
2545 1364 Acked Write 10 Done Done Sent false
2546 1365 Acked Read 10 Done Done Sent false
2547 1366 Acked Write 8 Done Done Sent false
2548 1367 Acked Read 8 Done Done Sent false
2549 1368 Acked Write 4 Done Done Sent false
2550 1369 Acked Flush 0 Done Done Sent false
2551 1370 Acked Read 6 Done Done Sent false
2552 1371 Acked Read 2 Done Done Sent false
2553 1372 Acked Write 4 Done Done Sent false
2554 1373 Acked Read 8 Done Done Sent false
2555 1374 Acked Read 9 Done Done Sent false
2556 1375 Acked Read 8 Done Done Sent false
2557 1376 Acked Write 7 Done Done Sent false
2558 1377 Acked Write 4 Done Done Sent false
2559 1378 Acked Write 1 Done Done Sent false
2560 1379 Acked Read 4 Done Done Sent false
2561 1380 Acked Write 4 Done Done Sent false
2562 1381 Acked Read 1 Done Done Sent false
2563 1382 Acked Read 3 Done Done Sent false
2564 1383 Acked Read 4 Done Done Sent false
2565 1384 Acked Write 4 Done Done Sent false
2566 1385 Acked Read 9 Done Done Sent false
2567 1386 Acked Read 7 Done Done Sent false
2568 1387 Acked Read 5 Done Done Sent false
2569 1388 Acked Read 10 Done Done Sent false
2570 1389 Acked Read 10 Done Done Sent false
2571 1390 Acked Read 5 Done Done Sent false
2572 1391 Acked Write 9 Done Done Sent false
2573 1392 Acked Read 8 Done Done Sent false
2574 1393 Acked Write 3 Done Done Sent false
2575 1394 Acked Write 5 Done Done Sent false
2576 1395 Acked Write 10 Done Done Sent false
2577 1396 Acked Read 6 Done Done Sent false
2578 1397 Acked Flush 0 Done Done Sent false
2579 1398 Acked Write 4 Done Done Sent false
2580 1399 Acked Write 8 Done Done Sent false
2581 1400 Acked Read 4 Done Done Sent false
2582 1401 Acked Read 1 Done Done Sent false
2583 1402 Acked Read 1 Done Done Sent false
2584 1403 Acked Write 1 Done Done Sent false
2585 1404 Acked Read 2 Done Done Sent false
2586 1405 Acked Read 4 Done Done Sent false
2587 1406 Acked Read 10 Done Done Sent false
2588 1407 Acked Read 4 Done Done Sent false
2589 1408 Acked Write 1 Done Done Sent false
2590 1409 Acked Read 4 Done Done Sent false
2591 1410 Acked Write 1 Done Done Sent false
2592 1411 Acked Flush 0 Done Done Sent false
2593 1412 Acked Write 6 Done Done Sent false
2594 1413 Acked Write 3 Done Done Sent false
2595 1414 Acked Write 8 Done Done Sent false
2596 1415 Acked Read 4 Done Done Sent false
2597 1416 Acked Flush 0 Done Done Sent false
2598 1417 Acked Write 9 Done Done Sent false
2599 1418 Acked Write 6 Done Done Sent false
2600 1419 Acked Write 7 Done Done Sent false
2601 1420 Acked Read 10 Done Done Sent false
2602 1421 Acked Read 9 Done Done Sent false
2603 1422 Acked Read 10 Done Done Sent false
2604 1423 Acked Read 6 Done Done Sent false
2605 1424 Acked Write 7 Done Done Sent false
2606 1425 Acked Write 6 Done Done Sent false
2607 1426 Acked Flush 0 Done Done Sent false
2608 1427 Acked Write 9 Done Done Sent false
2609 1428 Acked Write 5 Done Done Sent false
2610 1429 Acked Read 7 Done Done Sent false
2611 1430 Acked Read 5 Done Done Sent false
2612 1431 Acked Write 8 Done Done Sent false
2613 1432 Acked Read 9 Done Done Sent false
2614 1433 Acked Write 9 Done Done Sent false
2615 1434 Acked Write 9 Done Done Sent false
2616 1435 Acked Read 2 Done Done Sent false
2617 1436 Acked Write 4 Done Done Sent false
2618 1437 Acked Write 6 Done Done Sent false
2619 1438 Acked Read 5 Done Done Sent false
2620 1439 Acked Write 4 Done Done Sent false
2621 1440 Acked Flush 0 Done Done Sent false
2622 1441 Acked Write 8 Done Done Sent false
2623 1442 Acked Read 10 Done Done Sent false
2624 1443 Acked Read 2 Done Done Sent false
2625 1444 Acked Write 5 Done Done Sent false
2626 1445 Acked Read 4 Done Done Sent false
2627 1446 Acked Write 10 Done Done Sent false
2628 1447 Acked Read 7 Done Done Sent false
2629 1448 Acked Read 4 Done Done Sent false
2630 1449 Acked Flush 0 Done Done Sent false
2631 1450 Acked Write 7 Done Done Sent false
2632 1451 Acked Write 1 Done Done Sent false
2633 1452 Acked Write 1 Done Done Sent false
2634 1453 Acked Flush 0 Done Done Sent false
2635 1454 Acked Read 10 Done Done Sent false
2636 1455 Acked Read 6 Done Done Sent false
2637 1456 Acked Read 10 Done Done Sent false
2638 1457 Acked Write 1 Done Done Sent false
2639 1458 Acked Write 10 Done Done Sent false
2640 1459 Acked Read 5 Done Done Sent false
2641 1460 Acked Read 2 Done Done Sent false
2642 1461 Acked Write 4 Done Done Sent false
2643 1462 Acked Read 9 Done Done Sent false
2644 1463 Acked Read 8 Done Done Sent false
2645 1464 Acked Flush 0 Done Done Sent false
2646 1465 Acked Write 10 Done Done Sent false
2647 1466 Acked Write 2 Done Done Sent false
2648 1467 Acked Write 10 Done Done Sent false
2649 1468 Acked Read 3 Done Done Sent false
2650 1469 Acked Read 3 Done Done Sent false
2651 1470 Acked Write 2 Done Done Sent false
2652 1471 Acked Flush 0 Done Done Sent false
2653 1472 Acked Read 8 Done Done Sent false
2654 1473 Acked Write 1 Done Done Sent false
2655 1474 Acked Read 9 Done Done Sent false
2656 1475 Acked Read 1 Done Done Sent false
2657 1476 Acked Read 8 Done Done Sent false
2658 1477 Acked Write 1 Done Done Sent false
2659 1478 Acked Write 4 Done Done Sent false
2660 1479 Acked Write 5 Done Done Sent false
2661 1480 Acked Write 6 Done Done Sent false
2662 1481 Acked Read 6 Done Done Sent false
2663 1482 Acked Write 7 Done Done Sent false
2664 1483 Acked Write 3 Done Done Sent false
2665 1484 Acked Write 8 Done Done Sent false
2666 1485 Acked Write 4 Done Done Sent false
2667 1486 Acked Read 5 Done Done Sent false
2668 1487 Acked Flush 0 Done Done Sent false
2669 1488 Acked Write 9 Done Done Sent false
2670 1489 Acked Flush 0 Done Done Sent false
2671 1490 Acked Read 2 Done Done Sent false
2672 1491 Acked Read 1 Done Done Sent false
2673 1492 Acked Read 4 Done Done Sent false
2674 1493 Acked Read 3 Done Done Sent false
2675 1494 Acked Read 3 Done Done Sent false
2676 1495 Acked Read 6 Done Done Sent false
2677 1496 Acked Write 6 Done Done Sent false
2678 1497 Acked Flush 0 Done Done Sent false
2679 1498 Acked Write 4 Done Done Sent false
2680 1499 Acked Read 1 Done Done Sent false
2681 1500 Acked Write 9 Done Done Sent false
2682 1501 Acked Write 4 Done Done Sent false
2683 1502 Acked Read 1 Done Done Sent false
2684 1503 Acked Write 9 Done Done Sent false
2685 1504 Acked Write 6 Done Done Sent false
2686 1505 Acked Read 7 Done Done Sent false
2687 1506 Acked Write 9 Done Done Sent false
2688 1507 Acked Write 9 Done Done Sent false
2689 1508 Acked Write 7 Done Done Sent false
2690 1509 Acked Write 3 Done Done Sent false
2691 1510 Acked Read 7 Done Done Sent false
2692 1511 Acked Write 8 Done Done Sent false
2693 1512 Acked Read 1 Done Done Sent false
2694 1513 Acked Read 7 Done Done Sent false
2695 1514 Acked Read 2 Done Done Sent false
2696 1515 Acked Flush 0 Done Done Sent false
2697 1516 Acked Write 10 Done Done Sent false
2698 1517 Acked Read 2 Done Done Sent false
2699 1518 Acked Read 4 Done Done Sent false
2700 1519 Acked Flush 0 Done Done Sent false
2701 1520 Acked Write 9 Done Done Sent false
2702 1521 Acked Write 4 Done Done Sent false
2703 1522 Acked Read 6 Done Done Sent false
2704 1523 Acked Read 7 Done Done Sent false
2705 1524 Acked Write 1 Done Done Sent false
2706 1525 Acked Read 1 Done Done Sent false
2707 1526 Acked Read 4 Done Done Sent false
2708 1527 Acked Read 2 Done Done Sent false
2709 1528 Acked Write 9 Done Done Sent false
2710 1529 Acked Read 1 Done Done Sent false
2711 1530 Acked Write 5 Done Done Sent false
2712 1531 Acked Write 2 Done Done Sent false
2713 1532 Acked Read 5 Done Done Sent false
2714 1533 Acked Write 6 Done Done Sent false
2715 1534 Acked Read 8 Done Done Sent false
2716 1535 Acked Write 9 Done Done Sent false
2717 1536 Acked Read 2 Done Done Sent false
2718 1537 Acked Read 6 Done Done Sent false
2719 1538 Acked Flush 0 Done Done Sent false
2720 1539 Acked Write 2 Done Done Sent false
2721 1540 Acked Read 1 Done Done Sent false
2722 1541 Acked Write 2 Done Done Sent false
2723 1542 Acked Read 5 Done Done Sent false
2724 1543 Acked Write 4 Done Done Sent false
2725 1544 Acked Read 8 Done Done Sent false
2726 1545 Acked Write 1 Done Done Sent false
2727 1546 Acked Write 8 Done Done Sent false
2728 1547 Acked Write 8 Done Done Sent false
2729 1548 Acked Read 8 Done Done Sent false
2730 1549 Acked Write 1 Done Done Sent false
2731 1550 Acked Read 4 Done Done Sent false
2732 1551 Acked Read 8 Done Done Sent false
2733 1552 Acked Read 1 Done Done Sent false
2734 1553 Acked Read 5 Done Done Sent false
2735 1554 Acked Read 10 Done Done Sent false
2736 1555 Acked Write 5 Done Done Sent false
2737 1556 Acked Read 3 Done Done Sent false
2738 1557 Acked Read 6 Done Done Sent false
2739 1558 Acked Read 2 Done Done Sent false
2740 1559 Acked Write 9 Done Done Sent false
2741 1560 Acked Write 6 Done Done Sent false
2742 1561 Acked Write 1 Done Done Sent false
2743 1562 Acked Flush 0 Done Done Sent false
2744 1563 Acked Write 1 Done Done Sent false
2745 1564 Acked Write 10 Done Done Sent false
2746 1565 Acked Write 4 Done Done Sent false
2747 1566 Acked Read 8 Done Done Sent false
2748 1567 Acked Read 5 Done Done Sent false
2749 1568 Acked Write 4 Done Done Sent false
2750 1569 Acked Write 8 Done Done Sent false
2751 1570 Acked Write 7 Done Done Sent false
2752 1571 Acked Write 5 Done Done Sent false
2753 1572 Acked Read 9 Done Done Sent false
2754 1573 Acked Write 10 Done Done Sent false
2755 1574 Acked Write 7 Done Done Sent false
2756 1575 Acked Read 5 Done Done Sent false
2757 1576 Acked Write 1 Done Done Sent false
2758 1577 Acked Write 2 Done Done Sent false
2759 1578 Acked Write 9 Done Done Sent false
2760 1579 Acked Write 7 Done Done Sent false
2761 1580 Acked Flush 0 Done Done Sent false
2762 1581 Acked Read 10 Done Done Sent false
2763 1582 Acked Write 7 Done Done Sent false
2764 1583 Acked Write 9 Done Done Sent false
2765 1584 Acked Write 8 Done Done Sent false
2766 1585 Acked Write 4 Done Done Sent false
2767 1586 Acked Read 9 Done Done Sent false
2768 1587 Acked Read 10 Done Done Sent false
2769 1588 Acked Flush 0 Done Done Sent false
2770 1589 Acked Read 1 Done Done Sent false
2771 1590 Acked Write 1 Done Done Sent false
2772 1591 Acked Write 6 Done Done Sent false
2773 1592 Acked Read 7 Done Done Sent false
2774 1593 Acked Read 3 Done Done Sent false
2775 1594 Acked Write 7 Done Done Sent false
2776 1595 Acked Read 7 Done Done Sent false
2777 1596 Acked Read 4 Done Done Sent false
2778 1597 Acked Read 8 Done Done Sent false
2779 1598 Acked Read 9 Done Done Sent false
2780 1599 Acked Read 7 Done Done Sent false
2781 1600 Acked Read 4 Done Done Sent false
2782 1601 Acked Write 3 Done Done Sent false
2783 1602 Acked Read 4 Done Done Sent false
2784 STATES DS:0 DS:1 DS:2 TOTAL
2785 Sent 0 0 301 301
2786 Done 301 301 0 602
2787 Skipped 0 0 0 0
2788 Error 0 0 0 0
2789 Last Flush: Some(JobId(1588)) Some(JobId(1588)) Some(JobId(1301))
2790 Downstairs last five completed: 1301 1300 1299 1298 1297
2791 Upstairs last five completed: 1602 1601 1600 1599 1598
2792 CLIENT: Up:0 ds:301 act:2
27932025-03-17T22:19:27.095ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
27942025-03-17T22:19:27.599ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
27952025-03-17T22:19:28.601ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
27962025-03-17T22:19:29.603ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
27972025-03-17T22:19:30.348ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
27982025-03-17T22:19:30.348ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:37588) = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
27992025-03-17T22:19:30.350ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
28002025-03-17T22:19:30.350ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
28012025-03-17T22:19:30.350ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
28022025-03-17T22:19:30.350ZINFOcrucible: send last flush ID to this DS: Some(JobId(1301)) = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
28032025-03-17T22:19:30.350ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
28042025-03-17T22:19:30.350ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(1301)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
28052025-03-17T22:19:30.351ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(1301)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
2806 ----------------------------------------------------------------
2807 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:14
2808 States: Active Active Active
2809 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2810 1589 Acked Read 1 Done Done Done true
2811 1590 Acked Write 1 Done Done Done true
2812 1591 Acked Write 6 Done Done Done true
2813 1592 Acked Read 7 Done Done Done true
2814 1593 Acked Read 3 Done Done Done true
2815 1594 Acked Write 7 Done Done Done true
2816 1595 Acked Read 7 Done Done Done true
2817 1596 Acked Read 4 Done Done Done true
2818 1597 Acked Read 8 Done Done Done true
2819 1598 Acked Read 9 Done Done Done true
2820 1599 Acked Read 7 Done Done Done true
2821 1600 Acked Read 4 Done Done Done true
2822 1601 Acked Write 3 Done Done Done true
2823 1602 Acked Read 4 Done Done Done true
2824 STATES DS:0 DS:1 DS:2 TOTAL
2825 Sent 0 0 0 0
2826 Done 14 14 14 42
2827 Skipped 0 0 0 0
2828 Error 0 0 0 0
2829 Last Flush: Some(JobId(1588)) Some(JobId(1588)) Some(JobId(1588))
2830 Downstairs last five completed: 1588 1587 1586 1585 1584
2831 Upstairs last five completed: 1602 1601 1600 1599 1598
2832 CLIENT: Up:0 ds:14 act:3
2833 ----------------------------------------------------------------
2834 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
2835 Downstairs last five completed: 1603 1602 1601 1600 1599
2836 Upstairs last five completed: 1603 1602 1601 1600 1599
2837 CLIENT: Up:0 ds:0 act:3
2838 Replay: All jobs finished
2839 Main task has work to do, go find it
2840 got dsc Stop(2)
2841 stop 2
2842 Mar 17 22:19:34.597 INFO request completed, latency_us: 343, response_code: 204, uri: /stop/cid/2, method: POST, req_id: c250b042-2dc3-4e8d-983e-9e24085d88dc, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
2843 [8830] Got stop action so:false kr:false
2844 Mar 17 22:19:34.598 INFO request completed, latency_us: 120, response_code: 200, uri: /state/cid/2, method: GET, req_id: 84bcf3ab-9c29-4da0-a9d9-40bee120fe27, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
28452025-03-17T22:19:34.598ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
28462025-03-17T22:19:34.598ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
28472025-03-17T22:19:34.598ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
28482025-03-17T22:19:34.598ZINFOcrucible: client task is exiting = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
2849 [8830] Exited with: ExitStatus(unix_wait_status(9))
2850 [8830][2] reports Exit
2851 Mar 17 22:19:38.601 INFO request completed, latency_us: 253, response_code: 200, uri: /state/cid/2, method: GET, req_id: 49edae7b-27be-4182-a818-60ce57cb6c4f, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
2852 001/300 Read block 1306 len 36864
2853 002/300 Read block 2547 len 20480
2854 003/300 Write block 2418 len 32768 data: 2 2 2 2 2 2 2 2
2855 004/300 Write block 4772 len 8192 data: 3 3
2856 005/300 Read block 1038 len 32768
2857 006/300 Write block 2056 len 24576 data: 2 2 2 2 2 2
2858 007/300 Flush
2859 008/300 Write block 3125 len 28672 data: 2 2 2 2 2 2 2
2860 009/300 Read block 364 len 32768
2861 010/300 Read block 4746 len 4096
2862 011/300 Write block 299 len 16384 data: 2 2 2 2
2863 012/300 Write block 3978 len 20480 data: 3 3 3 2 2
2864 013/300 Read block 3968 len 4096
2865 014/300 Read block 2645 len 24576
2866 015/300 Read block 709 len 12288
2867 016/300 Read block 1108 len 28672
2868 017/300 Write block 2960 len 24576 data: 2 2 2 2 2 2
2869 018/300 Read block 2406 len 36864
2870 019/300 Write block 3956 len 28672 data: 2 2 2 2 2 2 2
2871 020/300 Write block 4478 len 40960 data: 2 3 3 3 3 3 3 2 2 2
2872 021/300 Write block 30 len 36864 data: 2 2 2 2 2 2 2 3 3
2873 022/300 Write block 1210 len 16384 data: 2 2 2 2
2874 023/300 Read block 4616 len 36864
2875 024/300 Flush
2876 025/300 Write block 472 len 32768 data: 2 2 2 2 2 2 2 2
2877 026/300 Read block 818 len 32768
2878 027/300 Write block 3499 len 32768 data: 2 2 2 3 3 3 3 3
2879 028/300 Read block 432 len 12288
2880 029/300 Write block 4595 len 36864 data: 2 2 2 2 2 2 2 2 2
2881 030/300 Read block 1688 len 4096
2882 031/300 Write block 1525 len 8192 data: 2 2
2883 032/300 Write block 769 len 24576 data: 2 2 2 3 3 3
2884 033/300 Write block 2219 len 36864 data: 2 2 2 2 2 2 2 2 2
2885 034/300 Write block 3622 len 20480 data: 2 2 2 2 2
2886 035/300 Write block 4987 len 28672 data: 2 2 2 2 2 2 2
2887 036/300 Read block 423 len 4096
2888 037/300 Write block 1866 len 28672 data: 3 3 3 3 3 3 3
2889 038/300 Write block 224 len 32768 data: 2 2 2 2 2 2 2 2
2890 039/300 Write block 4166 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2891 040/300 Read block 4399 len 36864
2892 041/300 Read block 1934 len 12288
2893 042/300 Write block 2969 len 20480 data: 2 2 2 2 2
2894 043/300 Write block 3968 len 24576 data: 2 2 2 2 3 3
2895 044/300 Read block 3289 len 32768
2896 045/300 Write block 29 len 36864 data: 2 3 3 3 3 3 3 3 4
2897 046/300 Write block 2864 len 16384 data: 3 3 2 2
2898 047/300 Write block 2382 len 20480 data: 2 2 2 2 2
2899 048/300 Read block 44 len 28672
2900 049/300 Write block 3018 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2901 050/300 Write block 1866 len 8192 data: 4 4
2902 051/300 Write block 2187 len 12288 data: 2 2 3
2903 052/300 Read block 426 len 24576
2904 053/300 Flush
2905 054/300 Read block 643 len 8192
2906 055/300 Flush
2907 056/300 Write block 3467 len 16384 data: 2 2 2 2
2908 057/300 Read block 3657 len 16384
2909 058/300 Write block 660 len 20480 data: 2 2 2 2 2
2910 059/300 Write block 4210 len 4096 data: 2
2911 060/300 Write block 2607 len 12288 data: 2 2 2
2912 061/300 Read block 3617 len 12288
2913 062/300 Write block 3370 len 28672 data: 2 2 2 2 2 2 2
2914 063/300 Read block 4709 len 4096
2915 064/300 Write block 3517 len 28672 data: 3 3 3 3 2 2 2
2916 065/300 Read block 4458 len 8192
2917 066/300 Flush
2918 067/300 Write block 1758 len 4096 data: 4
2919 068/300 Read block 2265 len 32768
2920 069/300 Read block 780 len 16384
2921 070/300 Write block 3214 len 12288 data: 2 2 2
2922 071/300 Write block 536 len 20480 data: 2 2 2 2 2
2923 072/300 Read block 203 len 12288
2924 073/300 Flush
2925 074/300 Write block 2779 len 32768 data: 2 2 2 2 2 2 2 2
2926 075/300 Read block 721 len 8192
2927 076/300 Write block 38 len 4096 data: 4
2928 077/300 Flush
2929 078/300 Write block 2688 len 8192 data: 2 2
2930 079/300 Write block 3440 len 4096 data: 2
2931 080/300 Read block 2297 len 16384
2932 081/300 Write block 3055 len 4096 data: 3
2933 082/300 Flush
2934 083/300 Write block 153 len 28672 data: 2 2 2 2 2 2 2
2935 084/300 Write block 27 len 24576 data: 2 2 3 4 4 4
2936 085/300 Read block 2773 len 36864
2937 086/300 Read block 819 len 12288
2938 087/300 Write block 4793 len 40960 data: 2 2 2 2 2 2 2 2 3 3
2939 088/300 Write block 3085 len 32768 data: 2 2 2 2 2 2 2 2
2940 089/300 Write block 115 len 36864 data: 2 2 2 2 2 3 3 3 3
2941 090/300 Read block 372 len 36864
2942 091/300 Read block 979 len 16384
2943 092/300 Read block 1875 len 40960
2944 093/300 Write block 3111 len 16384 data: 2 2 2 2
2945 094/300 Read block 2048 len 24576
2946 095/300 Read block 1675 len 8192
2947 096/300 Read block 494 len 28672
2948 097/300 Write block 3762 len 8192 data: 2 2
2949 098/300 Read block 78 len 8192
2950 099/300 Write block 646 len 4096 data: 2
2951 100/300 Read block 1196 len 40960
2952 101/300 Read block 4445 len 40960
2953 102/300 Read block 1995 len 16384
2954 103/300 Read block 2097 len 12288
2955 104/300 Read block 1804 len 8192
2956 105/300 Flush
2957 106/300 Read block 37 len 20480
2958 107/300 Write block 4218 len 40960 data: 2 2 3 3 3 3 3 3 3 3
2959 108/300 Read block 4094 len 36864
2960 109/300 Write block 1015 len 8192 data: 3 3
2961 110/300 Read block 76 len 32768
2962 111/300 Flush
2963 112/300 Read block 3445 len 4096
2964 113/300 Read block 1618 len 24576
2965 114/300 Write block 2491 len 28672 data: 2 2 2 2 2 3 3
2966 115/300 Write block 1932 len 8192 data: 3 3
2967 116/300 Read block 1252 len 24576
2968 117/300 Read block 890 len 12288
2969 118/300 Read block 3414 len 8192
2970 119/300 Write block 2538 len 16384 data: 2 2 2 2
2971 120/300 Read block 3453 len 4096
2972 121/300 Read block 4825 len 24576
2973 122/300 Write block 2387 len 4096 data: 2
2974 123/300 Write block 3849 len 16384 data: 2 2 2 2
2975 124/300 Write block 2764 len 8192 data: 2 2
2976 125/300 Read block 3588 len 20480
2977 126/300 Read block 2139 len 12288
2978 127/300 Write block 4554 len 24576 data: 2 2 2 2 2 2
2979 128/300 Write block 2141 len 32768 data: 3 3 2 2 2 4 4 3
2980 129/300 Write block 3343 len 12288 data: 2 2 3
2981 130/300 Write block 1717 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2982 131/300 Flush
2983 132/300 Write block 1010 len 28672 data: 2 3 3 3 3 4 4
2984 133/300 Flush
2985 134/300 Read block 4030 len 8192
2986 135/300 Write block 311 len 32768 data: 2 2 2 2 2 2 2 3
2987 136/300 Write block 366 len 16384 data: 2 2 2 2
2988 137/300 Read block 4223 len 4096
2989 138/300 Write block 1579 len 4096 data: 2
2990 139/300 Read block 3200 len 32768
2991 140/300 Read block 4220 len 36864
2992 141/300 Read block 430 len 8192
2993 142/300 Read block 1363 len 16384
2994 143/300 Write block 3715 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2995 144/300 Flush
2996 145/300 Flush
2997 146/300 Write block 3489 len 12288 data: 2 2 2
2998 147/300 Write block 2148 len 4096 data: 4
2999 148/300 Read block 1620 len 20480
3000 149/300 Read block 1250 len 4096
3001 150/300 Write block 447 len 4096 data: 4
3002 151/300 Read block 1174 len 16384
3003 152/300 Write block 895 len 36864 data: 4 3 2 2 2 2 2 2 2
3004 153/300 Read block 4093 len 16384
3005 154/300 Write block 2959 len 8192 data: 2 3
3006 155/300 Write block 3752 len 36864 data: 3 3 3 3 3 3 3 3 2
3007 156/300 Read block 519 len 36864
3008 157/300 Read block 4131 len 4096
3009 158/300 Read block 2429 len 24576
3010 159/300 Write block 1135 len 24576 data: 2 2 2 2 2 2
3011 160/300 Write block 4659 len 20480 data: 2 2 2 2 2
3012 161/300 Write block 1075 len 12288 data: 3 3 3
3013 162/300 Read block 2846 len 40960
3014 163/300 Read block 4508 len 28672
3015 164/300 Read block 1375 len 8192
3016 165/300 Read block 3779 len 24576
3017 166/300 Flush
3018 167/300 Read block 3073 len 32768
3019 168/300 Read block 1228 len 16384
3020 169/300 Write block 3589 len 28672 data: 2 2 2 2 2 2 2
3021 170/300 Write block 971 len 24576 data: 2 2 2 2 2 2
3022 171/300 Write block 2305 len 36864 data: 2 2 2 2 2 2 3 3 3
3023 172/300 Write block 4072 len 32768 data: 2 2 3 3 3 3 3 3
3024 173/300 Write block 4131 len 8192 data: 2 2
3025 174/300 Read block 4266 len 24576
3026 175/300 Write block 4175 len 32768 data: 3 3 3 3 3 2 2 2
3027 176/300 Read block 2915 len 12288
3028 177/300 Write block 2310 len 36864 data: 3 4 4 4 4 3 2 2 2
3029 178/300 Read block 955 len 20480
3030 179/300 Flush
3031 180/300 Read block 4376 len 32768
3032 181/300 Read block 1793 len 32768
3033 182/300 Write block 3759 len 12288 data: 4 3 2
3034 183/300 Read block 2038 len 36864
3035 184/300 Write block 4311 len 36864 data: 2 3 3 3 3 3 3 3 2
3036 185/300 Read block 1232 len 36864
3037 186/300 Read block 1608 len 12288
3038 187/300 Read block 72 len 16384
3039 188/300 Flush
3040 189/300 Read block 257 len 8192
3041 190/300 Write block 796 len 12288 data: 2 2 2
3042 191/300 Read block 2887 len 28672
3043 192/300 Write block 4268 len 24576 data: 2 2 2 2 2 2
3044 193/300 Write block 2739 len 32768 data: 2 2 2 3 3 3 3 3
3045 194/300 Flush
3046 195/300 Write block 370 len 36864 data: 2 3 3 3 3 3 3 2 2
3047 196/300 Write block 2735 len 24576 data: 2 2 2 2 3 3
3048 197/300 Flush
3049 198/300 Read block 3238 len 36864
3050 199/300 Write block 778 len 24576 data: 3 3 3 3 3 3
3051 200/300 Write block 4075 len 24576 data: 4 4 4 4 4 2
3052 201/300 Write block 181 len 12288 data: 2 2 2
3053 202/300 Read block 3266 len 32768
3054 203/300 Read block 503 len 8192
3055 204/300 Write block 3278 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3056 205/300 Write block 2023 len 24576 data: 3 3 2 2 2 2
3057 206/300 Write block 413 len 36864 data: 2 2 2 2 2 2 2 4 4
3058 207/300 Flush
3059 208/300 Read block 4013 len 8192
3060 209/300 Write block 2187 len 32768 data: 3 3 4 3 3 3 3 3
3061 210/300 Write block 2298 len 12288 data: 2 2 2
3062 211/300 Write block 908 len 16384 data: 2 2 3 2
3063 212/300 Write block 4733 len 28672 data: 2 2 2 2 3 3 3
3064 213/300 Read block 2114 len 28672
3065 214/300 Read block 2213 len 24576
3066 215/300 Write block 1812 len 8192 data: 3 3
3067 216/300 Write block 160 len 20480 data: 2 2 2 2 2
3068 217/300 Read block 375 len 8192
3069 218/300 Write block 1635 len 36864 data: 3 3 2 2 2 2 2 2 2
3070 219/300 Read block 1125 len 4096
3071 220/300 Write block 3055 len 16384 data: 4 3 3 2
3072 221/300 Read block 3821 len 32768
3073 222/300 Read block 3397 len 12288
3074 223/300 Read block 887 len 32768
3075 224/300 Write block 4780 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3076 225/300 Flush
3077 226/300 Read block 3142 len 20480
3078 227/300 Write block 3948 len 24576 data: 2 2 2 2 2 2
3079 228/300 Write block 3653 len 12288 data: 2 2 2
3080 229/300 Flush
3081 230/300 Write block 4843 len 16384 data: 2 2 2 2
3082 231/300 Read block 2758 len 20480
3083 232/300 Read block 2170 len 36864
3084 233/300 Flush
3085 234/300 Read block 3289 len 32768
3086 235/300 Read block 2830 len 20480
3087 236/300 Read block 4199 len 12288
3088 237/300 Read block 1680 len 20480
3089 238/300 Read block 622 len 36864
3090 239/300 Write block 2727 len 28672 data: 2 2 2 2 2 2 2
3091 240/300 Read block 1938 len 36864
3092 241/300 Flush
3093 242/300 Write block 1175 len 24576 data: 2 2 2 2 2 2
3094 243/300 Read block 1231 len 16384
3095 244/300 Read block 1000 len 4096
3096 245/300 Write block 2852 len 8192 data: 2 2
3097 246/300 Read block 4068 len 40960
3098 247/300 Write block 1279 len 36864 data: 3 3 3 2 2 2 3 4 4
3099 248/300 Write block 2472 len 24576 data: 2 2 3 3 3 3
3100 249/300 Write block 4903 len 20480 data: 2 2 2 2 2
3101 250/300 Read block 4678 len 4096
3102 251/300 Write block 1322 len 4096 data: 2
3103 252/300 Read block 38 len 12288
3104 253/300 Flush
3105 254/300 Write block 2057 len 8192 data: 3 3
3106 255/300 Read block 893 len 8192
3107 256/300 Read block 4203 len 12288
3108 257/300 Flush
3109 258/300 Read block 4540 len 28672
3110 259/300 Read block 1260 len 4096
3111 260/300 Read block 1937 len 36864
3112 261/300 Write block 2211 len 8192 data: 2 2
3113 262/300 Read block 2657 len 8192
3114 263/300 Flush
3115 264/300 Read block 4068 len 16384
3116 265/300 Read block 4234 len 16384
3117 266/300 Write block 4379 len 28672 data: 3 3 3 3 4 4 4
3118 267/300 Read block 520 len 32768
3119 268/300 Read block 3370 len 8192
3120 269/300 Read block 3546 len 24576
3121 270/300 Write block 2873 len 36864 data: 2 2 2 2 2 2 2 2 2
3122 271/300 Read block 4420 len 36864
3123 272/300 Read block 3738 len 12288
3124 273/300 Write block 1446 len 20480 data: 2 2 2 2 2
3125 274/300 Read block 2273 len 28672
3126 275/300 Write block 4868 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3127 276/300 Write block 2012 len 8192 data: 3 3
3128 277/300 Write block 2359 len 12288 data: 2 2 2
3129 278/300 Read block 3923 len 32768
3130 279/300 Write block 553 len 12288 data: 2 2 2
3131 280/300 Read block 75 len 36864
3132 281/300 Write block 4655 len 40960 data: 3 2 2 2 3 3 3 3 3 2
3133 282/300 Read block 4681 len 16384
3134 283/300 Read block 3482 len 28672
3135 284/300 Write block 4987 len 28672 data: 3 3 3 3 3 3 3
3136 285/300 Write block 88 len 4096 data: 2
3137 286/300 Read block 974 len 16384
3138 287/300 Read block 3732 len 20480
3139 288/300 Flush
3140 289/300 Write block 1587 len 28672 data: 2 2 2 2 2 3 3
3141 290/300 Write block 795 len 24576 data: 2 3 3 3 2 2
3142 291/300 Read block 4223 len 4096
3143 292/300 Read block 3699 len 8192
3144 293/300 Read block 936 len 32768
3145 294/300 Write block 2577 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3146 295/300 Write block 4740 len 28672 data: 3 3 2 2 2 2 2
3147 296/300 Flush
3148 297/300 Read block 2037 len 36864
3149 298/300 Write block 1933 len 4096 data: 4
3150 299/300 Write block 3738 len 40960 data: 2 2 2 3 3 3 3 3 3 3
3151 300/300 Write block 1109 len 8192 data: 2 2
3152 301/300 Write block 325 len 28672 data: 3 3 2 2 2 2 2
3153 Main task has work to do, go find it
3154 got dsc Start(2)
3155 start 2
3156 Mar 17 22:19:38.829 INFO request completed, latency_us: 213, response_code: 204, uri: /start/cid/2, method: POST, req_id: a8846b86-e5f1-4447-a37d-65e1d041b124, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
3157 Replay: started 2, returned:Ok(())
3158 ----------------------------------------------------------------
3159 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3160 States: Active Active Offline
3161 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3162 1604 Acked Read 9 Done Done Sent false
3163 1605 Acked Read 5 Done Done Sent false
3164 1606 Acked Write 8 Done Done Sent false
3165 1607 Acked Write 2 Done Done Sent false
3166 1608 Acked Read 8 Done Done Sent false
3167 1609 Acked Write 6 Done Done Sent false
3168 1610 Acked Flush 0 Done Done Sent false
3169 1611 Acked Write 7 Done Done Sent false
3170 1612 Acked Read 8 Done Done Sent false
3171 1613 Acked Read 1 Done Done Sent false
3172 1614 Acked Write 4 Done Done Sent false
3173 1615 Acked Write 5 Done Done Sent false
3174 1616 Acked Read 1 Done Done Sent false
3175 1617 Acked Read 6 Done Done Sent false
3176 1618 Acked Read 3 Done Done Sent false
3177 1619 Acked Read 7 Done Done Sent false
3178 1620 Acked Write 6 Done Done Sent false
3179 1621 Acked Read 9 Done Done Sent false
3180 1622 Acked Write 7 Done Done Sent false
3181 1623 Acked Write 10 Done Done Sent false
3182 1624 Acked Write 9 Done Done Sent false
3183 1625 Acked Write 4 Done Done Sent false
3184 1626 Acked Read 9 Done Done Sent false
3185 1627 Acked Flush 0 Done Done Sent false
3186 1628 Acked Write 8 Done Done Sent false
3187 1629 Acked Read 8 Done Done Sent false
3188 1630 Acked Write 8 Done Done Sent false
3189 1631 Acked Read 3 Done Done Sent false
3190 1632 Acked Write 9 Done Done Sent false
3191 1633 Acked Read 1 Done Done Sent false
3192 1634 Acked Write 2 Done Done Sent false
3193 1635 Acked Write 6 Done Done Sent false
3194 1636 Acked Write 9 Done Done Sent false
3195 1637 Acked Write 5 Done Done Sent false
3196 1638 Acked Write 7 Done Done Sent false
3197 1639 Acked Read 1 Done Done Sent false
3198 1640 Acked Write 7 Done Done Sent false
3199 1641 Acked Write 8 Done Done Sent false
3200 1642 Acked Write 10 Done Done Sent false
3201 1643 Acked Read 9 Done Done Sent false
3202 1644 Acked Read 3 Done Done Sent false
3203 1645 Acked Write 5 Done Done Sent false
3204 1646 Acked Write 6 Done Done Sent false
3205 1647 Acked Read 8 Done Done Sent false
3206 1648 Acked Write 9 Done Done Sent false
3207 1649 Acked Write 4 Done Done Sent false
3208 1650 Acked Write 5 Done Done Sent false
3209 1651 Acked Read 7 Done Done Sent false
3210 1652 Acked Write 10 Done Done Sent false
3211 1653 Acked Write 2 Done Done Sent false
3212 1654 Acked Write 3 Done Done Sent false
3213 1655 Acked Read 6 Done Done Sent false
3214 1656 Acked Flush 0 Done Done Sent false
3215 1657 Acked Read 2 Done Done Sent false
3216 1658 Acked Flush 0 Done Done Sent false
3217 1659 Acked Write 4 Done Done Sent false
3218 1660 Acked Read 4 Done Done Sent false
3219 1661 Acked Write 5 Done Done Sent false
3220 1662 Acked Write 1 Done Done Sent false
3221 1663 Acked Write 3 Done Done Sent false
3222 1664 Acked Read 3 Done Done Sent false
3223 1665 Acked Write 7 Done Done Sent false
3224 1666 Acked Read 1 Done Done Sent false
3225 1667 Acked Write 7 Done Done Sent false
3226 1668 Acked Read 2 Done Done Sent false
3227 1669 Acked Flush 0 Done Done Sent false
3228 1670 Acked Write 1 Done Done Sent false
3229 1671 Acked Read 8 Done Done Sent false
3230 1672 Acked Read 4 Done Done Sent false
3231 1673 Acked Write 3 Done Done Sent false
3232 1674 Acked Write 5 Done Done Sent false
3233 1675 Acked Read 3 Done Done Sent false
3234 1676 Acked Flush 0 Done Done Sent false
3235 1677 Acked Write 8 Done Done Sent false
3236 1678 Acked Read 2 Done Done Sent false
3237 1679 Acked Write 1 Done Done Sent false
3238 1680 Acked Flush 0 Done Done Sent false
3239 1681 Acked Write 2 Done Done Sent false
3240 1682 Acked Write 1 Done Done Sent false
3241 1683 Acked Read 4 Done Done Sent false
3242 1684 Acked Write 1 Done Done Sent false
3243 1685 Acked Flush 0 Done Done Sent false
3244 1686 Acked Write 7 Done Done Sent false
3245 1687 Acked Write 6 Done Done Sent false
3246 1688 Acked Read 9 Done Done Sent false
3247 1689 Acked Read 3 Done Done Sent false
3248 1690 Acked Write 10 Done Done Sent false
3249 1691 Acked Write 8 Done Done Sent false
3250 1692 Acked Write 9 Done Done Sent false
3251 1693 Acked Read 9 Done Done Sent false
3252 1694 Acked Read 4 Done Done Sent false
3253 1695 Acked Read 10 Done Done Sent false
3254 1696 Acked Write 4 Done Done Sent false
3255 1697 Acked Read 6 Done Done Sent false
3256 1698 Acked Read 2 Done Done Sent false
3257 1699 Acked Read 7 Done Done Sent false
3258 1700 Acked Write 2 Done Done Sent false
3259 1701 Acked Read 2 Done Done Sent false
3260 1702 Acked Write 1 Done Done Sent false
3261 1703 Acked Read 10 Done Done Sent false
3262 1704 Acked Read 10 Done Done Sent false
3263 1705 Acked Read 4 Done Done Sent false
3264 1706 Acked Read 3 Done Done Sent false
3265 1707 Acked Read 2 Done Done Sent false
3266 1708 Acked Flush 0 Done Done Sent false
3267 1709 Acked Read 5 Done Done Sent false
3268 1710 Acked Write 10 Done Done Sent false
3269 1711 Acked Read 9 Done Done Sent false
3270 1712 Acked Write 2 Done Done Sent false
3271 1713 Acked Read 8 Done Done Sent false
3272 1714 Acked Flush 0 Done Done Sent false
3273 1715 Acked Read 1 Done Done Sent false
3274 1716 Acked Read 6 Done Done Sent false
3275 1717 Acked Write 7 Done Done Sent false
3276 1718 Acked Write 2 Done Done Sent false
3277 1719 Acked Read 6 Done Done Sent false
3278 1720 Acked Read 3 Done Done Sent false
3279 1721 Acked Read 2 Done Done Sent false
3280 1722 Acked Write 4 Done Done Sent false
3281 1723 Acked Read 1 Done Done Sent false
3282 1724 Acked Read 6 Done Done Sent false
3283 1725 Acked Write 1 Done Done Sent false
3284 1726 Acked Write 4 Done Done Sent false
3285 1727 Acked Write 2 Done Done Sent false
3286 1728 Acked Read 5 Done Done Sent false
3287 1729 Acked Read 3 Done Done Sent false
3288 1730 Acked Write 6 Done Done Sent false
3289 1731 Acked Write 8 Done Done Sent false
3290 1732 Acked Write 3 Done Done Sent false
3291 1733 Acked Write 10 Done Done Sent false
3292 1734 Acked Flush 0 Done Done Sent false
3293 1735 Acked Write 7 Done Done Sent false
3294 1736 Acked Flush 0 Done Done Sent false
3295 1737 Acked Read 2 Done Done Sent false
3296 1738 Acked Write 8 Done Done Sent false
3297 1739 Acked Write 4 Done Done Sent false
3298 1740 Acked Read 1 Done Done Sent false
3299 1741 Acked Write 1 Done Done Sent false
3300 1742 Acked Read 8 Done Done Sent false
3301 1743 Acked Read 9 Done Done Sent false
3302 1744 Acked Read 2 Done Done Sent false
3303 1745 Acked Read 4 Done Done Sent false
3304 1746 Acked Write 10 Done Done Sent false
3305 1747 Acked Flush 0 Done Done Sent false
3306 1748 Acked Flush 0 Done Done Sent false
3307 1749 Acked Write 3 Done Done Sent false
3308 1750 Acked Write 1 Done Done Sent false
3309 1751 Acked Read 5 Done Done Sent false
3310 1752 Acked Read 1 Done Done Sent false
3311 1753 Acked Write 1 Done Done Sent false
3312 1754 Acked Read 4 Done Done Sent false
3313 1755 Acked Write 9 Done Done Sent false
3314 1756 Acked Read 4 Done Done Sent false
3315 1757 Acked Write 2 Done Done Sent false
3316 1758 Acked Write 9 Done Done Sent false
3317 1759 Acked Read 9 Done Done Sent false
3318 1760 Acked Read 1 Done Done Sent false
3319 1761 Acked Read 6 Done Done Sent false
3320 1762 Acked Write 6 Done Done Sent false
3321 1763 Acked Write 5 Done Done Sent false
3322 1764 Acked Write 3 Done Done Sent false
3323 1765 Acked Read 10 Done Done Sent false
3324 1766 Acked Read 7 Done Done Sent false
3325 1767 Acked Read 2 Done Done Sent false
3326 1768 Acked Read 6 Done Done Sent false
3327 1769 Acked Flush 0 Done Done Sent false
3328 1770 Acked Read 8 Done Done Sent false
3329 1771 Acked Read 4 Done Done Sent false
3330 1772 Acked Write 7 Done Done Sent false
3331 1773 Acked Write 6 Done Done Sent false
3332 1774 Acked Write 9 Done Done Sent false
3333 1775 Acked Write 8 Done Done Sent false
3334 1776 Acked Write 2 Done Done Sent false
3335 1777 Acked Read 6 Done Done Sent false
3336 1778 Acked Write 8 Done Done Sent false
3337 1779 Acked Read 3 Done Done Sent false
3338 1780 Acked Write 9 Done Done Sent false
3339 1781 Acked Read 5 Done Done Sent false
3340 1782 Acked Flush 0 Done Done Sent false
3341 1783 Acked Read 8 Done Done Sent false
3342 1784 Acked Read 8 Done Done Sent false
3343 1785 Acked Write 3 Done Done Sent false
3344 1786 Acked Read 9 Done Done Sent false
3345 1787 Acked Write 9 Done Done Sent false
3346 1788 Acked Read 9 Done Done Sent false
3347 1789 Acked Read 3 Done Done Sent false
3348 1790 Acked Read 4 Done Done Sent false
3349 1791 Acked Flush 0 Done Done Sent false
3350 1792 Acked Read 2 Done Done Sent false
3351 1793 Acked Write 3 Done Done Sent false
3352 1794 Acked Read 7 Done Done Sent false
3353 1795 Acked Write 6 Done Done Sent false
3354 1796 Acked Write 8 Done Done Sent false
3355 1797 Acked Flush 0 Done Done Sent false
3356 1798 Acked Write 9 Done Done Sent false
3357 1799 Acked Write 6 Done Done Sent false
3358 1800 Acked Flush 0 Done Done Sent false
3359 1801 Acked Read 9 Done Done Sent false
3360 1802 Acked Write 6 Done Done Sent false
3361 1803 Acked Write 6 Done Done Sent false
3362 1804 Acked Write 3 Done Done Sent false
3363 1805 Acked Read 8 Done Done Sent false
3364 1806 Acked Read 2 Done Done Sent false
3365 1807 Acked Write 10 Done Done Sent false
3366 1808 Acked Write 6 Done Done Sent false
3367 1809 Acked Write 9 Done Done Sent false
3368 1810 Acked Flush 0 Done Done Sent false
3369 1811 Acked Read 2 Done Done Sent false
3370 1812 Acked Write 8 Done Done Sent false
3371 1813 Acked Write 3 Done Done Sent false
3372 1814 Acked Write 4 Done Done Sent false
3373 1815 Acked Write 7 Done Done Sent false
3374 1816 Acked Read 7 Done Done Sent false
3375 1817 Acked Read 6 Done Done Sent false
3376 1818 Acked Write 2 Done Done Sent false
3377 1819 Acked Write 5 Done Done Sent false
3378 1820 Acked Read 2 Done Done Sent false
3379 1821 Acked Write 9 Done Done Sent false
3380 1822 Acked Read 1 Done Done Sent false
3381 1823 Acked Write 4 Done Done Sent false
3382 1824 Acked Read 8 Done Done Sent false
3383 1825 Acked Read 3 Done Done Sent false
3384 1826 Acked Read 8 Done Done Sent false
3385 1827 Acked Write 10 Done Done Sent false
3386 1828 Acked Flush 0 Done Done Sent false
3387 1829 Acked Read 5 Done Done Sent false
3388 1830 Acked Write 6 Done Done Sent false
3389 1831 Acked Write 3 Done Done Sent false
3390 1832 Acked Flush 0 Done Done Sent false
3391 1833 Acked Write 4 Done Done Sent false
3392 1834 Acked Read 5 Done Done Sent false
3393 1835 Acked Read 9 Done Done Sent false
3394 1836 Acked Flush 0 Done Done Sent false
3395 1837 Acked Read 8 Done Done Sent false
3396 1838 Acked Read 5 Done Done Sent false
3397 1839 Acked Read 3 Done Done Sent false
3398 1840 Acked Read 5 Done Done Sent false
3399 1841 Acked Read 9 Done Done Sent false
3400 1842 Acked Write 7 Done Done Sent false
3401 1843 Acked Read 9 Done Done Sent false
3402 1844 Acked Flush 0 Done Done Sent false
3403 1845 Acked Write 6 Done Done Sent false
3404 1846 Acked Read 4 Done Done Sent false
3405 1847 Acked Read 1 Done Done Sent false
3406 1848 Acked Write 2 Done Done Sent false
3407 1849 Acked Read 10 Done Done Sent false
3408 1850 Acked Write 9 Done Done Sent false
3409 1851 Acked Write 6 Done Done Sent false
3410 1852 Acked Write 5 Done Done Sent false
3411 1853 Acked Read 1 Done Done Sent false
3412 1854 Acked Write 1 Done Done Sent false
3413 1855 Acked Read 3 Done Done Sent false
3414 1856 Acked Flush 0 Done Done Sent false
3415 1857 Acked Write 2 Done Done Sent false
3416 1858 Acked Read 2 Done Done Sent false
3417 1859 Acked Read 3 Done Done Sent false
3418 1860 Acked Flush 0 Done Done Sent false
3419 1861 Acked Read 7 Done Done Sent false
3420 1862 Acked Read 1 Done Done Sent false
3421 1863 Acked Read 9 Done Done Sent false
3422 1864 Acked Write 2 Done Done Sent false
3423 1865 Acked Read 2 Done Done Sent false
3424 1866 Acked Flush 0 Done Done Sent false
3425 1867 Acked Read 4 Done Done Sent false
3426 1868 Acked Read 4 Done Done Sent false
3427 1869 Acked Write 7 Done Done Sent false
3428 1870 Acked Read 8 Done Done Sent false
3429 1871 Acked Read 2 Done Done Sent false
3430 1872 Acked Read 6 Done Done Sent false
3431 1873 Acked Write 9 Done Done Sent false
3432 1874 Acked Read 9 Done Done Sent false
3433 1875 Acked Read 3 Done Done Sent false
3434 1876 Acked Write 5 Done Done Sent false
3435 1877 Acked Read 7 Done Done Sent false
3436 1878 Acked Write 10 Done Done Sent false
3437 1879 Acked Write 2 Done Done Sent false
3438 1880 Acked Write 3 Done Done Sent false
3439 1881 Acked Read 8 Done Done Sent false
3440 1882 Acked Write 3 Done Done Sent false
3441 1883 Acked Read 9 Done Done Sent false
3442 1884 Acked Write 10 Done Done Sent false
3443 1885 Acked Read 4 Done Done Sent false
3444 1886 Acked Read 7 Done Done Sent false
3445 1887 Acked Write 7 Done Done Sent false
3446 1888 Acked Write 1 Done Done Sent false
3447 1889 Acked Read 4 Done Done Sent false
3448 1890 Acked Read 5 Done Done Sent false
3449 1891 Acked Flush 0 Done Done Sent false
3450 1892 Acked Write 7 Done Done Sent false
3451 1893 Acked Write 6 Done Done Sent false
3452 1894 Acked Read 1 Done Done Sent false
3453 1895 Acked Read 2 Done Done Sent false
3454 1896 Acked Read 8 Done Done Sent false
3455 1897 Acked Write 10 Done Done Sent false
3456 1898 Acked Write 7 Done Done Sent false
3457 1899 Acked Flush 0 Done Done Sent false
3458 1900 Acked Read 9 Done Done Sent false
3459 1901 Acked Write 1 Done Done Sent false
3460 1902 Acked Write 10 Done Done Sent false
3461 1903 Acked Write 2 Done Done Sent false
3462 1904 Acked Write 7 Done Done Sent false
3463 STATES DS:0 DS:1 DS:2 TOTAL
3464 Sent 0 0 301 301
3465 Done 301 301 0 602
3466 Skipped 0 0 0 0
3467 Error 0 0 0 0
3468 Last Flush: Some(JobId(1899)) Some(JobId(1899)) Some(JobId(1603))
3469 Downstairs last five completed: 1603 1602 1601 1600 1599
3470 Upstairs last five completed: 1904 1903 1902 1901 1900
3471 CLIENT: Up:0 ds:301 act:2
34722025-03-17T22:19:39.619ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
34732025-03-17T22:19:40.621ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
34742025-03-17T22:19:41.623ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
34752025-03-17T22:19:42.625ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
3476 ----------------------------------------------------------------
3477 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3478 States: Active Active Offline
3479 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3480 1604 Acked Read 9 Done Done Sent false
3481 1605 Acked Read 5 Done Done Sent false
3482 1606 Acked Write 8 Done Done Sent false
3483 1607 Acked Write 2 Done Done Sent false
3484 1608 Acked Read 8 Done Done Sent false
3485 1609 Acked Write 6 Done Done Sent false
3486 1610 Acked Flush 0 Done Done Sent false
3487 1611 Acked Write 7 Done Done Sent false
3488 1612 Acked Read 8 Done Done Sent false
3489 1613 Acked Read 1 Done Done Sent false
3490 1614 Acked Write 4 Done Done Sent false
3491 1615 Acked Write 5 Done Done Sent false
3492 1616 Acked Read 1 Done Done Sent false
3493 1617 Acked Read 6 Done Done Sent false
3494 1618 Acked Read 3 Done Done Sent false
3495 1619 Acked Read 7 Done Done Sent false
3496 1620 Acked Write 6 Done Done Sent false
3497 1621 Acked Read 9 Done Done Sent false
3498 1622 Acked Write 7 Done Done Sent false
3499 1623 Acked Write 10 Done Done Sent false
3500 1624 Acked Write 9 Done Done Sent false
3501 1625 Acked Write 4 Done Done Sent false
3502 1626 Acked Read 9 Done Done Sent false
3503 1627 Acked Flush 0 Done Done Sent false
3504 1628 Acked Write 8 Done Done Sent false
3505 1629 Acked Read 8 Done Done Sent false
3506 1630 Acked Write 8 Done Done Sent false
3507 1631 Acked Read 3 Done Done Sent false
3508 1632 Acked Write 9 Done Done Sent false
3509 1633 Acked Read 1 Done Done Sent false
3510 1634 Acked Write 2 Done Done Sent false
3511 1635 Acked Write 6 Done Done Sent false
3512 1636 Acked Write 9 Done Done Sent false
3513 1637 Acked Write 5 Done Done Sent false
3514 1638 Acked Write 7 Done Done Sent false
3515 1639 Acked Read 1 Done Done Sent false
3516 1640 Acked Write 7 Done Done Sent false
3517 1641 Acked Write 8 Done Done Sent false
3518 1642 Acked Write 10 Done Done Sent false
3519 1643 Acked Read 9 Done Done Sent false
3520 1644 Acked Read 3 Done Done Sent false
3521 1645 Acked Write 5 Done Done Sent false
3522 1646 Acked Write 6 Done Done Sent false
3523 1647 Acked Read 8 Done Done Sent false
3524 1648 Acked Write 9 Done Done Sent false
3525 1649 Acked Write 4 Done Done Sent false
3526 1650 Acked Write 5 Done Done Sent false
3527 1651 Acked Read 7 Done Done Sent false
3528 1652 Acked Write 10 Done Done Sent false
3529 1653 Acked Write 2 Done Done Sent false
3530 1654 Acked Write 3 Done Done Sent false
3531 1655 Acked Read 6 Done Done Sent false
3532 1656 Acked Flush 0 Done Done Sent false
3533 1657 Acked Read 2 Done Done Sent false
3534 1658 Acked Flush 0 Done Done Sent false
3535 1659 Acked Write 4 Done Done Sent false
3536 1660 Acked Read 4 Done Done Sent false
3537 1661 Acked Write 5 Done Done Sent false
3538 1662 Acked Write 1 Done Done Sent false
3539 1663 Acked Write 3 Done Done Sent false
3540 1664 Acked Read 3 Done Done Sent false
3541 1665 Acked Write 7 Done Done Sent false
3542 1666 Acked Read 1 Done Done Sent false
3543 1667 Acked Write 7 Done Done Sent false
3544 1668 Acked Read 2 Done Done Sent false
3545 1669 Acked Flush 0 Done Done Sent false
3546 1670 Acked Write 1 Done Done Sent false
3547 1671 Acked Read 8 Done Done Sent false
3548 1672 Acked Read 4 Done Done Sent false
3549 1673 Acked Write 3 Done Done Sent false
3550 1674 Acked Write 5 Done Done Sent false
3551 1675 Acked Read 3 Done Done Sent false
3552 1676 Acked Flush 0 Done Done Sent false
3553 1677 Acked Write 8 Done Done Sent false
3554 1678 Acked Read 2 Done Done Sent false
3555 1679 Acked Write 1 Done Done Sent false
3556 1680 Acked Flush 0 Done Done Sent false
3557 1681 Acked Write 2 Done Done Sent false
3558 1682 Acked Write 1 Done Done Sent false
3559 1683 Acked Read 4 Done Done Sent false
3560 1684 Acked Write 1 Done Done Sent false
3561 1685 Acked Flush 0 Done Done Sent false
3562 1686 Acked Write 7 Done Done Sent false
3563 1687 Acked Write 6 Done Done Sent false
3564 1688 Acked Read 9 Done Done Sent false
3565 1689 Acked Read 3 Done Done Sent false
3566 1690 Acked Write 10 Done Done Sent false
3567 1691 Acked Write 8 Done Done Sent false
3568 1692 Acked Write 9 Done Done Sent false
3569 1693 Acked Read 9 Done Done Sent false
3570 1694 Acked Read 4 Done Done Sent false
3571 1695 Acked Read 10 Done Done Sent false
3572 1696 Acked Write 4 Done Done Sent false
3573 1697 Acked Read 6 Done Done Sent false
3574 1698 Acked Read 2 Done Done Sent false
3575 1699 Acked Read 7 Done Done Sent false
3576 1700 Acked Write 2 Done Done Sent false
3577 1701 Acked Read 2 Done Done Sent false
3578 1702 Acked Write 1 Done Done Sent false
3579 1703 Acked Read 10 Done Done Sent false
3580 1704 Acked Read 10 Done Done Sent false
3581 1705 Acked Read 4 Done Done Sent false
3582 1706 Acked Read 3 Done Done Sent false
3583 1707 Acked Read 2 Done Done Sent false
3584 1708 Acked Flush 0 Done Done Sent false
3585 1709 Acked Read 5 Done Done Sent false
3586 1710 Acked Write 10 Done Done Sent false
3587 1711 Acked Read 9 Done Done Sent false
3588 1712 Acked Write 2 Done Done Sent false
3589 1713 Acked Read 8 Done Done Sent false
3590 1714 Acked Flush 0 Done Done Sent false
3591 1715 Acked Read 1 Done Done Sent false
3592 1716 Acked Read 6 Done Done Sent false
3593 1717 Acked Write 7 Done Done Sent false
3594 1718 Acked Write 2 Done Done Sent false
3595 1719 Acked Read 6 Done Done Sent false
3596 1720 Acked Read 3 Done Done Sent false
3597 1721 Acked Read 2 Done Done Sent false
3598 1722 Acked Write 4 Done Done Sent false
3599 1723 Acked Read 1 Done Done Sent false
3600 1724 Acked Read 6 Done Done Sent false
3601 1725 Acked Write 1 Done Done Sent false
3602 1726 Acked Write 4 Done Done Sent false
3603 1727 Acked Write 2 Done Done Sent false
3604 1728 Acked Read 5 Done Done Sent false
3605 1729 Acked Read 3 Done Done Sent false
3606 1730 Acked Write 6 Done Done Sent false
3607 1731 Acked Write 8 Done Done Sent false
3608 1732 Acked Write 3 Done Done Sent false
3609 1733 Acked Write 10 Done Done Sent false
3610 1734 Acked Flush 0 Done Done Sent false
3611 1735 Acked Write 7 Done Done Sent false
3612 1736 Acked Flush 0 Done Done Sent false
3613 1737 Acked Read 2 Done Done Sent false
3614 1738 Acked Write 8 Done Done Sent false
3615 1739 Acked Write 4 Done Done Sent false
3616 1740 Acked Read 1 Done Done Sent false
3617 1741 Acked Write 1 Done Done Sent false
3618 1742 Acked Read 8 Done Done Sent false
3619 1743 Acked Read 9 Done Done Sent false
3620 1744 Acked Read 2 Done Done Sent false
3621 1745 Acked Read 4 Done Done Sent false
3622 1746 Acked Write 10 Done Done Sent false
3623 1747 Acked Flush 0 Done Done Sent false
3624 1748 Acked Flush 0 Done Done Sent false
3625 1749 Acked Write 3 Done Done Sent false
3626 1750 Acked Write 1 Done Done Sent false
3627 1751 Acked Read 5 Done Done Sent false
3628 1752 Acked Read 1 Done Done Sent false
3629 1753 Acked Write 1 Done Done Sent false
3630 1754 Acked Read 4 Done Done Sent false
3631 1755 Acked Write 9 Done Done Sent false
3632 1756 Acked Read 4 Done Done Sent false
3633 1757 Acked Write 2 Done Done Sent false
3634 1758 Acked Write 9 Done Done Sent false
3635 1759 Acked Read 9 Done Done Sent false
3636 1760 Acked Read 1 Done Done Sent false
3637 1761 Acked Read 6 Done Done Sent false
3638 1762 Acked Write 6 Done Done Sent false
3639 1763 Acked Write 5 Done Done Sent false
3640 1764 Acked Write 3 Done Done Sent false
3641 1765 Acked Read 10 Done Done Sent false
3642 1766 Acked Read 7 Done Done Sent false
3643 1767 Acked Read 2 Done Done Sent false
3644 1768 Acked Read 6 Done Done Sent false
3645 1769 Acked Flush 0 Done Done Sent false
3646 1770 Acked Read 8 Done Done Sent false
3647 1771 Acked Read 4 Done Done Sent false
3648 1772 Acked Write 7 Done Done Sent false
3649 1773 Acked Write 6 Done Done Sent false
3650 1774 Acked Write 9 Done Done Sent false
3651 1775 Acked Write 8 Done Done Sent false
3652 1776 Acked Write 2 Done Done Sent false
3653 1777 Acked Read 6 Done Done Sent false
3654 1778 Acked Write 8 Done Done Sent false
3655 1779 Acked Read 3 Done Done Sent false
3656 1780 Acked Write 9 Done Done Sent false
3657 1781 Acked Read 5 Done Done Sent false
3658 1782 Acked Flush 0 Done Done Sent false
3659 1783 Acked Read 8 Done Done Sent false
3660 1784 Acked Read 8 Done Done Sent false
3661 1785 Acked Write 3 Done Done Sent false
3662 1786 Acked Read 9 Done Done Sent false
3663 1787 Acked Write 9 Done Done Sent false
3664 1788 Acked Read 9 Done Done Sent false
3665 1789 Acked Read 3 Done Done Sent false
3666 1790 Acked Read 4 Done Done Sent false
3667 1791 Acked Flush 0 Done Done Sent false
3668 1792 Acked Read 2 Done Done Sent false
3669 1793 Acked Write 3 Done Done Sent false
3670 1794 Acked Read 7 Done Done Sent false
3671 1795 Acked Write 6 Done Done Sent false
3672 1796 Acked Write 8 Done Done Sent false
3673 1797 Acked Flush 0 Done Done Sent false
3674 1798 Acked Write 9 Done Done Sent false
3675 1799 Acked Write 6 Done Done Sent false
3676 1800 Acked Flush 0 Done Done Sent false
3677 1801 Acked Read 9 Done Done Sent false
3678 1802 Acked Write 6 Done Done Sent false
3679 1803 Acked Write 6 Done Done Sent false
3680 1804 Acked Write 3 Done Done Sent false
3681 1805 Acked Read 8 Done Done Sent false
3682 1806 Acked Read 2 Done Done Sent false
3683 1807 Acked Write 10 Done Done Sent false
3684 1808 Acked Write 6 Done Done Sent false
3685 1809 Acked Write 9 Done Done Sent false
3686 1810 Acked Flush 0 Done Done Sent false
3687 1811 Acked Read 2 Done Done Sent false
3688 1812 Acked Write 8 Done Done Sent false
3689 1813 Acked Write 3 Done Done Sent false
3690 1814 Acked Write 4 Done Done Sent false
3691 1815 Acked Write 7 Done Done Sent false
3692 1816 Acked Read 7 Done Done Sent false
3693 1817 Acked Read 6 Done Done Sent false
3694 1818 Acked Write 2 Done Done Sent false
3695 1819 Acked Write 5 Done Done Sent false
3696 1820 Acked Read 2 Done Done Sent false
3697 1821 Acked Write 9 Done Done Sent false
3698 1822 Acked Read 1 Done Done Sent false
3699 1823 Acked Write 4 Done Done Sent false
3700 1824 Acked Read 8 Done Done Sent false
3701 1825 Acked Read 3 Done Done Sent false
3702 1826 Acked Read 8 Done Done Sent false
3703 1827 Acked Write 10 Done Done Sent false
3704 1828 Acked Flush 0 Done Done Sent false
3705 1829 Acked Read 5 Done Done Sent false
3706 1830 Acked Write 6 Done Done Sent false
3707 1831 Acked Write 3 Done Done Sent false
3708 1832 Acked Flush 0 Done Done Sent false
3709 1833 Acked Write 4 Done Done Sent false
3710 1834 Acked Read 5 Done Done Sent false
3711 1835 Acked Read 9 Done Done Sent false
3712 1836 Acked Flush 0 Done Done Sent false
3713 1837 Acked Read 8 Done Done Sent false
3714 1838 Acked Read 5 Done Done Sent false
3715 1839 Acked Read 3 Done Done Sent false
3716 1840 Acked Read 5 Done Done Sent false
3717 1841 Acked Read 9 Done Done Sent false
3718 1842 Acked Write 7 Done Done Sent false
3719 1843 Acked Read 9 Done Done Sent false
3720 1844 Acked Flush 0 Done Done Sent false
3721 1845 Acked Write 6 Done Done Sent false
3722 1846 Acked Read 4 Done Done Sent false
3723 1847 Acked Read 1 Done Done Sent false
3724 1848 Acked Write 2 Done Done Sent false
3725 1849 Acked Read 10 Done Done Sent false
3726 1850 Acked Write 9 Done Done Sent false
3727 1851 Acked Write 6 Done Done Sent false
3728 1852 Acked Write 5 Done Done Sent false
3729 1853 Acked Read 1 Done Done Sent false
3730 1854 Acked Write 1 Done Done Sent false
3731 1855 Acked Read 3 Done Done Sent false
3732 1856 Acked Flush 0 Done Done Sent false
3733 1857 Acked Write 2 Done Done Sent false
3734 1858 Acked Read 2 Done Done Sent false
3735 1859 Acked Read 3 Done Done Sent false
3736 1860 Acked Flush 0 Done Done Sent false
3737 1861 Acked Read 7 Done Done Sent false
3738 1862 Acked Read 1 Done Done Sent false
3739 1863 Acked Read 9 Done Done Sent false
3740 1864 Acked Write 2 Done Done Sent false
3741 1865 Acked Read 2 Done Done Sent false
3742 1866 Acked Flush 0 Done Done Sent false
3743 1867 Acked Read 4 Done Done Sent false
3744 1868 Acked Read 4 Done Done Sent false
3745 1869 Acked Write 7 Done Done Sent false
3746 1870 Acked Read 8 Done Done Sent false
3747 1871 Acked Read 2 Done Done Sent false
3748 1872 Acked Read 6 Done Done Sent false
3749 1873 Acked Write 9 Done Done Sent false
3750 1874 Acked Read 9 Done Done Sent false
3751 1875 Acked Read 3 Done Done Sent false
3752 1876 Acked Write 5 Done Done Sent false
3753 1877 Acked Read 7 Done Done Sent false
3754 1878 Acked Write 10 Done Done Sent false
3755 1879 Acked Write 2 Done Done Sent false
3756 1880 Acked Write 3 Done Done Sent false
3757 1881 Acked Read 8 Done Done Sent false
3758 1882 Acked Write 3 Done Done Sent false
3759 1883 Acked Read 9 Done Done Sent false
3760 1884 Acked Write 10 Done Done Sent false
3761 1885 Acked Read 4 Done Done Sent false
3762 1886 Acked Read 7 Done Done Sent false
3763 1887 Acked Write 7 Done Done Sent false
3764 1888 Acked Write 1 Done Done Sent false
3765 1889 Acked Read 4 Done Done Sent false
3766 1890 Acked Read 5 Done Done Sent false
3767 1891 Acked Flush 0 Done Done Sent false
3768 1892 Acked Write 7 Done Done Sent false
3769 1893 Acked Write 6 Done Done Sent false
3770 1894 Acked Read 1 Done Done Sent false
3771 1895 Acked Read 2 Done Done Sent false
3772 1896 Acked Read 8 Done Done Sent false
3773 1897 Acked Write 10 Done Done Sent false
3774 1898 Acked Write 7 Done Done Sent false
3775 1899 Acked Flush 0 Done Done Sent false
3776 1900 Acked Read 9 Done Done Sent false
3777 1901 Acked Write 1 Done Done Sent false
3778 1902 Acked Write 10 Done Done Sent false
3779 1903 Acked Write 2 Done Done Sent false
3780 1904 Acked Write 7 Done Done Sent false
3781 STATES DS:0 DS:1 DS:2 TOTAL
3782 Sent 0 0 301 301
3783 Done 301 301 0 602
3784 Skipped 0 0 0 0
3785 Error 0 0 0 0
3786 Last Flush: Some(JobId(1899)) Some(JobId(1899)) Some(JobId(1603))
3787 Downstairs last five completed: 1603 1602 1601 1600 1599
3788 Upstairs last five completed: 1904 1903 1902 1901 1900
3789 CLIENT: Up:0 ds:301 act:2
37902025-03-17T22:19:43.626ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
37912025-03-17T22:19:44.599ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
37922025-03-17T22:19:44.599ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
37932025-03-17T22:19:44.599ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
37942025-03-17T22:19:44.599ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
37952025-03-17T22:19:44.600ZINFOcrucible: client task is exiting = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
3796 [8830] got start action
3797 [8830] I am going to restart
3798 Starting downstairs at port 8830
3799 Make output file at "/tmp/dsc/downstairs-8830.txt"
3800 [8830][2] reports Starting
3801 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1214)
3802 [8830][2] reports Running
38032025-03-17T22:19:45.100ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
38042025-03-17T22:19:45.629ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
38052025-03-17T22:19:46.630ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
3806 ----------------------------------------------------------------
3807 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3808 States: Active Active Offline
3809 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3810 1604 Acked Read 9 Done Done Sent false
3811 1605 Acked Read 5 Done Done Sent false
3812 1606 Acked Write 8 Done Done Sent false
3813 1607 Acked Write 2 Done Done Sent false
3814 1608 Acked Read 8 Done Done Sent false
3815 1609 Acked Write 6 Done Done Sent false
3816 1610 Acked Flush 0 Done Done Sent false
3817 1611 Acked Write 7 Done Done Sent false
3818 1612 Acked Read 8 Done Done Sent false
3819 1613 Acked Read 1 Done Done Sent false
3820 1614 Acked Write 4 Done Done Sent false
3821 1615 Acked Write 5 Done Done Sent false
3822 1616 Acked Read 1 Done Done Sent false
3823 1617 Acked Read 6 Done Done Sent false
3824 1618 Acked Read 3 Done Done Sent false
3825 1619 Acked Read 7 Done Done Sent false
3826 1620 Acked Write 6 Done Done Sent false
3827 1621 Acked Read 9 Done Done Sent false
3828 1622 Acked Write 7 Done Done Sent false
3829 1623 Acked Write 10 Done Done Sent false
3830 1624 Acked Write 9 Done Done Sent false
3831 1625 Acked Write 4 Done Done Sent false
3832 1626 Acked Read 9 Done Done Sent false
3833 1627 Acked Flush 0 Done Done Sent false
3834 1628 Acked Write 8 Done Done Sent false
3835 1629 Acked Read 8 Done Done Sent false
3836 1630 Acked Write 8 Done Done Sent false
3837 1631 Acked Read 3 Done Done Sent false
3838 1632 Acked Write 9 Done Done Sent false
3839 1633 Acked Read 1 Done Done Sent false
3840 1634 Acked Write 2 Done Done Sent false
3841 1635 Acked Write 6 Done Done Sent false
3842 1636 Acked Write 9 Done Done Sent false
3843 1637 Acked Write 5 Done Done Sent false
3844 1638 Acked Write 7 Done Done Sent false
3845 1639 Acked Read 1 Done Done Sent false
3846 1640 Acked Write 7 Done Done Sent false
3847 1641 Acked Write 8 Done Done Sent false
3848 1642 Acked Write 10 Done Done Sent false
3849 1643 Acked Read 9 Done Done Sent false
3850 1644 Acked Read 3 Done Done Sent false
3851 1645 Acked Write 5 Done Done Sent false
3852 1646 Acked Write 6 Done Done Sent false
3853 1647 Acked Read 8 Done Done Sent false
3854 1648 Acked Write 9 Done Done Sent false
3855 1649 Acked Write 4 Done Done Sent false
3856 1650 Acked Write 5 Done Done Sent false
3857 1651 Acked Read 7 Done Done Sent false
3858 1652 Acked Write 10 Done Done Sent false
3859 1653 Acked Write 2 Done Done Sent false
3860 1654 Acked Write 3 Done Done Sent false
3861 1655 Acked Read 6 Done Done Sent false
3862 1656 Acked Flush 0 Done Done Sent false
3863 1657 Acked Read 2 Done Done Sent false
3864 1658 Acked Flush 0 Done Done Sent false
3865 1659 Acked Write 4 Done Done Sent false
3866 1660 Acked Read 4 Done Done Sent false
3867 1661 Acked Write 5 Done Done Sent false
3868 1662 Acked Write 1 Done Done Sent false
3869 1663 Acked Write 3 Done Done Sent false
3870 1664 Acked Read 3 Done Done Sent false
3871 1665 Acked Write 7 Done Done Sent false
3872 1666 Acked Read 1 Done Done Sent false
3873 1667 Acked Write 7 Done Done Sent false
3874 1668 Acked Read 2 Done Done Sent false
3875 1669 Acked Flush 0 Done Done Sent false
3876 1670 Acked Write 1 Done Done Sent false
3877 1671 Acked Read 8 Done Done Sent false
3878 1672 Acked Read 4 Done Done Sent false
3879 1673 Acked Write 3 Done Done Sent false
3880 1674 Acked Write 5 Done Done Sent false
3881 1675 Acked Read 3 Done Done Sent false
3882 1676 Acked Flush 0 Done Done Sent false
3883 1677 Acked Write 8 Done Done Sent false
3884 1678 Acked Read 2 Done Done Sent false
3885 1679 Acked Write 1 Done Done Sent false
3886 1680 Acked Flush 0 Done Done Sent false
3887 1681 Acked Write 2 Done Done Sent false
3888 1682 Acked Write 1 Done Done Sent false
3889 1683 Acked Read 4 Done Done Sent false
3890 1684 Acked Write 1 Done Done Sent false
3891 1685 Acked Flush 0 Done Done Sent false
3892 1686 Acked Write 7 Done Done Sent false
3893 1687 Acked Write 6 Done Done Sent false
3894 1688 Acked Read 9 Done Done Sent false
3895 1689 Acked Read 3 Done Done Sent false
3896 1690 Acked Write 10 Done Done Sent false
3897 1691 Acked Write 8 Done Done Sent false
3898 1692 Acked Write 9 Done Done Sent false
3899 1693 Acked Read 9 Done Done Sent false
3900 1694 Acked Read 4 Done Done Sent false
3901 1695 Acked Read 10 Done Done Sent false
3902 1696 Acked Write 4 Done Done Sent false
3903 1697 Acked Read 6 Done Done Sent false
3904 1698 Acked Read 2 Done Done Sent false
3905 1699 Acked Read 7 Done Done Sent false
3906 1700 Acked Write 2 Done Done Sent false
3907 1701 Acked Read 2 Done Done Sent false
3908 1702 Acked Write 1 Done Done Sent false
3909 1703 Acked Read 10 Done Done Sent false
3910 1704 Acked Read 10 Done Done Sent false
3911 1705 Acked Read 4 Done Done Sent false
3912 1706 Acked Read 3 Done Done Sent false
3913 1707 Acked Read 2 Done Done Sent false
3914 1708 Acked Flush 0 Done Done Sent false
3915 1709 Acked Read 5 Done Done Sent false
3916 1710 Acked Write 10 Done Done Sent false
3917 1711 Acked Read 9 Done Done Sent false
3918 1712 Acked Write 2 Done Done Sent false
3919 1713 Acked Read 8 Done Done Sent false
3920 1714 Acked Flush 0 Done Done Sent false
3921 1715 Acked Read 1 Done Done Sent false
3922 1716 Acked Read 6 Done Done Sent false
3923 1717 Acked Write 7 Done Done Sent false
3924 1718 Acked Write 2 Done Done Sent false
3925 1719 Acked Read 6 Done Done Sent false
3926 1720 Acked Read 3 Done Done Sent false
3927 1721 Acked Read 2 Done Done Sent false
3928 1722 Acked Write 4 Done Done Sent false
3929 1723 Acked Read 1 Done Done Sent false
3930 1724 Acked Read 6 Done Done Sent false
3931 1725 Acked Write 1 Done Done Sent false
3932 1726 Acked Write 4 Done Done Sent false
3933 1727 Acked Write 2 Done Done Sent false
3934 1728 Acked Read 5 Done Done Sent false
3935 1729 Acked Read 3 Done Done Sent false
3936 1730 Acked Write 6 Done Done Sent false
3937 1731 Acked Write 8 Done Done Sent false
3938 1732 Acked Write 3 Done Done Sent false
3939 1733 Acked Write 10 Done Done Sent false
3940 1734 Acked Flush 0 Done Done Sent false
3941 1735 Acked Write 7 Done Done Sent false
3942 1736 Acked Flush 0 Done Done Sent false
3943 1737 Acked Read 2 Done Done Sent false
3944 1738 Acked Write 8 Done Done Sent false
3945 1739 Acked Write 4 Done Done Sent false
3946 1740 Acked Read 1 Done Done Sent false
3947 1741 Acked Write 1 Done Done Sent false
3948 1742 Acked Read 8 Done Done Sent false
3949 1743 Acked Read 9 Done Done Sent false
3950 1744 Acked Read 2 Done Done Sent false
3951 1745 Acked Read 4 Done Done Sent false
3952 1746 Acked Write 10 Done Done Sent false
3953 1747 Acked Flush 0 Done Done Sent false
3954 1748 Acked Flush 0 Done Done Sent false
3955 1749 Acked Write 3 Done Done Sent false
3956 1750 Acked Write 1 Done Done Sent false
3957 1751 Acked Read 5 Done Done Sent false
3958 1752 Acked Read 1 Done Done Sent false
3959 1753 Acked Write 1 Done Done Sent false
3960 1754 Acked Read 4 Done Done Sent false
3961 1755 Acked Write 9 Done Done Sent false
3962 1756 Acked Read 4 Done Done Sent false
3963 1757 Acked Write 2 Done Done Sent false
3964 1758 Acked Write 9 Done Done Sent false
3965 1759 Acked Read 9 Done Done Sent false
3966 1760 Acked Read 1 Done Done Sent false
3967 1761 Acked Read 6 Done Done Sent false
3968 1762 Acked Write 6 Done Done Sent false
3969 1763 Acked Write 5 Done Done Sent false
3970 1764 Acked Write 3 Done Done Sent false
3971 1765 Acked Read 10 Done Done Sent false
3972 1766 Acked Read 7 Done Done Sent false
3973 1767 Acked Read 2 Done Done Sent false
3974 1768 Acked Read 6 Done Done Sent false
3975 1769 Acked Flush 0 Done Done Sent false
3976 1770 Acked Read 8 Done Done Sent false
3977 1771 Acked Read 4 Done Done Sent false
3978 1772 Acked Write 7 Done Done Sent false
3979 1773 Acked Write 6 Done Done Sent false
3980 1774 Acked Write 9 Done Done Sent false
3981 1775 Acked Write 8 Done Done Sent false
3982 1776 Acked Write 2 Done Done Sent false
3983 1777 Acked Read 6 Done Done Sent false
3984 1778 Acked Write 8 Done Done Sent false
3985 1779 Acked Read 3 Done Done Sent false
3986 1780 Acked Write 9 Done Done Sent false
3987 1781 Acked Read 5 Done Done Sent false
3988 1782 Acked Flush 0 Done Done Sent false
3989 1783 Acked Read 8 Done Done Sent false
3990 1784 Acked Read 8 Done Done Sent false
3991 1785 Acked Write 3 Done Done Sent false
3992 1786 Acked Read 9 Done Done Sent false
3993 1787 Acked Write 9 Done Done Sent false
3994 1788 Acked Read 9 Done Done Sent false
3995 1789 Acked Read 3 Done Done Sent false
3996 1790 Acked Read 4 Done Done Sent false
3997 1791 Acked Flush 0 Done Done Sent false
3998 1792 Acked Read 2 Done Done Sent false
3999 1793 Acked Write 3 Done Done Sent false
4000 1794 Acked Read 7 Done Done Sent false
4001 1795 Acked Write 6 Done Done Sent false
4002 1796 Acked Write 8 Done Done Sent false
4003 1797 Acked Flush 0 Done Done Sent false
4004 1798 Acked Write 9 Done Done Sent false
4005 1799 Acked Write 6 Done Done Sent false
4006 1800 Acked Flush 0 Done Done Sent false
4007 1801 Acked Read 9 Done Done Sent false
4008 1802 Acked Write 6 Done Done Sent false
4009 1803 Acked Write 6 Done Done Sent false
4010 1804 Acked Write 3 Done Done Sent false
4011 1805 Acked Read 8 Done Done Sent false
4012 1806 Acked Read 2 Done Done Sent false
4013 1807 Acked Write 10 Done Done Sent false
4014 1808 Acked Write 6 Done Done Sent false
4015 1809 Acked Write 9 Done Done Sent false
4016 1810 Acked Flush 0 Done Done Sent false
4017 1811 Acked Read 2 Done Done Sent false
4018 1812 Acked Write 8 Done Done Sent false
4019 1813 Acked Write 3 Done Done Sent false
4020 1814 Acked Write 4 Done Done Sent false
4021 1815 Acked Write 7 Done Done Sent false
4022 1816 Acked Read 7 Done Done Sent false
4023 1817 Acked Read 6 Done Done Sent false
4024 1818 Acked Write 2 Done Done Sent false
4025 1819 Acked Write 5 Done Done Sent false
4026 1820 Acked Read 2 Done Done Sent false
4027 1821 Acked Write 9 Done Done Sent false
4028 1822 Acked Read 1 Done Done Sent false
4029 1823 Acked Write 4 Done Done Sent false
4030 1824 Acked Read 8 Done Done Sent false
4031 1825 Acked Read 3 Done Done Sent false
4032 1826 Acked Read 8 Done Done Sent false
4033 1827 Acked Write 10 Done Done Sent false
4034 1828 Acked Flush 0 Done Done Sent false
4035 1829 Acked Read 5 Done Done Sent false
4036 1830 Acked Write 6 Done Done Sent false
4037 1831 Acked Write 3 Done Done Sent false
4038 1832 Acked Flush 0 Done Done Sent false
4039 1833 Acked Write 4 Done Done Sent false
4040 1834 Acked Read 5 Done Done Sent false
4041 1835 Acked Read 9 Done Done Sent false
4042 1836 Acked Flush 0 Done Done Sent false
4043 1837 Acked Read 8 Done Done Sent false
4044 1838 Acked Read 5 Done Done Sent false
4045 1839 Acked Read 3 Done Done Sent false
4046 1840 Acked Read 5 Done Done Sent false
4047 1841 Acked Read 9 Done Done Sent false
4048 1842 Acked Write 7 Done Done Sent false
4049 1843 Acked Read 9 Done Done Sent false
4050 1844 Acked Flush 0 Done Done Sent false
4051 1845 Acked Write 6 Done Done Sent false
4052 1846 Acked Read 4 Done Done Sent false
4053 1847 Acked Read 1 Done Done Sent false
4054 1848 Acked Write 2 Done Done Sent false
4055 1849 Acked Read 10 Done Done Sent false
4056 1850 Acked Write 9 Done Done Sent false
4057 1851 Acked Write 6 Done Done Sent false
4058 1852 Acked Write 5 Done Done Sent false
4059 1853 Acked Read 1 Done Done Sent false
4060 1854 Acked Write 1 Done Done Sent false
4061 1855 Acked Read 3 Done Done Sent false
4062 1856 Acked Flush 0 Done Done Sent false
4063 1857 Acked Write 2 Done Done Sent false
4064 1858 Acked Read 2 Done Done Sent false
4065 1859 Acked Read 3 Done Done Sent false
4066 1860 Acked Flush 0 Done Done Sent false
4067 1861 Acked Read 7 Done Done Sent false
4068 1862 Acked Read 1 Done Done Sent false
4069 1863 Acked Read 9 Done Done Sent false
4070 1864 Acked Write 2 Done Done Sent false
4071 1865 Acked Read 2 Done Done Sent false
4072 1866 Acked Flush 0 Done Done Sent false
4073 1867 Acked Read 4 Done Done Sent false
4074 1868 Acked Read 4 Done Done Sent false
4075 1869 Acked Write 7 Done Done Sent false
4076 1870 Acked Read 8 Done Done Sent false
4077 1871 Acked Read 2 Done Done Sent false
4078 1872 Acked Read 6 Done Done Sent false
4079 1873 Acked Write 9 Done Done Sent false
4080 1874 Acked Read 9 Done Done Sent false
4081 1875 Acked Read 3 Done Done Sent false
4082 1876 Acked Write 5 Done Done Sent false
4083 1877 Acked Read 7 Done Done Sent false
4084 1878 Acked Write 10 Done Done Sent false
4085 1879 Acked Write 2 Done Done Sent false
4086 1880 Acked Write 3 Done Done Sent false
4087 1881 Acked Read 8 Done Done Sent false
4088 1882 Acked Write 3 Done Done Sent false
4089 1883 Acked Read 9 Done Done Sent false
4090 1884 Acked Write 10 Done Done Sent false
4091 1885 Acked Read 4 Done Done Sent false
4092 1886 Acked Read 7 Done Done Sent false
4093 1887 Acked Write 7 Done Done Sent false
4094 1888 Acked Write 1 Done Done Sent false
4095 1889 Acked Read 4 Done Done Sent false
4096 1890 Acked Read 5 Done Done Sent false
4097 1891 Acked Flush 0 Done Done Sent false
4098 1892 Acked Write 7 Done Done Sent false
4099 1893 Acked Write 6 Done Done Sent false
4100 1894 Acked Read 1 Done Done Sent false
4101 1895 Acked Read 2 Done Done Sent false
4102 1896 Acked Read 8 Done Done Sent false
4103 1897 Acked Write 10 Done Done Sent false
4104 1898 Acked Write 7 Done Done Sent false
4105 1899 Acked Flush 0 Done Done Sent false
4106 1900 Acked Read 9 Done Done Sent false
4107 1901 Acked Write 1 Done Done Sent false
4108 1902 Acked Write 10 Done Done Sent false
4109 1903 Acked Write 2 Done Done Sent false
4110 1904 Acked Write 7 Done Done Sent false
4111 STATES DS:0 DS:1 DS:2 TOTAL
4112 Sent 0 0 301 301
4113 Done 301 301 0 602
4114 Skipped 0 0 0 0
4115 Error 0 0 0 0
4116 Last Flush: Some(JobId(1899)) Some(JobId(1899)) Some(JobId(1603))
4117 Downstairs last five completed: 1603 1602 1601 1600 1599
4118 Upstairs last five completed: 1904 1903 1902 1901 1900
4119 CLIENT: Up:0 ds:301 act:2
41202025-03-17T22:19:47.632ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
41212025-03-17T22:19:48.634ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
41222025-03-17T22:19:49.135ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
41232025-03-17T22:19:49.636ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
41242025-03-17T22:19:50.638ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
4125 ----------------------------------------------------------------
4126 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4127 States: Active Active Offline
4128 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4129 1604 Acked Read 9 Done Done Sent false
4130 1605 Acked Read 5 Done Done Sent false
4131 1606 Acked Write 8 Done Done Sent false
4132 1607 Acked Write 2 Done Done Sent false
4133 1608 Acked Read 8 Done Done Sent false
4134 1609 Acked Write 6 Done Done Sent false
4135 1610 Acked Flush 0 Done Done Sent false
4136 1611 Acked Write 7 Done Done Sent false
4137 1612 Acked Read 8 Done Done Sent false
4138 1613 Acked Read 1 Done Done Sent false
4139 1614 Acked Write 4 Done Done Sent false
4140 1615 Acked Write 5 Done Done Sent false
4141 1616 Acked Read 1 Done Done Sent false
4142 1617 Acked Read 6 Done Done Sent false
4143 1618 Acked Read 3 Done Done Sent false
4144 1619 Acked Read 7 Done Done Sent false
4145 1620 Acked Write 6 Done Done Sent false
4146 1621 Acked Read 9 Done Done Sent false
4147 1622 Acked Write 7 Done Done Sent false
4148 1623 Acked Write 10 Done Done Sent false
4149 1624 Acked Write 9 Done Done Sent false
4150 1625 Acked Write 4 Done Done Sent false
4151 1626 Acked Read 9 Done Done Sent false
4152 1627 Acked Flush 0 Done Done Sent false
4153 1628 Acked Write 8 Done Done Sent false
4154 1629 Acked Read 8 Done Done Sent false
4155 1630 Acked Write 8 Done Done Sent false
4156 1631 Acked Read 3 Done Done Sent false
4157 1632 Acked Write 9 Done Done Sent false
4158 1633 Acked Read 1 Done Done Sent false
4159 1634 Acked Write 2 Done Done Sent false
4160 1635 Acked Write 6 Done Done Sent false
4161 1636 Acked Write 9 Done Done Sent false
4162 1637 Acked Write 5 Done Done Sent false
4163 1638 Acked Write 7 Done Done Sent false
4164 1639 Acked Read 1 Done Done Sent false
4165 1640 Acked Write 7 Done Done Sent false
4166 1641 Acked Write 8 Done Done Sent false
4167 1642 Acked Write 10 Done Done Sent false
4168 1643 Acked Read 9 Done Done Sent false
4169 1644 Acked Read 3 Done Done Sent false
4170 1645 Acked Write 5 Done Done Sent false
4171 1646 Acked Write 6 Done Done Sent false
4172 1647 Acked Read 8 Done Done Sent false
4173 1648 Acked Write 9 Done Done Sent false
4174 1649 Acked Write 4 Done Done Sent false
4175 1650 Acked Write 5 Done Done Sent false
4176 1651 Acked Read 7 Done Done Sent false
4177 1652 Acked Write 10 Done Done Sent false
4178 1653 Acked Write 2 Done Done Sent false
4179 1654 Acked Write 3 Done Done Sent false
4180 1655 Acked Read 6 Done Done Sent false
4181 1656 Acked Flush 0 Done Done Sent false
4182 1657 Acked Read 2 Done Done Sent false
4183 1658 Acked Flush 0 Done Done Sent false
4184 1659 Acked Write 4 Done Done Sent false
4185 1660 Acked Read 4 Done Done Sent false
4186 1661 Acked Write 5 Done Done Sent false
4187 1662 Acked Write 1 Done Done Sent false
4188 1663 Acked Write 3 Done Done Sent false
4189 1664 Acked Read 3 Done Done Sent false
4190 1665 Acked Write 7 Done Done Sent false
4191 1666 Acked Read 1 Done Done Sent false
4192 1667 Acked Write 7 Done Done Sent false
4193 1668 Acked Read 2 Done Done Sent false
4194 1669 Acked Flush 0 Done Done Sent false
4195 1670 Acked Write 1 Done Done Sent false
4196 1671 Acked Read 8 Done Done Sent false
4197 1672 Acked Read 4 Done Done Sent false
4198 1673 Acked Write 3 Done Done Sent false
4199 1674 Acked Write 5 Done Done Sent false
4200 1675 Acked Read 3 Done Done Sent false
4201 1676 Acked Flush 0 Done Done Sent false
4202 1677 Acked Write 8 Done Done Sent false
4203 1678 Acked Read 2 Done Done Sent false
4204 1679 Acked Write 1 Done Done Sent false
4205 1680 Acked Flush 0 Done Done Sent false
4206 1681 Acked Write 2 Done Done Sent false
4207 1682 Acked Write 1 Done Done Sent false
4208 1683 Acked Read 4 Done Done Sent false
4209 1684 Acked Write 1 Done Done Sent false
4210 1685 Acked Flush 0 Done Done Sent false
4211 1686 Acked Write 7 Done Done Sent false
4212 1687 Acked Write 6 Done Done Sent false
4213 1688 Acked Read 9 Done Done Sent false
4214 1689 Acked Read 3 Done Done Sent false
4215 1690 Acked Write 10 Done Done Sent false
4216 1691 Acked Write 8 Done Done Sent false
4217 1692 Acked Write 9 Done Done Sent false
4218 1693 Acked Read 9 Done Done Sent false
4219 1694 Acked Read 4 Done Done Sent false
4220 1695 Acked Read 10 Done Done Sent false
4221 1696 Acked Write 4 Done Done Sent false
4222 1697 Acked Read 6 Done Done Sent false
4223 1698 Acked Read 2 Done Done Sent false
4224 1699 Acked Read 7 Done Done Sent false
4225 1700 Acked Write 2 Done Done Sent false
4226 1701 Acked Read 2 Done Done Sent false
4227 1702 Acked Write 1 Done Done Sent false
4228 1703 Acked Read 10 Done Done Sent false
4229 1704 Acked Read 10 Done Done Sent false
4230 1705 Acked Read 4 Done Done Sent false
4231 1706 Acked Read 3 Done Done Sent false
4232 1707 Acked Read 2 Done Done Sent false
4233 1708 Acked Flush 0 Done Done Sent false
4234 1709 Acked Read 5 Done Done Sent false
4235 1710 Acked Write 10 Done Done Sent false
4236 1711 Acked Read 9 Done Done Sent false
4237 1712 Acked Write 2 Done Done Sent false
4238 1713 Acked Read 8 Done Done Sent false
4239 1714 Acked Flush 0 Done Done Sent false
4240 1715 Acked Read 1 Done Done Sent false
4241 1716 Acked Read 6 Done Done Sent false
4242 1717 Acked Write 7 Done Done Sent false
4243 1718 Acked Write 2 Done Done Sent false
4244 1719 Acked Read 6 Done Done Sent false
4245 1720 Acked Read 3 Done Done Sent false
4246 1721 Acked Read 2 Done Done Sent false
4247 1722 Acked Write 4 Done Done Sent false
4248 1723 Acked Read 1 Done Done Sent false
4249 1724 Acked Read 6 Done Done Sent false
4250 1725 Acked Write 1 Done Done Sent false
4251 1726 Acked Write 4 Done Done Sent false
4252 1727 Acked Write 2 Done Done Sent false
4253 1728 Acked Read 5 Done Done Sent false
4254 1729 Acked Read 3 Done Done Sent false
4255 1730 Acked Write 6 Done Done Sent false
4256 1731 Acked Write 8 Done Done Sent false
4257 1732 Acked Write 3 Done Done Sent false
4258 1733 Acked Write 10 Done Done Sent false
4259 1734 Acked Flush 0 Done Done Sent false
4260 1735 Acked Write 7 Done Done Sent false
4261 1736 Acked Flush 0 Done Done Sent false
4262 1737 Acked Read 2 Done Done Sent false
4263 1738 Acked Write 8 Done Done Sent false
4264 1739 Acked Write 4 Done Done Sent false
4265 1740 Acked Read 1 Done Done Sent false
4266 1741 Acked Write 1 Done Done Sent false
4267 1742 Acked Read 8 Done Done Sent false
4268 1743 Acked Read 9 Done Done Sent false
4269 1744 Acked Read 2 Done Done Sent false
4270 1745 Acked Read 4 Done Done Sent false
4271 1746 Acked Write 10 Done Done Sent false
4272 1747 Acked Flush 0 Done Done Sent false
4273 1748 Acked Flush 0 Done Done Sent false
4274 1749 Acked Write 3 Done Done Sent false
4275 1750 Acked Write 1 Done Done Sent false
4276 1751 Acked Read 5 Done Done Sent false
4277 1752 Acked Read 1 Done Done Sent false
4278 1753 Acked Write 1 Done Done Sent false
4279 1754 Acked Read 4 Done Done Sent false
4280 1755 Acked Write 9 Done Done Sent false
4281 1756 Acked Read 4 Done Done Sent false
4282 1757 Acked Write 2 Done Done Sent false
4283 1758 Acked Write 9 Done Done Sent false
4284 1759 Acked Read 9 Done Done Sent false
4285 1760 Acked Read 1 Done Done Sent false
4286 1761 Acked Read 6 Done Done Sent false
4287 1762 Acked Write 6 Done Done Sent false
4288 1763 Acked Write 5 Done Done Sent false
4289 1764 Acked Write 3 Done Done Sent false
4290 1765 Acked Read 10 Done Done Sent false
4291 1766 Acked Read 7 Done Done Sent false
4292 1767 Acked Read 2 Done Done Sent false
4293 1768 Acked Read 6 Done Done Sent false
4294 1769 Acked Flush 0 Done Done Sent false
4295 1770 Acked Read 8 Done Done Sent false
4296 1771 Acked Read 4 Done Done Sent false
4297 1772 Acked Write 7 Done Done Sent false
4298 1773 Acked Write 6 Done Done Sent false
4299 1774 Acked Write 9 Done Done Sent false
4300 1775 Acked Write 8 Done Done Sent false
4301 1776 Acked Write 2 Done Done Sent false
4302 1777 Acked Read 6 Done Done Sent false
4303 1778 Acked Write 8 Done Done Sent false
4304 1779 Acked Read 3 Done Done Sent false
4305 1780 Acked Write 9 Done Done Sent false
4306 1781 Acked Read 5 Done Done Sent false
4307 1782 Acked Flush 0 Done Done Sent false
4308 1783 Acked Read 8 Done Done Sent false
4309 1784 Acked Read 8 Done Done Sent false
4310 1785 Acked Write 3 Done Done Sent false
4311 1786 Acked Read 9 Done Done Sent false
4312 1787 Acked Write 9 Done Done Sent false
4313 1788 Acked Read 9 Done Done Sent false
4314 1789 Acked Read 3 Done Done Sent false
4315 1790 Acked Read 4 Done Done Sent false
4316 1791 Acked Flush 0 Done Done Sent false
4317 1792 Acked Read 2 Done Done Sent false
4318 1793 Acked Write 3 Done Done Sent false
4319 1794 Acked Read 7 Done Done Sent false
4320 1795 Acked Write 6 Done Done Sent false
4321 1796 Acked Write 8 Done Done Sent false
4322 1797 Acked Flush 0 Done Done Sent false
4323 1798 Acked Write 9 Done Done Sent false
4324 1799 Acked Write 6 Done Done Sent false
4325 1800 Acked Flush 0 Done Done Sent false
4326 1801 Acked Read 9 Done Done Sent false
4327 1802 Acked Write 6 Done Done Sent false
4328 1803 Acked Write 6 Done Done Sent false
4329 1804 Acked Write 3 Done Done Sent false
4330 1805 Acked Read 8 Done Done Sent false
4331 1806 Acked Read 2 Done Done Sent false
4332 1807 Acked Write 10 Done Done Sent false
4333 1808 Acked Write 6 Done Done Sent false
4334 1809 Acked Write 9 Done Done Sent false
4335 1810 Acked Flush 0 Done Done Sent false
4336 1811 Acked Read 2 Done Done Sent false
4337 1812 Acked Write 8 Done Done Sent false
4338 1813 Acked Write 3 Done Done Sent false
4339 1814 Acked Write 4 Done Done Sent false
4340 1815 Acked Write 7 Done Done Sent false
4341 1816 Acked Read 7 Done Done Sent false
4342 1817 Acked Read 6 Done Done Sent false
4343 1818 Acked Write 2 Done Done Sent false
4344 1819 Acked Write 5 Done Done Sent false
4345 1820 Acked Read 2 Done Done Sent false
4346 1821 Acked Write 9 Done Done Sent false
4347 1822 Acked Read 1 Done Done Sent false
4348 1823 Acked Write 4 Done Done Sent false
4349 1824 Acked Read 8 Done Done Sent false
4350 1825 Acked Read 3 Done Done Sent false
4351 1826 Acked Read 8 Done Done Sent false
4352 1827 Acked Write 10 Done Done Sent false
4353 1828 Acked Flush 0 Done Done Sent false
4354 1829 Acked Read 5 Done Done Sent false
4355 1830 Acked Write 6 Done Done Sent false
4356 1831 Acked Write 3 Done Done Sent false
4357 1832 Acked Flush 0 Done Done Sent false
4358 1833 Acked Write 4 Done Done Sent false
4359 1834 Acked Read 5 Done Done Sent false
4360 1835 Acked Read 9 Done Done Sent false
4361 1836 Acked Flush 0 Done Done Sent false
4362 1837 Acked Read 8 Done Done Sent false
4363 1838 Acked Read 5 Done Done Sent false
4364 1839 Acked Read 3 Done Done Sent false
4365 1840 Acked Read 5 Done Done Sent false
4366 1841 Acked Read 9 Done Done Sent false
4367 1842 Acked Write 7 Done Done Sent false
4368 1843 Acked Read 9 Done Done Sent false
4369 1844 Acked Flush 0 Done Done Sent false
4370 1845 Acked Write 6 Done Done Sent false
4371 1846 Acked Read 4 Done Done Sent false
4372 1847 Acked Read 1 Done Done Sent false
4373 1848 Acked Write 2 Done Done Sent false
4374 1849 Acked Read 10 Done Done Sent false
4375 1850 Acked Write 9 Done Done Sent false
4376 1851 Acked Write 6 Done Done Sent false
4377 1852 Acked Write 5 Done Done Sent false
4378 1853 Acked Read 1 Done Done Sent false
4379 1854 Acked Write 1 Done Done Sent false
4380 1855 Acked Read 3 Done Done Sent false
4381 1856 Acked Flush 0 Done Done Sent false
4382 1857 Acked Write 2 Done Done Sent false
4383 1858 Acked Read 2 Done Done Sent false
4384 1859 Acked Read 3 Done Done Sent false
4385 1860 Acked Flush 0 Done Done Sent false
4386 1861 Acked Read 7 Done Done Sent false
4387 1862 Acked Read 1 Done Done Sent false
4388 1863 Acked Read 9 Done Done Sent false
4389 1864 Acked Write 2 Done Done Sent false
4390 1865 Acked Read 2 Done Done Sent false
4391 1866 Acked Flush 0 Done Done Sent false
4392 1867 Acked Read 4 Done Done Sent false
4393 1868 Acked Read 4 Done Done Sent false
4394 1869 Acked Write 7 Done Done Sent false
4395 1870 Acked Read 8 Done Done Sent false
4396 1871 Acked Read 2 Done Done Sent false
4397 1872 Acked Read 6 Done Done Sent false
4398 1873 Acked Write 9 Done Done Sent false
4399 1874 Acked Read 9 Done Done Sent false
4400 1875 Acked Read 3 Done Done Sent false
4401 1876 Acked Write 5 Done Done Sent false
4402 1877 Acked Read 7 Done Done Sent false
4403 1878 Acked Write 10 Done Done Sent false
4404 1879 Acked Write 2 Done Done Sent false
4405 1880 Acked Write 3 Done Done Sent false
4406 1881 Acked Read 8 Done Done Sent false
4407 1882 Acked Write 3 Done Done Sent false
4408 1883 Acked Read 9 Done Done Sent false
4409 1884 Acked Write 10 Done Done Sent false
4410 1885 Acked Read 4 Done Done Sent false
4411 1886 Acked Read 7 Done Done Sent false
4412 1887 Acked Write 7 Done Done Sent false
4413 1888 Acked Write 1 Done Done Sent false
4414 1889 Acked Read 4 Done Done Sent false
4415 1890 Acked Read 5 Done Done Sent false
4416 1891 Acked Flush 0 Done Done Sent false
4417 1892 Acked Write 7 Done Done Sent false
4418 1893 Acked Write 6 Done Done Sent false
4419 1894 Acked Read 1 Done Done Sent false
4420 1895 Acked Read 2 Done Done Sent false
4421 1896 Acked Read 8 Done Done Sent false
4422 1897 Acked Write 10 Done Done Sent false
4423 1898 Acked Write 7 Done Done Sent false
4424 1899 Acked Flush 0 Done Done Sent false
4425 1900 Acked Read 9 Done Done Sent false
4426 1901 Acked Write 1 Done Done Sent false
4427 1902 Acked Write 10 Done Done Sent false
4428 1903 Acked Write 2 Done Done Sent false
4429 1904 Acked Write 7 Done Done Sent false
4430 STATES DS:0 DS:1 DS:2 TOTAL
4431 Sent 0 0 301 301
4432 Done 301 301 0 602
4433 Skipped 0 0 0 0
4434 Error 0 0 0 0
4435 Last Flush: Some(JobId(1899)) Some(JobId(1899)) Some(JobId(1603))
4436 Downstairs last five completed: 1603 1602 1601 1600 1599
4437 Upstairs last five completed: 1904 1903 1902 1901 1900
4438 CLIENT: Up:0 ds:301 act:2
44392025-03-17T22:19:51.640ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
44402025-03-17T22:19:52.640ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
44412025-03-17T22:19:53.641ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
44422025-03-17T22:19:54.601ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
44432025-03-17T22:19:54.601ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:60927) = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
44442025-03-17T22:19:54.602ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
44452025-03-17T22:19:54.603ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
44462025-03-17T22:19:54.603ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
44472025-03-17T22:19:54.603ZINFOcrucible: send last flush ID to this DS: Some(JobId(1603)) = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
44482025-03-17T22:19:54.603ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
44492025-03-17T22:19:54.603ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(1603)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
44502025-03-17T22:19:54.603ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(1603)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
4451 ----------------------------------------------------------------
4452 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:5
4453 States: Active Active Active
4454 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4455 1900 Acked Read 9 Done Done Done true
4456 1901 Acked Write 1 Done Done Done true
4457 1902 Acked Write 10 Done Done Done true
4458 1903 Acked Write 2 Done Done Done true
4459 1904 Acked Write 7 Done Done Done true
4460 STATES DS:0 DS:1 DS:2 TOTAL
4461 Sent 0 0 0 0
4462 Done 5 5 5 15
4463 Skipped 0 0 0 0
4464 Error 0 0 0 0
4465 Last Flush: Some(JobId(1899)) Some(JobId(1899)) Some(JobId(1899))
4466 Downstairs last five completed: 1899 1898 1897 1896 1895
4467 Upstairs last five completed: 1904 1903 1902 1901 1900
4468 CLIENT: Up:0 ds:5 act:3
4469 ----------------------------------------------------------------
4470 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
4471 Downstairs last five completed: 1905 1904 1903 1902 1901
4472 Upstairs last five completed: 1905 1904 1903 1902 1901
4473 CLIENT: Up:0 ds:0 act:3
4474 Replay: All jobs finished
4475 Main task has work to do, go find it
4476 got dsc Stop(1)
4477 stop 1
4478 Mar 17 22:19:58.840 INFO request completed, latency_us: 302, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 525bb424-e0ec-4736-bed1-e8ed94466c41, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
4479 [8820] Got stop action so:false kr:false
44802025-03-17T22:19:58.841ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
44812025-03-17T22:19:58.841ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
4482 Mar 17 22:19:58.841 INFO request completed, latency_us: 151, response_code:{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:19:58.841417622Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
4483 200, uri: /state/cid/1, method: GET, req_id: e78b62bb-f850-482c-8dbd-b1148f2c428a, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
44842025-03-17T22:19:58.841ZINFOcrucible: client task is exiting = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
4485 [8820] Exited with: ExitStatus(unix_wait_status(9))
4486 [8820][1] reports Exit
4487 Mar 17 22:20:02.842 INFO request completed, latency_us: 219, response_code: 200, uri: /state/cid/1, method: GET, req_id: a71555c8-acaf-4042-b5c9-ad3c326d2b37, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
4488 001/300 Flush
4489 002/300 Read block 3765 len 12288
4490 003/300 Read block 4071 len 40960
4491 004/300 Write block 4482 len 36864 data: 4 4 4 3 3 3 2 2 2
4492 005/300 Read block 4719 len 40960
4493 006/300 Write block 399 len 28672 data: 2 2 2 2 2 2 2
4494 007/300 Read block 4357 len 40960
4495 008/300 Flush
4496 009/300 Write block 1864 len 20480 data: 3 3 5 5 4
4497 010/300 Read block 3116 len 28672
4498 011/300 Write block 1141 len 8192 data: 2 2
4499 012/300 Write block 555 len 4096 data: 3
4500 013/300 Write block 3971 len 24576 data: 3 4 4 3 3 3
4501 014/300 Flush
4502 015/300 Read block 325 len 16384
4503 016/300 Read block 4499 len 28672
4504 017/300 Write block 4526 len 36864 data: 2 2 2 2 2 3 3 3 3
4505 018/300 Read block 7 len 4096
4506 019/300 Read block 2455 len 16384
4507 020/300 Read block 3648 len 36864
4508 021/300 Write block 944 len 32768 data: 2 3 3 3 3 3 3 3
4509 022/300 Write block 2842 len 24576 data: 2 2 2 2 2 2
4510 023/300 Write block 670 len 4096 data: 2
4511 024/300 Read block 764 len 4096
4512 025/300 Flush
4513 026/300 Flush
4514 027/300 Read block 382 len 24576
4515 028/300 Read block 1465 len 8192
4516 029/300 Read block 831 len 36864
4517 030/300 Read block 1552 len 28672
4518 031/300 Read block 4814 len 32768
4519 032/300 Flush
4520 033/300 Write block 744 len 24576 data: 3 2 2 2 2 2
4521 034/300 Write block 3399 len 40960 data: 2 2 2 2 2 2 2 3 2 2
4522 035/300 Write block 2870 len 8192 data: 2 2
4523 036/300 Flush
4524 037/300 Read block 4614 len 40960
4525 038/300 Write block 1602 len 20480 data: 3 3 3 3 3
4526 039/300 Read block 1849 len 24576
4527 040/300 Flush
4528 041/300 Read block 1791 len 32768
4529 042/300 Read block 3643 len 28672
4530 043/300 Read block 3091 len 16384
4531 044/300 Write block 1045 len 28672 data: 2 2 2 2 2 2 2
4532 045/300 Write block 4160 len 16384 data: 2 2 2 2
4533 046/300 Write block 2636 len 20480 data: 2 2 2 2 2
4534 047/300 Read block 426 len 8192
4535 048/300 Read block 3991 len 40960
4536 049/300 Write block 3713 len 8192 data: 2 2
4537 050/300 Read block 2630 len 8192
4538 051/300 Read block 1581 len 4096
4539 052/300 Flush
4540 053/300 Read block 2735 len 40960
4541 054/300 Write block 1966 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4542 055/300 Read block 2108 len 32768
4543 056/300 Write block 4620 len 24576 data: 2 2 2 2 2 2
4544 057/300 Write block 1396 len 16384 data: 2 2 2 2
4545 058/300 Flush
4546 059/300 Write block 3248 len 24576 data: 4 4 4 4 3 3
4547 060/300 Flush
4548 061/300 Read block 3227 len 8192
4549 062/300 Read block 768 len 4096
4550 063/300 Write block 4191 len 8192 data: 3 2
4551 064/300 Write block 2568 len 12288 data: 2 2 2
4552 065/300 Read block 2238 len 24576
4553 066/300 Write block 2875 len 40960 data: 3 3 3 3 3 3 3 2 2 2
4554 067/300 Read block 3237 len 8192
4555 068/300 Read block 85 len 16384
4556 069/300 Read block 3469 len 24576
4557 070/300 Write block 4101 len 24576 data: 2 2 2 2 2 2
4558 071/300 Read block 2033 len 12288
4559 072/300 Write block 2939 len 16384 data: 2 2 2 2
4560 073/300 Read block 564 len 36864
4561 074/300 Flush
4562 075/300 Write block 3280 len 24576 data: 3 3 3 3 3 3
4563 076/300 Read block 1339 len 12288
4564 077/300 Read block 2181 len 16384
4565 078/300 Write block 1270 len 28672 data: 2 2 3 3 3 3 2
4566 079/300 Write block 2227 len 32768 data: 3 2 2 2 2 2 2 2
4567 080/300 Write block 418 len 4096 data: 3
4568 081/300 Write block 85 len 16384 data: 2 2 2 3
4569 082/300 Read block 2024 len 28672
4570 083/300 Read block 1245 len 20480
4571 084/300 Write block 4480 len 24576 data: 4 4 5 5 5 4
4572 085/300 Write block 3535 len 8192 data: 4 4
4573 086/300 Read block 3151 len 20480
4574 087/300 Write block 1108 len 24576 data: 2 3 3 2 2 2
4575 088/300 Read block 2229 len 36864
4576 089/300 Read block 1082 len 28672
4577 090/300 Write block 808 len 32768 data: 2 3 3 3 3 3 3 2
4578 091/300 Read block 944 len 36864
4579 092/300 Read block 1866 len 8192
4580 093/300 Read block 4251 len 36864
4581 094/300 Read block 3349 len 36864
4582 095/300 Write block 419 len 4096 data: 3
4583 096/300 Write block 3961 len 12288 data: 3 3 2
4584 097/300 Write block 4028 len 4096 data: 2
4585 098/300 Flush
4586 099/300 Read block 4030 len 36864
4587 100/300 Write block 4961 len 12288 data: 3 3 3
4588 101/300 Write block 3575 len 32768 data: 2 2 2 3 3 2 2 2
4589 102/300 Read block 2876 len 20480
4590 103/300 Read block 4868 len 8192
4591 104/300 Write block 4804 len 8192 data: 3 3
4592 105/300 Read block 2985 len 32768
4593 106/300 Flush
4594 107/300 Write block 4142 len 16384 data: 2 3 3 4
4595 108/300 Read block 919 len 40960
4596 109/300 Write block 4123 len 8192 data: 2 2
4597 110/300 Read block 1219 len 32768
4598 111/300 Write block 2433 len 8192 data: 3 3
4599 112/300 Write block 4862 len 20480 data: 2 2 2 2 2
4600 113/300 Flush
4601 114/300 Write block 4640 len 32768 data: 2 3 3 3 3 3 3 3
4602 115/300 Flush
4603 116/300 Write block 3564 len 12288 data: 2 2 2
4604 117/300 Read block 491 len 16384
4605 118/300 Write block 3284 len 12288 data: 4 4 3
4606 119/300 Read block 3386 len 12288
4607 120/300 Read block 1573 len 20480
4608 121/300 Read block 3119 len 32768
4609 122/300 Read block 532 len 16384
4610 123/300 Read block 149 len 4096
4611 124/300 Read block 2659 len 36864
4612 125/300 Flush
4613 126/300 Read block 3807 len 28672
4614 127/300 Read block 4302 len 8192
4615 128/300 Write block 982 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4616 129/300 Flush
4617 130/300 Write block 4454 len 8192 data: 2 2
4618 131/300 Flush
4619 132/300 Read block 2085 len 4096
4620 133/300 Write block 2571 len 32768 data: 2 2 2 2 2 2 3 3
4621 134/300 Write block 389 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4622 135/300 Write block 493 len 12288 data: 2 2 2
4623 136/300 Read block 1625 len 12288
4624 137/300 Read block 4466 len 36864
4625 138/300 Write block 1069 len 28672 data: 2 2 2 2 2 3 4
4626 139/300 Read block 4593 len 12288
4627 140/300 Write block 1749 len 32768 data: 3 3 3 4 3 4 4 4
4628 141/300 Read block 634 len 20480
4629 142/300 Write block 781 len 32768 data: 4 4 4 3 3 3 3 3
4630 143/300 Write block 2314 len 24576 data: 5 4 3 3 3 2
4631 144/300 Read block 1368 len 4096
4632 145/300 Read block 2804 len 20480
4633 146/300 Write block 3693 len 16384 data: 3 2 2 2
4634 147/300 Read block 539 len 40960
4635 148/300 Read block 2391 len 16384
4636 149/300 Write block 3201 len 24576 data: 4 3 3 3 2 2
4637 150/300 Write block 4026 len 24576 data: 2 2 3 2 2 2
4638 151/300 Write block 1856 len 28672 data: 2 2 2 2 2 2 2
4639 152/300 Read block 2901 len 16384
4640 153/300 Write block 2541 len 36864 data: 3 2 2 2 3 3 3 3 3
4641 154/300 Read block 4867 len 24576
4642 155/300 Read block 4268 len 28672
4643 156/300 Read block 2081 len 24576
4644 157/300 Read block 4474 len 12288
4645 158/300 Write block 2853 len 36864 data: 3 2 2 2 2 2 2 2 2
4646 159/300 Write block 44 len 20480 data: 3 3 2 2 2
4647 160/300 Write block 1361 len 8192 data: 2 2
4648 161/300 Write block 603 len 4096 data: 2
4649 162/300 Read block 875 len 4096
4650 163/300 Write block 4031 len 8192 data: 3 2
4651 164/300 Read block 2044 len 16384
4652 165/300 Read block 1672 len 32768
4653 166/300 Read block 4796 len 16384
4654 167/300 Flush
4655 168/300 Write block 4576 len 36864 data: 2 2 2 2 2 2 2 2 2
4656 169/300 Write block 3461 len 16384 data: 2 2 2 2
4657 170/300 Read block 2794 len 20480
4658 171/300 Write block 3039 len 16384 data: 4 4 4 4
4659 172/300 Write block 2921 len 8192 data: 2 2
4660 173/300 Read block 3628 len 4096
4661 174/300 Read block 1538 len 36864
4662 175/300 Write block 1488 len 24576 data: 2 2 2 2 2 2
4663 176/300 Write block 1709 len 32768 data: 2 4 3 3 3 3 3 3
4664 177/300 Write block 1092 len 20480 data: 2 2 2 2 2
4665 178/300 Flush
4666 179/300 Flush
4667 180/300 Read block 2232 len 24576
4668 181/300 Write block 3055 len 40960 data: 5 4 4 3 2 2 2 2 2 2
4669 182/300 Read block 896 len 36864
4670 183/300 Write block 152 len 24576 data: 2 3 3 3 3 3
4671 184/300 Flush
4672 185/300 Write block 4187 len 4096 data: 2
4673 186/300 Write block 2637 len 28672 data: 3 3 3 3 2 2 2
4674 187/300 Write block 764 len 16384 data: 2 2 2 2
4675 188/300 Write block 3767 len 12288 data: 2 2 2
4676 189/300 Write block 2203 len 40960 data: 2 2 2 2 2 2 2 2 3 3
4677 190/300 Write block 3230 len 12288 data: 2 2 2
4678 191/300 Read block 216 len 20480
4679 192/300 Write block 3113 len 36864 data: 3 3 2 2 2 2 2 2 2
4680 193/300 Write block 4641 len 32768 data: 4 4 4 4 4 4 4 3
4681 194/300 Read block 549 len 8192
4682 195/300 Read block 4317 len 16384
4683 196/300 Write block 3184 len 28672 data: 2 2 2 2 2 2 2
4684 197/300 Read block 3289 len 12288
4685 198/300 Flush
4686 199/300 Read block 3329 len 12288
4687 200/300 Read block 1372 len 20480
4688 201/300 Read block 154 len 16384
4689 202/300 Read block 3237 len 12288
4690 203/300 Read block 4868 len 12288
4691 204/300 Read block 3536 len 20480
4692 205/300 Read block 1699 len 32768
4693 206/300 Read block 3401 len 4096
4694 207/300 Write block 3396 len 32768 data: 2 2 2 3 3 3 3 3
4695 208/300 Read block 2873 len 40960
4696 209/300 Read block 577 len 20480
4697 210/300 Read block 1092 len 20480
4698 211/300 Write block 4341 len 28672 data: 2 2 2 2 2 2 2
4699 212/300 Read block 1273 len 8192
4700 213/300 Read block 486 len 8192
4701 214/300 Read block 3988 len 8192
4702 215/300 Read block 642 len 16384
4703 216/300 Write block 2399 len 36864 data: 3 3 3 3 2 2 2 2 2
4704 217/300 Read block 3742 len 40960
4705 218/300 Write block 503 len 40960 data: 2 2 2 2 3 3 3 2 2 2
4706 219/300 Read block 279 len 8192
4707 220/300 Write block 4748 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4708 221/300 Read block 2689 len 4096
4709 222/300 Flush
4710 223/300 Write block 202 len 32768 data: 2 2 2 2 2 2 2 2
4711 224/300 Read block 4662 len 8192
4712 225/300 Read block 1760 len 28672
4713 226/300 Write block 3153 len 24576 data: 2 2 2 2 2 2
4714 227/300 Read block 695 len 12288
4715 228/300 Read block 2674 len 24576
4716 229/300 Write block 4697 len 20480 data: 3 3 3 3 3
4717 230/300 Write block 520 len 8192 data: 3 3
4718 231/300 Read block 4419 len 28672
4719 232/300 Write block 905 len 36864 data: 2 2 2 3 3 4 3 2 2
4720 233/300 Read block 1377 len 16384
4721 234/300 Write block 539 len 4096 data: 3
4722 235/300 Write block 4272 len 24576 data: 3 3 2 2 2 2
4723 236/300 Read block 2773 len 24576
4724 237/300 Read block 760 len 12288
4725 238/300 Read block 4076 len 28672
4726 239/300 Read block 1597 len 24576
4727 240/300 Read block 3147 len 12288
4728 241/300 Read block 1968 len 12288
4729 242/300 Read block 4476 len 36864
4730 243/300 Write block 61 len 16384 data: 3 3 3 3
4731 244/300 Write block 1774 len 20480 data: 2 2 2 2 2
4732 245/300 Write block 1167 len 32768 data: 2 2 2 2 2 2 2 2
4733 246/300 Write block 3861 len 4096 data: 2
4734 247/300 Read block 4714 len 24576
4735 248/300 Read block 818 len 32768
4736 249/300 Write block 1757 len 12288 data: 4 5 4
4737 250/300 Write block 1344 len 28672 data: 2 2 2 3 3 3 3
4738 251/300 Write block 163 len 28672 data: 3 3 2 2 2 2 2
4739 252/300 Read block 1895 len 36864
4740 253/300 Read block 1335 len 32768
4741 254/300 Read block 1939 len 4096
4742 255/300 Write block 1799 len 24576 data: 2 2 2 2 2 2
4743 256/300 Write block 1686 len 20480 data: 2 2 2 2 2
4744 257/300 Write block 778 len 36864 data: 4 4 4 5 5 5 4 4 4
4745 258/300 Write block 1578 len 12288 data: 2 3 2
4746 259/300 Write block 3321 len 16384 data: 2 2 2 2
4747 260/300 Write block 2032 len 16384 data: 2 2 2 2
4748 261/300 Flush
4749 262/300 Read block 1875 len 24576
4750 263/300 Write block 3858 len 40960 data: 2 2 2 3 2 2 2 2 2 2
4751 264/300 Read block 4235 len 12288
4752 265/300 Flush
4753 266/300 Write block 3730 len 20480 data: 2 2 2 2 2
4754 267/300 Write block 753 len 32768 data: 2 2 2 2 2 2 2 2
4755 268/300 Read block 2291 len 4096
4756 269/300 Flush
4757 270/300 Write block 1670 len 8192 data: 2 2
4758 271/300 Read block 301 len 12288
4759 272/300 Read block 4813 len 16384
4760 273/300 Read block 776 len 20480
4761 274/300 Write block 1132 len 8192 data: 2 2
4762 275/300 Write block 2417 len 24576 data: 2 3 3 3 3 3
4763 276/300 Read block 893 len 40960
4764 277/300 Read block 845 len 28672
4765 278/300 Write block 4714 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4766 279/300 Read block 909 len 28672
4767 280/300 Read block 429 len 20480
4768 281/300 Write block 3180 len 32768 data: 3 3 2 2 3 3 3 3
4769 282/300 Flush
4770 283/300 Read block 4736 len 40960
4771 284/300 Read block 1442 len 28672
4772 285/300 Read block 2076 len 36864
4773 286/300 Read block 1907 len 40960
4774 287/300 Read block 3100 len 20480
4775 288/300 Read block 3849 len 28672
4776 289/300 Write block 3490 len 28672 data: 3 3 2 2 2 2 2
4777 290/300 Read block 1509 len 4096
4778 291/300 Read block 3169 len 4096
4779 292/300 Read block 3668 len 32768
4780 293/300 Write block 1839 len 16384 data: 3 3 3 2
4781 294/300 Write block 573 len 32768 data: 2 2 2 2 2 2 2 3
4782 295/300 Write block 3515 len 8192 data: 2 3
4783 296/300 Write block 2918 len 8192 data: 2 2
4784 297/300 Read block 4480 len 12288
4785 298/300 Read block 1554 len 16384
4786 299/300 Write block 4946 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4787 300/300 Read block 336 len 8192
4788 301/300 Write block 4885 len 8192 data: 2 2
4789 Main task has work to do, go find it
4790 got dsc Start(1)
4791 start 1
4792 Mar 17 22:20:03.079 INFO request completed, latency_us: 233, response_code: 204, uri: /start/cid/1, method: POST, req_id: a2313524-e06c-4141-8e2e-f646bb63df91, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
4793 Replay: started 1, returned:Ok(())
4794 ----------------------------------------------------------------
4795 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4796 States: Active Offline Active
4797 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4798 1906 Acked Flush 0 Done Sent Done false
4799 1907 Acked Read 3 Done Sent Done false
4800 1908 Acked Read 10 Done Sent Done false
4801 1909 Acked Write 9 Done Sent Done false
4802 1910 Acked Read 10 Done Sent Done false
4803 1911 Acked Write 7 Done Sent Done false
4804 1912 Acked Read 10 Done Sent Done false
4805 1913 Acked Flush 0 Done Sent Done false
4806 1914 Acked Write 5 Done Sent Done false
4807 1915 Acked Read 7 Done Sent Done false
4808 1916 Acked Write 2 Done Sent Done false
4809 1917 Acked Write 1 Done Sent Done false
4810 1918 Acked Write 6 Done Sent Done false
4811 1919 Acked Flush 0 Done Sent Done false
4812 1920 Acked Read 4 Done Sent Done false
4813 1921 Acked Read 7 Done Sent Done false
4814 1922 Acked Write 9 Done Sent Done false
4815 1923 Acked Read 1 Done Sent Done false
4816 1924 Acked Read 4 Done Sent Done false
4817 1925 Acked Read 9 Done Sent Done false
4818 1926 Acked Write 8 Done Sent Done false
4819 1927 Acked Write 6 Done Sent Done false
4820 1928 Acked Write 1 Done Sent Done false
4821 1929 Acked Read 1 Done Sent Done false
4822 1930 Acked Flush 0 Done Sent Done false
4823 1931 Acked Flush 0 Done Sent Done false
4824 1932 Acked Read 6 Done Sent Done false
4825 1933 Acked Read 2 Done Sent Done false
4826 1934 Acked Read 9 Done Sent Done false
4827 1935 Acked Read 7 Done Sent Done false
4828 1936 Acked Read 8 Done Sent Done false
4829 1937 Acked Flush 0 Done Sent Done false
4830 1938 Acked Write 6 Done Sent Done false
4831 1939 Acked Write 10 Done Sent Done false
4832 1940 Acked Write 2 Done Sent Done false
4833 1941 Acked Flush 0 Done Sent Done false
4834 1942 Acked Read 10 Done Sent Done false
4835 1943 Acked Write 5 Done Sent Done false
4836 1944 Acked Read 6 Done Sent Done false
4837 1945 Acked Flush 0 Done Sent Done false
4838 1946 Acked Read 8 Done Sent Done false
4839 1947 Acked Read 7 Done Sent Done false
4840 1948 Acked Read 4 Done Sent Done false
4841 1949 Acked Write 7 Done Sent Done false
4842 1950 Acked Write 4 Done Sent Done false
4843 1951 Acked Write 5 Done Sent Done false
4844 1952 Acked Read 2 Done Sent Done false
4845 1953 Acked Read 10 Done Sent Done false
4846 1954 Acked Write 2 Done Sent Done false
4847 1955 Acked Read 2 Done Sent Done false
4848 1956 Acked Read 1 Done Sent Done false
4849 1957 Acked Flush 0 Done Sent Done false
4850 1958 Acked Read 10 Done Sent Done false
4851 1959 Acked Write 10 Done Sent Done false
4852 1960 Acked Read 8 Done Sent Done false
4853 1961 Acked Write 6 Done Sent Done false
4854 1962 Acked Write 4 Done Sent Done false
4855 1963 Acked Flush 0 Done Sent Done false
4856 1964 Acked Write 6 Done Sent Done false
4857 1965 Acked Flush 0 Done Sent Done false
4858 1966 Acked Read 2 Done Sent Done false
4859 1967 Acked Read 1 Done Sent Done false
4860 1968 Acked Write 2 Done Sent Done false
4861 1969 Acked Write 3 Done Sent Done false
4862 1970 Acked Read 6 Done Sent Done false
4863 1971 Acked Write 10 Done Sent Done false
4864 1972 Acked Read 2 Done Sent Done false
4865 1973 Acked Read 4 Done Sent Done false
4866 1974 Acked Read 6 Done Sent Done false
4867 1975 Acked Write 6 Done Sent Done false
4868 1976 Acked Read 3 Done Sent Done false
4869 1977 Acked Write 4 Done Sent Done false
4870 1978 Acked Read 9 Done Sent Done false
4871 1979 Acked Flush 0 Done Sent Done false
4872 1980 Acked Write 6 Done Sent Done false
4873 1981 Acked Read 3 Done Sent Done false
4874 1982 Acked Read 4 Done Sent Done false
4875 1983 Acked Write 7 Done Sent Done false
4876 1984 Acked Write 8 Done Sent Done false
4877 1985 Acked Write 1 Done Sent Done false
4878 1986 Acked Write 4 Done Sent Done false
4879 1987 Acked Read 7 Done Sent Done false
4880 1988 Acked Read 5 Done Sent Done false
4881 1989 Acked Write 6 Done Sent Done false
4882 1990 Acked Write 2 Done Sent Done false
4883 1991 Acked Read 5 Done Sent Done false
4884 1992 Acked Write 6 Done Sent Done false
4885 1993 Acked Read 9 Done Sent Done false
4886 1994 Acked Read 7 Done Sent Done false
4887 1995 Acked Write 8 Done Sent Done false
4888 1996 Acked Read 9 Done Sent Done false
4889 1997 Acked Read 2 Done Sent Done false
4890 1998 Acked Read 9 Done Sent Done false
4891 1999 Acked Read 9 Done Sent Done false
4892 2000 Acked Write 1 Done Sent Done false
4893 2001 Acked Write 3 Done Sent Done false
4894 2002 Acked Write 1 Done Sent Done false
4895 2003 Acked Flush 0 Done Sent Done false
4896 2004 Acked Read 9 Done Sent Done false
4897 2005 Acked Write 3 Done Sent Done false
4898 2006 Acked Write 8 Done Sent Done false
4899 2007 Acked Read 5 Done Sent Done false
4900 2008 Acked Read 2 Done Sent Done false
4901 2009 Acked Write 2 Done Sent Done false
4902 2010 Acked Read 8 Done Sent Done false
4903 2011 Acked Flush 0 Done Sent Done false
4904 2012 Acked Write 4 Done Sent Done false
4905 2013 Acked Read 10 Done Sent Done false
4906 2014 Acked Write 2 Done Sent Done false
4907 2015 Acked Read 8 Done Sent Done false
4908 2016 Acked Write 2 Done Sent Done false
4909 2017 Acked Write 5 Done Sent Done false
4910 2018 Acked Flush 0 Done Sent Done false
4911 2019 Acked Write 8 Done Sent Done false
4912 2020 Acked Flush 0 Done Sent Done false
4913 2021 Acked Write 3 Done Sent Done false
4914 2022 Acked Read 4 Done Sent Done false
4915 2023 Acked Write 3 Done Sent Done false
4916 2024 Acked Read 3 Done Sent Done false
4917 2025 Acked Read 5 Done Sent Done false
4918 2026 Acked Read 8 Done Sent Done false
4919 2027 Acked Read 4 Done Sent Done false
4920 2028 Acked Read 1 Done Sent Done false
4921 2029 Acked Read 9 Done Sent Done false
4922 2030 Acked Flush 0 Done Sent Done false
4923 2031 Acked Read 7 Done Sent Done false
4924 2032 Acked Read 2 Done Sent Done false
4925 2033 Acked Write 10 Done Sent Done false
4926 2034 Acked Flush 0 Done Sent Done false
4927 2035 Acked Write 2 Done Sent Done false
4928 2036 Acked Flush 0 Done Sent Done false
4929 2037 Acked Read 1 Done Sent Done false
4930 2038 Acked Write 8 Done Sent Done false
4931 2039 Acked Write 10 Done Sent Done false
4932 2040 Acked Write 3 Done Sent Done false
4933 2041 Acked Read 3 Done Sent Done false
4934 2042 Acked Read 9 Done Sent Done false
4935 2043 Acked Write 7 Done Sent Done false
4936 2044 Acked Read 3 Done Sent Done false
4937 2045 Acked Write 8 Done Sent Done false
4938 2046 Acked Read 5 Done Sent Done false
4939 2047 Acked Write 8 Done Sent Done false
4940 2048 Acked Write 6 Done Sent Done false
4941 2049 Acked Read 1 Done Sent Done false
4942 2050 Acked Read 5 Done Sent Done false
4943 2051 Acked Write 4 Done Sent Done false
4944 2052 Acked Read 10 Done Sent Done false
4945 2053 Acked Read 4 Done Sent Done false
4946 2054 Acked Write 6 Done Sent Done false
4947 2055 Acked Write 6 Done Sent Done false
4948 2056 Acked Write 7 Done Sent Done false
4949 2057 Acked Read 4 Done Sent Done false
4950 2058 Acked Write 9 Done Sent Done false
4951 2059 Acked Read 6 Done Sent Done false
4952 2060 Acked Read 7 Done Sent Done false
4953 2061 Acked Read 6 Done Sent Done false
4954 2062 Acked Read 3 Done Sent Done false
4955 2063 Acked Write 9 Done Sent Done false
4956 2064 Acked Write 5 Done Sent Done false
4957 2065 Acked Write 2 Done Sent Done false
4958 2066 Acked Write 1 Done Sent Done false
4959 2067 Acked Read 1 Done Sent Done false
4960 2068 Acked Write 2 Done Sent Done false
4961 2069 Acked Read 4 Done Sent Done false
4962 2070 Acked Read 8 Done Sent Done false
4963 2071 Acked Read 4 Done Sent Done false
4964 2072 Acked Flush 0 Done Sent Done false
4965 2073 Acked Write 9 Done Sent Done false
4966 2074 Acked Write 4 Done Sent Done false
4967 2075 Acked Read 5 Done Sent Done false
4968 2076 Acked Write 4 Done Sent Done false
4969 2077 Acked Write 2 Done Sent Done false
4970 2078 Acked Read 1 Done Sent Done false
4971 2079 Acked Read 9 Done Sent Done false
4972 2080 Acked Write 6 Done Sent Done false
4973 2081 Acked Write 8 Done Sent Done false
4974 2082 Acked Write 5 Done Sent Done false
4975 2083 Acked Flush 0 Done Sent Done false
4976 2084 Acked Flush 0 Done Sent Done false
4977 2085 Acked Read 6 Done Sent Done false
4978 2086 Acked Write 10 Done Sent Done false
4979 2087 Acked Read 9 Done Sent Done false
4980 2088 Acked Write 6 Done Sent Done false
4981 2089 Acked Flush 0 Done Sent Done false
4982 2090 Acked Write 1 Done Sent Done false
4983 2091 Acked Write 7 Done Sent Done false
4984 2092 Acked Write 4 Done Sent Done false
4985 2093 Acked Write 3 Done Sent Done false
4986 2094 Acked Write 10 Done Sent Done false
4987 2095 Acked Write 3 Done Sent Done false
4988 2096 Acked Read 5 Done Sent Done false
4989 2097 Acked Write 9 Done Sent Done false
4990 2098 Acked Write 8 Done Sent Done false
4991 2099 Acked Read 2 Done Sent Done false
4992 2100 Acked Read 4 Done Sent Done false
4993 2101 Acked Write 7 Done Sent Done false
4994 2102 Acked Read 3 Done Sent Done false
4995 2103 Acked Flush 0 Done Sent Done false
4996 2104 Acked Read 3 Done Sent Done false
4997 2105 Acked Read 5 Done Sent Done false
4998 2106 Acked Read 4 Done Sent Done false
4999 2107 Acked Read 3 Done Sent Done false
5000 2108 Acked Read 3 Done Sent Done false
5001 2109 Acked Read 5 Done Sent Done false
5002 2110 Acked Read 8 Done Sent Done false
5003 2111 Acked Read 1 Done Sent Done false
5004 2112 Acked Write 8 Done Sent Done false
5005 2113 Acked Read 10 Done Sent Done false
5006 2114 Acked Read 5 Done Sent Done false
5007 2115 Acked Read 5 Done Sent Done false
5008 2116 Acked Write 7 Done Sent Done false
5009 2117 Acked Read 2 Done Sent Done false
5010 2118 Acked Read 2 Done Sent Done false
5011 2119 Acked Read 2 Done Sent Done false
5012 2120 Acked Read 4 Done Sent Done false
5013 2121 Acked Write 9 Done Sent Done false
5014 2122 Acked Read 10 Done Sent Done false
5015 2123 Acked Write 10 Done Sent Done false
5016 2124 Acked Read 2 Done Sent Done false
5017 2125 Acked Write 10 Done Sent Done false
5018 2126 Acked Read 1 Done Sent Done false
5019 2127 Acked Flush 0 Done Sent Done false
5020 2128 Acked Write 8 Done Sent Done false
5021 2129 Acked Read 2 Done Sent Done false
5022 2130 Acked Read 7 Done Sent Done false
5023 2131 Acked Write 6 Done Sent Done false
5024 2132 Acked Read 3 Done Sent Done false
5025 2133 Acked Read 6 Done Sent Done false
5026 2134 Acked Write 5 Done Sent Done false
5027 2135 Acked Write 2 Done Sent Done false
5028 2136 Acked Read 7 Done Sent Done false
5029 2137 Acked Write 9 Done Sent Done false
5030 2138 Acked Read 4 Done Sent Done false
5031 2139 Acked Write 1 Done Sent Done false
5032 2140 Acked Write 6 Done Sent Done false
5033 2141 Acked Read 6 Done Sent Done false
5034 2142 Acked Read 3 Done Sent Done false
5035 2143 Acked Read 7 Done Sent Done false
5036 2144 Acked Read 6 Done Sent Done false
5037 2145 Acked Read 3 Done Sent Done false
5038 2146 Acked Read 3 Done Sent Done false
5039 2147 Acked Read 9 Done Sent Done false
5040 2148 Acked Write 4 Done Sent Done false
5041 2149 Acked Write 5 Done Sent Done false
5042 2150 Acked Write 8 Done Sent Done false
5043 2151 Acked Write 1 Done Sent Done false
5044 2152 Acked Read 6 Done Sent Done false
5045 2153 Acked Read 8 Done Sent Done false
5046 2154 Acked Write 3 Done Sent Done false
5047 2155 Acked Write 7 Done Sent Done false
5048 2156 Acked Write 7 Done Sent Done false
5049 2157 Acked Read 9 Done Sent Done false
5050 2158 Acked Read 8 Done Sent Done false
5051 2159 Acked Read 1 Done Sent Done false
5052 2160 Acked Write 6 Done Sent Done false
5053 2161 Acked Write 5 Done Sent Done false
5054 2162 Acked Write 9 Done Sent Done false
5055 2163 Acked Write 3 Done Sent Done false
5056 2164 Acked Write 4 Done Sent Done false
5057 2165 Acked Write 4 Done Sent Done false
5058 2166 Acked Flush 0 Done Sent Done false
5059 2167 Acked Read 6 Done Sent Done false
5060 2168 Acked Write 10 Done Sent Done false
5061 2169 Acked Read 3 Done Sent Done false
5062 2170 Acked Flush 0 Done Sent Done false
5063 2171 Acked Write 5 Done Sent Done false
5064 2172 Acked Write 8 Done Sent Done false
5065 2173 Acked Read 1 Done Sent Done false
5066 2174 Acked Flush 0 Done Sent Done false
5067 2175 Acked Write 2 Done Sent Done false
5068 2176 Acked Read 3 Done Sent Done false
5069 2177 Acked Read 4 Done Sent Done false
5070 2178 Acked Read 5 Done Sent Done false
5071 2179 Acked Write 2 Done Sent Done false
5072 2180 Acked Write 6 Done Sent Done false
5073 2181 Acked Read 10 Done Sent Done false
5074 2182 Acked Read 7 Done Sent Done false
5075 2183 Acked Write 10 Done Sent Done false
5076 2184 Acked Read 7 Done Sent Done false
5077 2185 Acked Read 5 Done Sent Done false
5078 2186 Acked Write 8 Done Sent Done false
5079 2187 Acked Flush 0 Done Sent Done false
5080 2188 Acked Read 10 Done Sent Done false
5081 2189 Acked Read 7 Done Sent Done false
5082 2190 Acked Read 9 Done Sent Done false
5083 2191 Acked Read 10 Done Sent Done false
5084 2192 Acked Read 5 Done Sent Done false
5085 2193 Acked Read 7 Done Sent Done false
5086 2194 Acked Write 7 Done Sent Done false
5087 2195 Acked Read 1 Done Sent Done false
5088 2196 Acked Read 1 Done Sent Done false
5089 2197 Acked Read 8 Done Sent Done false
5090 2198 Acked Write 4 Done Sent Done false
5091 2199 Acked Write 8 Done Sent Done false
5092 2200 Acked Write 2 Done Sent Done false
5093 2201 Acked Write 2 Done Sent Done false
5094 2202 Acked Read 3 Done Sent Done false
5095 2203 Acked Read 4 Done Sent Done false
5096 2204 Acked Write 10 Done Sent Done false
5097 2205 Acked Read 2 Done Sent Done false
5098 2206 Acked Write 2 Done Sent Done false
5099 STATES DS:0 DS:1 DS:2 TOTAL
5100 Sent 0 301 0 301
5101 Done 301 0 301 602
5102 Skipped 0 0 0 0
5103 Error 0 0 0 0
5104 Last Flush: Some(JobId(2187)) Some(JobId(1905)) Some(JobId(2187))
5105 Downstairs last five completed: 1905 1904 1903 1902 1901
5106 Upstairs last five completed: 2206 2205 2204 2203 2202
5107 CLIENT: Up:0 ds:301 act:2
51082025-03-17T22:20:03.656ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
5109 [8820] got start action
5110 [8820] I am going to restart
5111 Starting downstairs at port 8820
5112 Make output file at "/tmp/dsc/downstairs-8820.txt"
5113 [8820][1] reports Starting
5114 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1215)
5115 [8820][1] reports Running
51162025-03-17T22:20:05.105ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
51172025-03-17T22:20:05.660ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
51182025-03-17T22:20:06.662ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
5119 ----------------------------------------------------------------
5120 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5121 States: Active Offline Active
5122 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5123 1906 Acked Flush 0 Done Sent Done false
5124 1907 Acked Read 3 Done Sent Done false
5125 1908 Acked Read 10 Done Sent Done false
5126 1909 Acked Write 9 Done Sent Done false
5127 1910 Acked Read 10 Done Sent Done false
5128 1911 Acked Write 7 Done Sent Done false
5129 1912 Acked Read 10 Done Sent Done false
5130 1913 Acked Flush 0 Done Sent Done false
5131 1914 Acked Write 5 Done Sent Done false
5132 1915 Acked Read 7 Done Sent Done false
5133 1916 Acked Write 2 Done Sent Done false
5134 1917 Acked Write 1 Done Sent Done false
5135 1918 Acked Write 6 Done Sent Done false
5136 1919 Acked Flush 0 Done Sent Done false
5137 1920 Acked Read 4 Done Sent Done false
5138 1921 Acked Read 7 Done Sent Done false
5139 1922 Acked Write 9 Done Sent Done false
5140 1923 Acked Read 1 Done Sent Done false
5141 1924 Acked Read 4 Done Sent Done false
5142 1925 Acked Read 9 Done Sent Done false
5143 1926 Acked Write 8 Done Sent Done false
5144 1927 Acked Write 6 Done Sent Done false
5145 1928 Acked Write 1 Done Sent Done false
5146 1929 Acked Read 1 Done Sent Done false
5147 1930 Acked Flush 0 Done Sent Done false
5148 1931 Acked Flush 0 Done Sent Done false
5149 1932 Acked Read 6 Done Sent Done false
5150 1933 Acked Read 2 Done Sent Done false
5151 1934 Acked Read 9 Done Sent Done false
5152 1935 Acked Read 7 Done Sent Done false
5153 1936 Acked Read 8 Done Sent Done false
5154 1937 Acked Flush 0 Done Sent Done false
5155 1938 Acked Write 6 Done Sent Done false
5156 1939 Acked Write 10 Done Sent Done false
5157 1940 Acked Write 2 Done Sent Done false
5158 1941 Acked Flush 0 Done Sent Done false
5159 1942 Acked Read 10 Done Sent Done false
5160 1943 Acked Write 5 Done Sent Done false
5161 1944 Acked Read 6 Done Sent Done false
5162 1945 Acked Flush 0 Done Sent Done false
5163 1946 Acked Read 8 Done Sent Done false
5164 1947 Acked Read 7 Done Sent Done false
5165 1948 Acked Read 4 Done Sent Done false
5166 1949 Acked Write 7 Done Sent Done false
5167 1950 Acked Write 4 Done Sent Done false
5168 1951 Acked Write 5 Done Sent Done false
5169 1952 Acked Read 2 Done Sent Done false
5170 1953 Acked Read 10 Done Sent Done false
5171 1954 Acked Write 2 Done Sent Done false
5172 1955 Acked Read 2 Done Sent Done false
5173 1956 Acked Read 1 Done Sent Done false
5174 1957 Acked Flush 0 Done Sent Done false
5175 1958 Acked Read 10 Done Sent Done false
5176 1959 Acked Write 10 Done Sent Done false
5177 1960 Acked Read 8 Done Sent Done false
5178 1961 Acked Write 6 Done Sent Done false
5179 1962 Acked Write 4 Done Sent Done false
5180 1963 Acked Flush 0 Done Sent Done false
5181 1964 Acked Write 6 Done Sent Done false
5182 1965 Acked Flush 0 Done Sent Done false
5183 1966 Acked Read 2 Done Sent Done false
5184 1967 Acked Read 1 Done Sent Done false
5185 1968 Acked Write 2 Done Sent Done false
5186 1969 Acked Write 3 Done Sent Done false
5187 1970 Acked Read 6 Done Sent Done false
5188 1971 Acked Write 10 Done Sent Done false
5189 1972 Acked Read 2 Done Sent Done false
5190 1973 Acked Read 4 Done Sent Done false
5191 1974 Acked Read 6 Done Sent Done false
5192 1975 Acked Write 6 Done Sent Done false
5193 1976 Acked Read 3 Done Sent Done false
5194 1977 Acked Write 4 Done Sent Done false
5195 1978 Acked Read 9 Done Sent Done false
5196 1979 Acked Flush 0 Done Sent Done false
5197 1980 Acked Write 6 Done Sent Done false
5198 1981 Acked Read 3 Done Sent Done false
5199 1982 Acked Read 4 Done Sent Done false
5200 1983 Acked Write 7 Done Sent Done false
5201 1984 Acked Write 8 Done Sent Done false
5202 1985 Acked Write 1 Done Sent Done false
5203 1986 Acked Write 4 Done Sent Done false
5204 1987 Acked Read 7 Done Sent Done false
5205 1988 Acked Read 5 Done Sent Done false
5206 1989 Acked Write 6 Done Sent Done false
5207 1990 Acked Write 2 Done Sent Done false
5208 1991 Acked Read 5 Done Sent Done false
5209 1992 Acked Write 6 Done Sent Done false
5210 1993 Acked Read 9 Done Sent Done false
5211 1994 Acked Read 7 Done Sent Done false
5212 1995 Acked Write 8 Done Sent Done false
5213 1996 Acked Read 9 Done Sent Done false
5214 1997 Acked Read 2 Done Sent Done false
5215 1998 Acked Read 9 Done Sent Done false
5216 1999 Acked Read 9 Done Sent Done false
5217 2000 Acked Write 1 Done Sent Done false
5218 2001 Acked Write 3 Done Sent Done false
5219 2002 Acked Write 1 Done Sent Done false
5220 2003 Acked Flush 0 Done Sent Done false
5221 2004 Acked Read 9 Done Sent Done false
5222 2005 Acked Write 3 Done Sent Done false
5223 2006 Acked Write 8 Done Sent Done false
5224 2007 Acked Read 5 Done Sent Done false
5225 2008 Acked Read 2 Done Sent Done false
5226 2009 Acked Write 2 Done Sent Done false
5227 2010 Acked Read 8 Done Sent Done false
5228 2011 Acked Flush 0 Done Sent Done false
5229 2012 Acked Write 4 Done Sent Done false
5230 2013 Acked Read 10 Done Sent Done false
5231 2014 Acked Write 2 Done Sent Done false
5232 2015 Acked Read 8 Done Sent Done false
5233 2016 Acked Write 2 Done Sent Done false
5234 2017 Acked Write 5 Done Sent Done false
5235 2018 Acked Flush 0 Done Sent Done false
5236 2019 Acked Write 8 Done Sent Done false
5237 2020 Acked Flush 0 Done Sent Done false
5238 2021 Acked Write 3 Done Sent Done false
5239 2022 Acked Read 4 Done Sent Done false
5240 2023 Acked Write 3 Done Sent Done false
5241 2024 Acked Read 3 Done Sent Done false
5242 2025 Acked Read 5 Done Sent Done false
5243 2026 Acked Read 8 Done Sent Done false
5244 2027 Acked Read 4 Done Sent Done false
5245 2028 Acked Read 1 Done Sent Done false
5246 2029 Acked Read 9 Done Sent Done false
5247 2030 Acked Flush 0 Done Sent Done false
5248 2031 Acked Read 7 Done Sent Done false
5249 2032 Acked Read 2 Done Sent Done false
5250 2033 Acked Write 10 Done Sent Done false
5251 2034 Acked Flush 0 Done Sent Done false
5252 2035 Acked Write 2 Done Sent Done false
5253 2036 Acked Flush 0 Done Sent Done false
5254 2037 Acked Read 1 Done Sent Done false
5255 2038 Acked Write 8 Done Sent Done false
5256 2039 Acked Write 10 Done Sent Done false
5257 2040 Acked Write 3 Done Sent Done false
5258 2041 Acked Read 3 Done Sent Done false
5259 2042 Acked Read 9 Done Sent Done false
5260 2043 Acked Write 7 Done Sent Done false
5261 2044 Acked Read 3 Done Sent Done false
5262 2045 Acked Write 8 Done Sent Done false
5263 2046 Acked Read 5 Done Sent Done false
5264 2047 Acked Write 8 Done Sent Done false
5265 2048 Acked Write 6 Done Sent Done false
5266 2049 Acked Read 1 Done Sent Done false
5267 2050 Acked Read 5 Done Sent Done false
5268 2051 Acked Write 4 Done Sent Done false
5269 2052 Acked Read 10 Done Sent Done false
5270 2053 Acked Read 4 Done Sent Done false
5271 2054 Acked Write 6 Done Sent Done false
5272 2055 Acked Write 6 Done Sent Done false
5273 2056 Acked Write 7 Done Sent Done false
5274 2057 Acked Read 4 Done Sent Done false
5275 2058 Acked Write 9 Done Sent Done false
5276 2059 Acked Read 6 Done Sent Done false
5277 2060 Acked Read 7 Done Sent Done false
5278 2061 Acked Read 6 Done Sent Done false
5279 2062 Acked Read 3 Done Sent Done false
5280 2063 Acked Write 9 Done Sent Done false
5281 2064 Acked Write 5 Done Sent Done false
5282 2065 Acked Write 2 Done Sent Done false
5283 2066 Acked Write 1 Done Sent Done false
5284 2067 Acked Read 1 Done Sent Done false
5285 2068 Acked Write 2 Done Sent Done false
5286 2069 Acked Read 4 Done Sent Done false
5287 2070 Acked Read 8 Done Sent Done false
5288 2071 Acked Read 4 Done Sent Done false
5289 2072 Acked Flush 0 Done Sent Done false
5290 2073 Acked Write 9 Done Sent Done false
5291 2074 Acked Write 4 Done Sent Done false
5292 2075 Acked Read 5 Done Sent Done false
5293 2076 Acked Write 4 Done Sent Done false
5294 2077 Acked Write 2 Done Sent Done false
5295 2078 Acked Read 1 Done Sent Done false
5296 2079 Acked Read 9 Done Sent Done false
5297 2080 Acked Write 6 Done Sent Done false
5298 2081 Acked Write 8 Done Sent Done false
5299 2082 Acked Write 5 Done Sent Done false
5300 2083 Acked Flush 0 Done Sent Done false
5301 2084 Acked Flush 0 Done Sent Done false
5302 2085 Acked Read 6 Done Sent Done false
5303 2086 Acked Write 10 Done Sent Done false
5304 2087 Acked Read 9 Done Sent Done false
5305 2088 Acked Write 6 Done Sent Done false
5306 2089 Acked Flush 0 Done Sent Done false
5307 2090 Acked Write 1 Done Sent Done false
5308 2091 Acked Write 7 Done Sent Done false
5309 2092 Acked Write 4 Done Sent Done false
5310 2093 Acked Write 3 Done Sent Done false
5311 2094 Acked Write 10 Done Sent Done false
5312 2095 Acked Write 3 Done Sent Done false
5313 2096 Acked Read 5 Done Sent Done false
5314 2097 Acked Write 9 Done Sent Done false
5315 2098 Acked Write 8 Done Sent Done false
5316 2099 Acked Read 2 Done Sent Done false
5317 2100 Acked Read 4 Done Sent Done false
5318 2101 Acked Write 7 Done Sent Done false
5319 2102 Acked Read 3 Done Sent Done false
5320 2103 Acked Flush 0 Done Sent Done false
5321 2104 Acked Read 3 Done Sent Done false
5322 2105 Acked Read 5 Done Sent Done false
5323 2106 Acked Read 4 Done Sent Done false
5324 2107 Acked Read 3 Done Sent Done false
5325 2108 Acked Read 3 Done Sent Done false
5326 2109 Acked Read 5 Done Sent Done false
5327 2110 Acked Read 8 Done Sent Done false
5328 2111 Acked Read 1 Done Sent Done false
5329 2112 Acked Write 8 Done Sent Done false
5330 2113 Acked Read 10 Done Sent Done false
5331 2114 Acked Read 5 Done Sent Done false
5332 2115 Acked Read 5 Done Sent Done false
5333 2116 Acked Write 7 Done Sent Done false
5334 2117 Acked Read 2 Done Sent Done false
5335 2118 Acked Read 2 Done Sent Done false
5336 2119 Acked Read 2 Done Sent Done false
5337 2120 Acked Read 4 Done Sent Done false
5338 2121 Acked Write 9 Done Sent Done false
5339 2122 Acked Read 10 Done Sent Done false
5340 2123 Acked Write 10 Done Sent Done false
5341 2124 Acked Read 2 Done Sent Done false
5342 2125 Acked Write 10 Done Sent Done false
5343 2126 Acked Read 1 Done Sent Done false
5344 2127 Acked Flush 0 Done Sent Done false
5345 2128 Acked Write 8 Done Sent Done false
5346 2129 Acked Read 2 Done Sent Done false
5347 2130 Acked Read 7 Done Sent Done false
5348 2131 Acked Write 6 Done Sent Done false
5349 2132 Acked Read 3 Done Sent Done false
5350 2133 Acked Read 6 Done Sent Done false
5351 2134 Acked Write 5 Done Sent Done false
5352 2135 Acked Write 2 Done Sent Done false
5353 2136 Acked Read 7 Done Sent Done false
5354 2137 Acked Write 9 Done Sent Done false
5355 2138 Acked Read 4 Done Sent Done false
5356 2139 Acked Write 1 Done Sent Done false
5357 2140 Acked Write 6 Done Sent Done false
5358 2141 Acked Read 6 Done Sent Done false
5359 2142 Acked Read 3 Done Sent Done false
5360 2143 Acked Read 7 Done Sent Done false
5361 2144 Acked Read 6 Done Sent Done false
5362 2145 Acked Read 3 Done Sent Done false
5363 2146 Acked Read 3 Done Sent Done false
5364 2147 Acked Read 9 Done Sent Done false
5365 2148 Acked Write 4 Done Sent Done false
5366 2149 Acked Write 5 Done Sent Done false
5367 2150 Acked Write 8 Done Sent Done false
5368 2151 Acked Write 1 Done Sent Done false
5369 2152 Acked Read 6 Done Sent Done false
5370 2153 Acked Read 8 Done Sent Done false
5371 2154 Acked Write 3 Done Sent Done false
5372 2155 Acked Write 7 Done Sent Done false
5373 2156 Acked Write 7 Done Sent Done false
5374 2157 Acked Read 9 Done Sent Done false
5375 2158 Acked Read 8 Done Sent Done false
5376 2159 Acked Read 1 Done Sent Done false
5377 2160 Acked Write 6 Done Sent Done false
5378 2161 Acked Write 5 Done Sent Done false
5379 2162 Acked Write 9 Done Sent Done false
5380 2163 Acked Write 3 Done Sent Done false
5381 2164 Acked Write 4 Done Sent Done false
5382 2165 Acked Write 4 Done Sent Done false
5383 2166 Acked Flush 0 Done Sent Done false
5384 2167 Acked Read 6 Done Sent Done false
5385 2168 Acked Write 10 Done Sent Done false
5386 2169 Acked Read 3 Done Sent Done false
5387 2170 Acked Flush 0 Done Sent Done false
5388 2171 Acked Write 5 Done Sent Done false
5389 2172 Acked Write 8 Done Sent Done false
5390 2173 Acked Read 1 Done Sent Done false
5391 2174 Acked Flush 0 Done Sent Done false
5392 2175 Acked Write 2 Done Sent Done false
5393 2176 Acked Read 3 Done Sent Done false
5394 2177 Acked Read 4 Done Sent Done false
5395 2178 Acked Read 5 Done Sent Done false
5396 2179 Acked Write 2 Done Sent Done false
5397 2180 Acked Write 6 Done Sent Done false
5398 2181 Acked Read 10 Done Sent Done false
5399 2182 Acked Read 7 Done Sent Done false
5400 2183 Acked Write 10 Done Sent Done false
5401 2184 Acked Read 7 Done Sent Done false
5402 2185 Acked Read 5 Done Sent Done false
5403 2186 Acked Write 8 Done Sent Done false
5404 2187 Acked Flush 0 Done Sent Done false
5405 2188 Acked Read 10 Done Sent Done false
5406 2189 Acked Read 7 Done Sent Done false
5407 2190 Acked Read 9 Done Sent Done false
5408 2191 Acked Read 10 Done Sent Done false
5409 2192 Acked Read 5 Done Sent Done false
5410 2193 Acked Read 7 Done Sent Done false
5411 2194 Acked Write 7 Done Sent Done false
5412 2195 Acked Read 1 Done Sent Done false
5413 2196 Acked Read 1 Done Sent Done false
5414 2197 Acked Read 8 Done Sent Done false
5415 2198 Acked Write 4 Done Sent Done false
5416 2199 Acked Write 8 Done Sent Done false
5417 2200 Acked Write 2 Done Sent Done false
5418 2201 Acked Write 2 Done Sent Done false
5419 2202 Acked Read 3 Done Sent Done false
5420 2203 Acked Read 4 Done Sent Done false
5421 2204 Acked Write 10 Done Sent Done false
5422 2205 Acked Read 2 Done Sent Done false
5423 2206 Acked Write 2 Done Sent Done false
5424 STATES DS:0 DS:1 DS:2 TOTAL
5425 Sent 0 301 0 301
5426 Done 301 0 301 602
5427 Skipped 0 0 0 0
5428 Error 0 0 0 0
5429 Last Flush: Some(JobId(2187)) Some(JobId(1905)) Some(JobId(2187))
5430 Downstairs last five completed: 1905 1904 1903 1902 1901
5431 Upstairs last five completed: 2206 2205 2204 2203 2202
5432 CLIENT: Up:0 ds:301 act:2
54332025-03-17T22:20:07.664ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
54342025-03-17T22:20:08.664ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
54352025-03-17T22:20:08.842ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
54362025-03-17T22:20:08.842ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:33784) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
54372025-03-17T22:20:08.843ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
54382025-03-17T22:20:08.844ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
54392025-03-17T22:20:08.844ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
54402025-03-17T22:20:08.844ZINFOcrucible: send last flush ID to this DS: Some(JobId(1905)) = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
54412025-03-17T22:20:08.844ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
54422025-03-17T22:20:08.844ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(1905)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
54432025-03-17T22:20:08.844ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(1905)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
5444 ----------------------------------------------------------------
5445 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
5446 Downstairs last five completed: 2207 2206 2205 2204 2203
5447 Upstairs last five completed: 2207 2206 2205 2204 2203
5448 CLIENT: Up:0 ds:0 act:3
5449 Replay: All jobs finished
5450 Main task has work to do, go find it
5451 got dsc Stop(0)
5452 stop 0
5453 Mar 17 22:20:11.084 INFO request completed, latency_us: 367, response_code: 204, uri[8810] Got stop action so:false kr:false
5454 : /stop/cid/0, method: POST, req_id: 98ae9c41-0d99-403b-839d-f6cb5f89aaa3, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
54552025-03-17T22:20:11.085ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
54562025-03-17T22:20:11.085ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
54572025-03-17T22:20:11.085ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
5458 Mar 17 22:20:11.085 INFO request completed, {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:20:11.085794971Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
5459 latency_us: 121, response_code: 200, uri: /state/cid/0, method: GET, req_id: 209dcc12-4e78-468c-b9b1-4cc172f5cf42, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
5460 [8810] Exited with: ExitStatus(unix_wait_status(9))
5461 [8810][0] reports Exit
5462 Mar 17 22:20:15.088 INFO request completed, latency_us: 244, response_code: 200, uri: /state/cid/0, method: GET, req_id: 058a9091-124d-46e6-8d1a-8187e437d6ee, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
5463 001/300 Write block 3761 len 40960 data: 3 3 3 2 2 2 3 3 3 2
5464 002/300 Write block 2257 len 24576 data: 2 3 3 3 3 3
5465 003/300 Write block 3123 len 40960 data: 2 2 3 3 3 3 3 3 3 2
5466 004/300 Flush
5467 005/300 Write block 3609 len 16384 data: 3 3 3 3
5468 006/300 Write block 1541 len 4096 data: 2
5469 007/300 Write block 4447 len 24576 data: 2 3 3 2 2 2
5470 008/300 Flush
5471 009/300 Read block 3961 len 4096
5472 010/300 Read block 24 len 12288
5473 011/300 Write block 3998 len 36864 data: 2 2 2 2 2 2 2 2 2
5474 012/300 Write block 3165 len 16384 data: 3 3 3 2
5475 013/300 Write block 1048 len 12288 data: 3 3 3
5476 014/300 Read block 3232 len 28672
5477 015/300 Read block 2160 len 24576
5478 016/300 Write block 4531 len 40960 data: 4 4 4 4 3 3 3 3 2 2
5479 017/300 Read block 1646 len 24576
5480 018/300 Flush
5481 019/300 Write block 1105 len 28672 data: 2 2 2 3 4 4 3
5482 020/300 Write block 1923 len 28672 data: 4 4 3 3 2 2 2
5483 021/300 Read block 3902 len 4096
5484 022/300 Write block 2446 len 20480 data: 3 3 3 4 3
5485 023/300 Flush
5486 024/300 Read block 3711 len 28672
5487 025/300 Read block 2596 len 40960
5488 026/300 Read block 4486 len 24576
5489 027/300 Read block 579 len 12288
5490 028/300 Write block 4692 len 28672 data: 2 2 2 2 2 4 4
5491 029/300 Write block 2656 len 16384 data: 2 2 2 2
5492 030/300 Read block 3688 len 32768
5493 031/300 Read block 2355 len 36864
5494 032/300 Read block 2526 len 8192
5495 033/300 Read block 600 len 24576
5496 034/300 Read block 1458 len 40960
5497 035/300 Read block 3311 len 40960
5498 036/300 Write block 2726 len 40960 data: 2 3 3 3 3 3 3 3 2 3
5499 037/300 Write block 1585 len 36864 data: 2 2 3 3 3 3 3 4 4
5500 038/300 Write block 3079 len 16384 data: 3 4 4 4
5501 039/300 Read block 3995 len 40960
5502 040/300 Read block 441 len 32768
5503 041/300 Read block 2318 len 8192
5504 042/300 Write block 4815 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5505 043/300 Write block 4145 len 28672 data: 5 4 3 3 3 3 3
5506 044/300 Write block 642 len 4096 data: 2
5507 045/300 Write block 1908 len 16384 data: 2 2 2 2
5508 046/300 Read block 1604 len 28672
5509 047/300 Read block 1156 len 4096
5510 048/300 Read block 3627 len 12288
5511 049/300 Read block 3593 len 8192
5512 050/300 Read block 3948 len 36864
5513 051/300 Write block 4533 len 12288 data: 5 5 4
5514 052/300 Read block 2474 len 40960
5515 053/300 Read block 3176 len 20480
5516 054/300 Write block 2313 len 32768 data: 5 6 5 4 4 4 3 2
5517 055/300 Read block 3970 len 36864
5518 056/300 Flush
5519 057/300 Write block 412 len 28672 data: 2 3 3 3 3 3 4
5520 058/300 Write block 85 len 20480 data: 3 3 3 4 2
5521 059/300 Read block 3841 len 36864
5522 060/300 Write block 199 len 20480 data: 2 2 2 3 3
5523 061/300 Write block 1046 len 24576 data: 3 3 4 4 4 3
5524 062/300 Write block 1243 len 12288 data: 2 2 2
5525 063/300 Read block 3699 len 40960
5526 064/300 Read block 1004 len 40960
5527 065/300 Read block 1484 len 20480
5528 066/300 Read block 1956 len 20480
5529 067/300 Write block 1280 len 40960 data: 4 4 3 3 3 4 5 5 4 4
5530 068/300 Read block 4698 len 24576
5531 069/300 Write block 2139 len 12288 data: 2 3 4
5532 070/300 Write block 3783 len 36864 data: 2 2 2 2 2 2 2 2 2
5533 071/300 Write block 182 len 40960 data: 3 3 2 2 2 3 3 2 2 2
5534 072/300 Read block 2849 len 12288
5535 073/300 Read block 1750 len 8192
5536 074/300 Read block 4521 len 28672
5537 075/300 Flush
5538 076/300 Write block 2527 len 24576 data: 2 2 2 2 2 2
5539 077/300 Read block 3215 len 12288
5540 078/300 Read block 224 len 8192
5541 079/300 Read block 1296 len 20480
5542 080/300 Read block 774 len 16384
5543 081/300 Read block 443 len 12288
5544 082/300 Flush
5545 083/300 Read block 4838 len 16384
5546 084/300 Read block 249 len 12288
5547 085/300 Flush
5548 086/300 Read block 3944 len 24576
5549 087/300 Write block 1327 len 4096 data: 2
5550 088/300 Write block 1892 len 4096 data: 2
5551 089/300 Write block 3523 len 36864 data: 3 2 2 2 2 3 3 3 4
5552 090/300 Read block 3836 len 28672
5553 091/300 Read block 672 len 8192
5554 092/300 Read block 3590 len 20480
5555 093/300 Flush
5556 094/300 Read block 4212 len 24576
5557 095/300 Read block 3445 len 40960
5558 096/300 Write block 700 len 16384 data: 2 2 2 2
5559 097/300 Write block 4760 len 28672 data: 2 2 2 2 3 3 4
5560 098/300 Read block 484 len 36864
5561 099/300 Read block 4152 len 16384
5562 100/300 Read block 3583 len 8192
5563 101/300 Read block 4561 len 28672
5564 102/300 Flush
5565 103/300 Read block 2334 len 20480
5566 104/300 Write block 2203 len 4096 data: 3
5567 105/300 Write block 3529 len 24576 data: 4 4 5 3 3 3
5568 106/300 Read block 297 len 36864
5569 107/300 Write block 2998 len 4096 data: 2
5570 108/300 Read block 451 len 24576
5571 109/300 Read block 3571 len 32768
5572 110/300 Write block 4256 len 40960 data: 2 3 3 4 3 3 3 3 3 2
5573 111/300 Write block 1669 len 12288 data: 2 3 3
5574 112/300 Read block 166 len 12288
5575 113/300 Read block 4041 len 40960
5576 114/300 Read block 2186 len 32768
5577 115/300 Read block 3292 len 12288
5578 116/300 Read block 4621 len 20480
5579 117/300 Write block 1626 len 4096 data: 2
5580 118/300 Read block 1854 len 12288
5581 119/300 Flush
5582 120/300 Write block 4178 len 24576 data: 4 4 3 3 3 2
5583 121/300 Read block 2712 len 36864
5584 122/300 Read block 2506 len 8192
5585 123/300 Flush
5586 124/300 Read block 3058 len 28672
5587 125/300 Write block 970 len 4096 data: 2
5588 126/300 Write block 4843 len 20480 data: 3 3 3 3 2
5589 127/300 Read block 4554 len 28672
5590 128/300 Read block 286 len 24576
5591 129/300 Flush
5592 130/300 Write block 1019 len 8192 data: 2 2
5593 131/300 Read block 915 len 24576
5594 132/300 Flush
5595 133/300 Read block 3623 len 12288
5596 134/300 Read block 1758 len 20480
5597 135/300 Write block 1583 len 12288 data: 2 2 3
5598 136/300 Write block 3774 len 12288 data: 2 2 2
5599 137/300 Read block 282 len 36864
5600 138/300 Read block 1380 len 36864
5601 139/300 Read block 3163 len 28672
5602 140/300 Flush
5603 141/300 Write block 3156 len 4096 data: 3
5604 142/300 Write block 2827 len 20480 data: 2 2 2 2 3
5605 143/300 Read block 562 len 4096
5606 144/300 Write block 1907 len 16384 data: 2 3 3 3
5607 145/300 Write block 459 len 24576 data: 2 3 3 3 3 3
5608 146/300 Read block 1549 len 36864
5609 147/300 Flush
5610 148/300 Write block 2932 len 4096 data: 2
5611 149/300 Read block 3421 len 24576
5612 150/300 Flush
5613 151/300 Read block 4047 len 4096
5614 152/300 Read block 3020 len 4096
5615 153/300 Write block 4242 len 32768 data: 2 2 2 2 2 2 2 2
5616 154/300 Flush
5617 155/300 Read block 593 len 40960
5618 156/300 Read block 2988 len 20480
5619 157/300 Write block 2742 len 4096 data: 4
5620 158/300 Write block 3268 len 16384 data: 2 2 2 3
5621 159/300 Write block 1326 len 36864 data: 2 3 2 2 2 2 2 2 2
5622 160/300 Read block 450 len 36864
5623 161/300 Flush
5624 162/300 Read block 1043 len 8192
5625 163/300 Read block 3893 len 32768
5626 164/300 Write block 269 len 20480 data: 3 3 3 3 3
5627 165/300 Write block 3830 len 4096 data: 2
5628 166/300 Write block 2224 len 4096 data: 3
5629 167/300 Read block 2263 len 36864
5630 168/300 Read block 4275 len 40960
5631 169/300 Write block 3321 len 28672 data: 3 3 3 3 2 2 3
5632 170/300 Write block 1000 len 16384 data: 2 2 2 2
5633 171/300 Write block 4194 len 28672 data: 2 3 3 3 3 2 2
5634 172/300 Read block 4353 len 16384
5635 173/300 Flush
5636 174/300 Write block 1549 len 24576 data: 3 3 3 3 3 3
5637 175/300 Read block 4823 len 32768
5638 176/300 Read block 2048 len 40960
5639 177/300 Write block 450 len 16384 data: 4 4 3 3
5640 178/300 Read block 1026 len 36864
5641 179/300 Write block 1646 len 20480 data: 2 2 3 2 2
5642 180/300 Write block 3833 len 16384 data: 2 3 3 3
5643 181/300 Read block 127 len 24576
5644 182/300 Write block 2737 len 12288 data: 3 3 4
5645 183/300 Read block 1263 len 36864
5646 184/300 Read block 1517 len 16384
5647 185/300 Write block 740 len 20480 data: 3 3 3 3 4
5648 186/300 Flush
5649 187/300 Write block 1951 len 8192 data: 2 2
5650 188/300 Read block 1172 len 8192
5651 189/300 Write block 4801 len 24576 data: 4 4 3 4 4 2
5652 190/300 Flush
5653 191/300 Read block 98 len 20480
5654 192/300 Read block 968 len 20480
5655 193/300 Write block 289 len 36864 data: 3 2 2 2 2 2 2 2 2
5656 194/300 Write block 388 len 36864 data: 2 3 3 3 3 3 3 3 3
5657 195/300 Flush
5658 196/300 Read block 2987 len 12288
5659 197/300 Write block 572 len 28672 data: 2 3 3 3 3 3 3
5660 198/300 Flush
5661 199/300 Read block 3764 len 4096
5662 200/300 Read block 215 len 24576
5663 201/300 Read block 3992 len 8192
5664 202/300 Write block 4508 len 36864 data: 2 2 2 3 3 3 3 3 3
5665 203/300 Write block 1269 len 4096 data: 2
5666 204/300 Write block 2583 len 16384 data: 3 3 3 3
5667 205/300 Read block 1312 len 28672
5668 206/300 Read block 1340 len 12288
5669 207/300 Flush
5670 208/300 Write block 4420 len 28672 data: 2 2 2 2 2 2 2
5671 209/300 Write block 1052 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5672 210/300 Read block 897 len 4096
5673 211/300 Read block 1307 len 28672
5674 212/300 Read block 2358 len 32768
5675 213/300 Read block 207 len 8192
5676 214/300 Write block 3853 len 8192 data: 2 2
5677 215/300 Read block 1625 len 20480
5678 216/300 Read block 2053 len 32768
5679 217/300 Read block 4067 len 16384
5680 218/300 Read block 3244 len 36864
5681 219/300 Read block 4346 len 4096
5682 220/300 Read block 4808 len 8192
5683 221/300 Read block 833 len 28672
5684 222/300 Write block 483 len 28672 data: 2 2 2 2 2 2 2
5685 223/300 Flush
5686 224/300 Read block 2925 len 4096
5687 225/300 Read block 409 len 4096
5688 226/300 Read block 2247 len 24576
5689 227/300 Write block 2610 len 8192 data: 2 2
5690 228/300 Read block 2459 len 16384
5691 229/300 Write block 274 len 12288 data: 2 2 2
5692 230/300 Read block 1216 len 20480
5693 231/300 Read block 3355 len 28672
5694 232/300 Write block 3197 len 4096 data: 4
5695 233/300 Read block 781 len 8192
5696 234/300 Read block 2778 len 28672
5697 235/300 Write block 3010 len 36864 data: 2 2 2 3 3 3 3 3 3
5698 236/300 Write block 1331 len 24576 data: 3 3 3 3 2 2
5699 237/300 Write block 3004 len 36864 data: 2 2 2 2 2 2 3 3 3
5700 238/300 Write block 2469 len 16384 data: 3 3 3 3
5701 239/300 Read block 3883 len 12288
5702 240/300 Read block 2495 len 28672
5703 241/300 Write block 2410 len 32768 data: 2 2 2 2 2 2 2 3
5704 242/300 Write block 3468 len 32768 data: 3 3 3 2 2 2 2 2
5705 243/300 Flush
5706 244/300 Write block 3017 len 32768 data: 4 4 3 3 3 3 3 3
5707 245/300 Write block 903 len 8192 data: 3 2
5708 246/300 Write block 3651 len 36864 data: 2 2 3 3 3 2 2 2 2
5709 247/300 Write block 1799 len 8192 data: 3 3
5710 248/300 Flush
5711 249/300 Write block 3437 len 8192 data: 2 2
5712 250/300 Write block 1124 len 20480 data: 3 3 3 3 2
5713 251/300 Read block 2517 len 12288
5714 252/300 Read block 2261 len 12288
5715 253/300 Read block 3836 len 4096
5716 254/300 Read block 4323 len 28672
5717 255/300 Write block 4930 len 16384 data: 2 2 2 2
5718 256/300 Read block 2157 len 16384
5719 257/300 Flush
5720 258/300 Read block 850 len 36864
5721 259/300 Write block 1307 len 32768 data: 3 3 3 3 3 2 2 2
5722 260/300 Read block 4105 len 4096
5723 261/300 Write block 3773 len 36864 data: 2 3 3 3 2 2 2 2 2
5724 262/300 Write block 4939 len 20480 data: 2 2 2 2 2
5725 263/300 Read block 4054 len 12288
5726 264/300 Write block 4133 len 20480 data: 2 2 2 2 2
5727 265/300 Flush
5728 266/300 Read block 4578 len 24576
5729 267/300 Write block 2920 len 24576 data: 2 3 3 2 2 2
5730 268/300 Write block 1397 len 28672 data: 3 3 3 2 2 2 2
5731 269/300 Read block 2052 len 40960
5732 270/300 Write block 2010 len 24576 data: 2 3 4 4 3 3
5733 271/300 Flush
5734 272/300 Write block 3135 len 36864 data: 3 3 3 3 2 2 2 2 2
5735 273/300 Read block 3392 len 28672
5736 274/300 Flush
5737 275/300 Write block 683 len 28672 data: 2 2 2 2 2 2 2
5738 276/300 Read block 1704 len 32768
5739 277/300 Read block 1872 len 4096
5740 278/300 Read block 4485 len 36864
5741 279/300 Read block 714 len 40960
5742 280/300 Write block 3370 len 28672 data: 3 3 3 3 3 3 3
5743 281/300 Write block 3089 len 28672 data: 3 3 3 3 2 2 2
5744 282/300 Read block 246 len 4096
5745 283/300 Write block 1822 len 24576 data: 2 2 2 2 2 2
5746 284/300 Flush
5747 285/300 Read block 801 len 20480
5748 286/300 Read block 2589 len 24576
5749 287/300 Read block 4252 len 24576
5750 288/300 Write block 3211 len 20480 data: 2 2 2 3 3
5751 289/300 Write block 2610 len 28672 data: 3 3 2 2 2 2 2
5752 290/300 Write block 189 len 8192 data: 3 3
5753 291/300 Write block 149 len 28672 data: 3 3 2 3 4 4 4
5754 292/300 Read block 139 len 24576
5755 293/300 Write block 1589 len 20480 data: 4 4 4 5 5
5756 294/300 Write block 4083 len 12288 data: 3 3 3
5757 295/300 Write block 4728 len 40960 data: 2 2 2 2 2 3 3 3 3 4
5758 296/300 Read block 3662 len 4096
5759 297/300 Read block 1671 len 12288
5760 298/300 Read block 2419 len 36864
5761 299/300 Flush
5762 300/300 Read block 3178 len 12288
5763 301/300 Write block 2521 len 16384 data: 2 2 3 2
5764 Main task has work to do, go find it
5765 got dsc Start(0)
5766 start 0
5767 Mar 17 22:20:15.328 INFO request completed, latency_us: 254, response_code: 204, uri: /start/cid/0, method: POST, req_id: c9d8bd2b-72db-4569-9eb7-323ef94b47cf, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
5768 Replay: started 0, returned:Ok(())
5769 ----------------------------------------------------------------
5770 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5771 States: Offline Active Active
5772 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5773 2208 Acked Write 10 Sent Done Done false
5774 2209 Acked Write 6 Sent Done Done false
5775 2210 Acked Write 10 Sent Done Done false
5776 2211 Acked Flush 0 Sent Done Done false
5777 2212 Acked Write 4 Sent Done Done false
5778 2213 Acked Write 1 Sent Done Done false
5779 2214 Acked Write 6 Sent Done Done false
5780 2215 Acked Flush 0 Sent Done Done false
5781 2216 Acked Read 1 Sent Done Done false
5782 2217 Acked Read 3 Sent Done Done false
5783 2218 Acked Write 9 Sent Done Done false
5784 2219 Acked Write 4 Sent Done Done false
5785 2220 Acked Write 3 Sent Done Done false
5786 2221 Acked Read 7 Sent Done Done false
5787 2222 Acked Read 6 Sent Done Done false
5788 2223 Acked Write 10 Sent Done Done false
5789 2224 Acked Read 6 Sent Done Done false
5790 2225 Acked Flush 0 Sent Done Done false
5791 2226 Acked Write 7 Sent Done Done false
5792 2227 Acked Write 7 Sent Done Done false
5793 2228 Acked Read 1 Sent Done Done false
5794 2229 Acked Write 5 Sent Done Done false
5795 2230 Acked Flush 0 Sent Done Done false
5796 2231 Acked Read 7 Sent Done Done false
5797 2232 Acked Read 10 Sent Done Done false
5798 2233 Acked Read 6 Sent Done Done false
5799 2234 Acked Read 3 Sent Done Done false
5800 2235 Acked Write 7 Sent Done Done false
5801 2236 Acked Write 4 Sent Done Done false
5802 2237 Acked Read 8 Sent Done Done false
5803 2238 Acked Read 9 Sent Done Done false
5804 2239 Acked Read 2 Sent Done Done false
5805 2240 Acked Read 6 Sent Done Done false
5806 2241 Acked Read 10 Sent Done Done false
5807 2242 Acked Read 10 Sent Done Done false
5808 2243 Acked Write 10 Sent Done Done false
5809 2244 Acked Write 9 Sent Done Done false
5810 2245 Acked Write 4 Sent Done Done false
5811 2246 Acked Read 10 Sent Done Done false
5812 2247 Acked Read 8 Sent Done Done false
5813 2248 Acked Read 2 Sent Done Done false
5814 2249 Acked Write 10 Sent Done Done false
5815 2250 Acked Write 7 Sent Done Done false
5816 2251 Acked Write 1 Sent Done Done false
5817 2252 Acked Write 4 Sent Done Done false
5818 2253 Acked Read 7 Sent Done Done false
5819 2254 Acked Read 1 Sent Done Done false
5820 2255 Acked Read 3 Sent Done Done false
5821 2256 Acked Read 2 Sent Done Done false
5822 2257 Acked Read 9 Sent Done Done false
5823 2258 Acked Write 3 Sent Done Done false
5824 2259 Acked Read 10 Sent Done Done false
5825 2260 Acked Read 5 Sent Done Done false
5826 2261 Acked Write 8 Sent Done Done false
5827 2262 Acked Read 9 Sent Done Done false
5828 2263 Acked Flush 0 Sent Done Done false
5829 2264 Acked Write 7 Sent Done Done false
5830 2265 Acked Write 5 Sent Done Done false
5831 2266 Acked Read 9 Sent Done Done false
5832 2267 Acked Write 5 Sent Done Done false
5833 2268 Acked Write 6 Sent Done Done false
5834 2269 Acked Write 3 Sent Done Done false
5835 2270 Acked Read 10 Sent Done Done false
5836 2271 Acked Read 10 Sent Done Done false
5837 2272 Acked Read 5 Sent Done Done false
5838 2273 Acked Read 5 Sent Done Done false
5839 2274 Acked Write 10 Sent Done Done false
5840 2275 Acked Read 6 Sent Done Done false
5841 2276 Acked Write 3 Sent Done Done false
5842 2277 Acked Write 9 Sent Done Done false
5843 2278 Acked Write 10 Sent Done Done false
5844 2279 Acked Read 3 Sent Done Done false
5845 2280 Acked Read 2 Sent Done Done false
5846 2281 Acked Read 7 Sent Done Done false
5847 2282 Acked Flush 0 Sent Done Done false
5848 2283 Acked Write 6 Sent Done Done false
5849 2284 Acked Read 3 Sent Done Done false
5850 2285 Acked Read 2 Sent Done Done false
5851 2286 Acked Read 5 Sent Done Done false
5852 2287 Acked Read 4 Sent Done Done false
5853 2288 Acked Read 3 Sent Done Done false
5854 2289 Acked Flush 0 Sent Done Done false
5855 2290 Acked Read 4 Sent Done Done false
5856 2291 Acked Read 3 Sent Done Done false
5857 2292 Acked Flush 0 Sent Done Done false
5858 2293 Acked Read 6 Sent Done Done false
5859 2294 Acked Write 1 Sent Done Done false
5860 2295 Acked Write 1 Sent Done Done false
5861 2296 Acked Write 9 Sent Done Done false
5862 2297 Acked Read 7 Sent Done Done false
5863 2298 Acked Read 2 Sent Done Done false
5864 2299 Acked Read 5 Sent Done Done false
5865 2300 Acked Flush 0 Sent Done Done false
5866 2301 Acked Read 6 Sent Done Done false
5867 2302 Acked Read 10 Sent Done Done false
5868 2303 Acked Write 4 Sent Done Done false
5869 2304 Acked Write 7 Sent Done Done false
5870 2305 Acked Read 9 Sent Done Done false
5871 2306 Acked Read 4 Sent Done Done false
5872 2307 Acked Read 2 Sent Done Done false
5873 2308 Acked Read 7 Sent Done Done false
5874 2309 Acked Flush 0 Sent Done Done false
5875 2310 Acked Read 5 Sent Done Done false
5876 2311 Acked Write 1 Sent Done Done false
5877 2312 Acked Write 6 Sent Done Done false
5878 2313 Acked Read 9 Sent Done Done false
5879 2314 Acked Write 1 Sent Done Done false
5880 2315 Acked Read 6 Sent Done Done false
5881 2316 Acked Read 8 Sent Done Done false
5882 2317 Acked Write 10 Sent Done Done false
5883 2318 Acked Write 3 Sent Done Done false
5884 2319 Acked Read 3 Sent Done Done false
5885 2320 Acked Read 10 Sent Done Done false
5886 2321 Acked Read 8 Sent Done Done false
5887 2322 Acked Read 3 Sent Done Done false
5888 2323 Acked Read 5 Sent Done Done false
5889 2324 Acked Write 1 Sent Done Done false
5890 2325 Acked Read 3 Sent Done Done false
5891 2326 Acked Flush 0 Sent Done Done false
5892 2327 Acked Write 6 Sent Done Done false
5893 2328 Acked Read 9 Sent Done Done false
5894 2329 Acked Read 2 Sent Done Done false
5895 2330 Acked Flush 0 Sent Done Done false
5896 2331 Acked Read 7 Sent Done Done false
5897 2332 Acked Write 1 Sent Done Done false
5898 2333 Acked Write 5 Sent Done Done false
5899 2334 Acked Read 7 Sent Done Done false
5900 2335 Acked Read 6 Sent Done Done false
5901 2336 Acked Flush 0 Sent Done Done false
5902 2337 Acked Write 2 Sent Done Done false
5903 2338 Acked Read 6 Sent Done Done false
5904 2339 Acked Flush 0 Sent Done Done false
5905 2340 Acked Read 3 Sent Done Done false
5906 2341 Acked Read 5 Sent Done Done false
5907 2342 Acked Write 3 Sent Done Done false
5908 2343 Acked Write 3 Sent Done Done false
5909 2344 Acked Read 9 Sent Done Done false
5910 2345 Acked Read 9 Sent Done Done false
5911 2346 Acked Read 7 Sent Done Done false
5912 2347 Acked Flush 0 Sent Done Done false
5913 2348 Acked Write 1 Sent Done Done false
5914 2349 Acked Write 5 Sent Done Done false
5915 2350 Acked Read 1 Sent Done Done false
5916 2351 Acked Write 4 Sent Done Done false
5917 2352 Acked Write 6 Sent Done Done false
5918 2353 Acked Read 9 Sent Done Done false
5919 2354 Acked Flush 0 Sent Done Done false
5920 2355 Acked Write 1 Sent Done Done false
5921 2356 Acked Read 6 Sent Done Done false
5922 2357 Acked Flush 0 Sent Done Done false
5923 2358 Acked Read 1 Sent Done Done false
5924 2359 Acked Read 1 Sent Done Done false
5925 2360 Acked Write 8 Sent Done Done false
5926 2361 Acked Flush 0 Sent Done Done false
5927 2362 Acked Read 10 Sent Done Done false
5928 2363 Acked Read 5 Sent Done Done false
5929 2364 Acked Write 1 Sent Done Done false
5930 2365 Acked Write 4 Sent Done Done false
5931 2366 Acked Write 9 Sent Done Done false
5932 2367 Acked Read 9 Sent Done Done false
5933 2368 Acked Flush 0 Sent Done Done false
5934 2369 Acked Read 2 Sent Done Done false
5935 2370 Acked Read 8 Sent Done Done false
5936 2371 Acked Write 5 Sent Done Done false
5937 2372 Acked Write 1 Sent Done Done false
5938 2373 Acked Write 1 Sent Done Done false
5939 2374 Acked Read 9 Sent Done Done false
5940 2375 Acked Read 10 Sent Done Done false
5941 2376 Acked Write 7 Sent Done Done false
5942 2377 Acked Write 4 Sent Done Done false
5943 2378 Acked Write 7 Sent Done Done false
5944 2379 Acked Read 4 Sent Done Done false
5945 2380 Acked Flush 0 Sent Done Done false
5946 2381 Acked Write 6 Sent Done Done false
5947 2382 Acked Read 8 Sent Done Done false
5948 2383 Acked Read 10 Sent Done Done false
5949 2384 Acked Write 4 Sent Done Done false
5950 2385 Acked Read 9 Sent Done Done false
5951 2386 Acked Write 5 Sent Done Done false
5952 2387 Acked Write 4 Sent Done Done false
5953 2388 Acked Read 6 Sent Done Done false
5954 2389 Acked Write 3 Sent Done Done false
5955 2390 Acked Read 9 Sent Done Done false
5956 2391 Acked Read 4 Sent Done Done false
5957 2392 Acked Write 5 Sent Done Done false
5958 2393 Acked Flush 0 Sent Done Done false
5959 2394 Acked Write 2 Sent Done Done false
5960 2395 Acked Read 2 Sent Done Done false
5961 2396 Acked Write 6 Sent Done Done false
5962 2397 Acked Flush 0 Sent Done Done false
5963 2398 Acked Read 5 Sent Done Done false
5964 2399 Acked Read 5 Sent Done Done false
5965 2400 Acked Write 9 Sent Done Done false
5966 2401 Acked Write 9 Sent Done Done false
5967 2402 Acked Flush 0 Sent Done Done false
5968 2403 Acked Read 3 Sent Done Done false
5969 2404 Acked Write 7 Sent Done Done false
5970 2405 Acked Flush 0 Sent Done Done false
5971 2406 Acked Read 1 Sent Done Done false
5972 2407 Acked Read 6 Sent Done Done false
5973 2408 Acked Read 2 Sent Done Done false
5974 2409 Acked Write 9 Sent Done Done false
5975 2410 Acked Write 1 Sent Done Done false
5976 2411 Acked Write 4 Sent Done Done false
5977 2412 Acked Read 7 Sent Done Done false
5978 2413 Acked Read 3 Sent Done Done false
5979 2414 Acked Flush 0 Sent Done Done false
5980 2415 Acked Write 7 Sent Done Done false
5981 2416 Acked Write 10 Sent Done Done false
5982 2417 Acked Read 1 Sent Done Done false
5983 2418 Acked Read 7 Sent Done Done false
5984 2419 Acked Read 8 Sent Done Done false
5985 2420 Acked Read 2 Sent Done Done false
5986 2421 Acked Write 2 Sent Done Done false
5987 2422 Acked Read 5 Sent Done Done false
5988 2423 Acked Read 8 Sent Done Done false
5989 2424 Acked Read 4 Sent Done Done false
5990 2425 Acked Read 9 Sent Done Done false
5991 2426 Acked Read 1 Sent Done Done false
5992 2427 Acked Read 2 Sent Done Done false
5993 2428 Acked Read 7 Sent Done Done false
5994 2429 Acked Write 7 Sent Done Done false
5995 2430 Acked Flush 0 Sent Done Done false
5996 2431 Acked Read 1 Sent Done Done false
5997 2432 Acked Read 1 Sent Done Done false
5998 2433 Acked Read 6 Sent Done Done false
5999 2434 Acked Write 2 Sent Done Done false
6000 2435 Acked Read 4 Sent Done Done false
6001 2436 Acked Write 3 Sent Done Done false
6002 2437 Acked Read 5 Sent Done Done false
6003 2438 Acked Read 7 Sent Done Done false
6004 2439 Acked Write 1 Sent Done Done false
6005 2440 Acked Read 2 Sent Done Done false
6006 2441 Acked Read 7 Sent Done Done false
6007 2442 Acked Write 9 Sent Done Done false
6008 2443 Acked Write 6 Sent Done Done false
6009 2444 Acked Write 9 Sent Done Done false
6010 2445 Acked Write 4 Sent Done Done false
6011 2446 Acked Read 3 Sent Done Done false
6012 2447 Acked Read 7 Sent Done Done false
6013 2448 Acked Write 8 Sent Done Done false
6014 2449 Acked Write 8 Sent Done Done false
6015 2450 Acked Flush 0 Sent Done Done false
6016 2451 Acked Write 8 Sent Done Done false
6017 2452 Acked Write 2 Sent Done Done false
6018 2453 Acked Write 9 Sent Done Done false
6019 2454 Acked Write 2 Sent Done Done false
6020 2455 Acked Flush 0 Sent Done Done false
6021 2456 Acked Write 2 Sent Done Done false
6022 2457 Acked Write 5 Sent Done Done false
6023 2458 Acked Read 3 Sent Done Done false
6024 2459 Acked Read 3 Sent Done Done false
6025 2460 Acked Read 1 Sent Done Done false
6026 2461 Acked Read 7 Sent Done Done false
6027 2462 Acked Write 4 Sent Done Done false
6028 2463 Acked Read 4 Sent Done Done false
6029 2464 Acked Flush 0 Sent Done Done false
6030 2465 Acked Read 9 Sent Done Done false
6031 2466 Acked Write 8 Sent Done Done false
6032 2467 Acked Read 1 Sent Done Done false
6033 2468 Acked Write 9 Sent Done Done false
6034 2469 Acked Write 5 Sent Done Done false
6035 2470 Acked Read 3 Sent Done Done false
6036 2471 Acked Write 5 Sent Done Done false
6037 2472 Acked Flush 0 Sent Done Done false
6038 2473 Acked Read 6 Sent Done Done false
6039 2474 Acked Write 6 Sent Done Done false
6040 2475 Acked Write 7 Sent Done Done false
6041 2476 Acked Read 10 Sent Done Done false
6042 2477 Acked Write 6 Sent Done Done false
6043 2478 Acked Flush 0 Sent Done Done false
6044 2479 Acked Write 9 Sent Done Done false
6045 2480 Acked Read 7 Sent Done Done false
6046 2481 Acked Flush 0 Sent Done Done false
6047 2482 Acked Write 7 Sent Done Done false
6048 2483 Acked Read 8 Sent Done Done false
6049 2484 Acked Read 1 Sent Done Done false
6050 2485 Acked Read 9 Sent Done Done false
6051 2486 Acked Read 10 Sent Done Done false
6052 2487 Acked Write 7 Sent Done Done false
6053 2488 Acked Write 7 Sent Done Done false
6054 2489 Acked Read 1 Sent Done Done false
6055 2490 Acked Write 6 Sent Done Done false
6056 2491 Acked Flush 0 Sent Done Done false
6057 2492 Acked Read 5 Sent Done Done false
6058 2493 Acked Read 6 Sent Done Done false
6059 2494 Acked Read 6 Sent Done Done false
6060 2495 Acked Write 5 Sent Done Done false
6061 2496 Acked Write 7 Sent Done Done false
6062 2497 Acked Write 2 Sent Done Done false
6063 2498 Acked Write 7 Sent Done Done false
6064 2499 Acked Read 6 Sent Done Done false
6065 2500 Acked Write 5 Sent Done Done false
6066 2501 Acked Write 3 Sent Done Done false
6067 2502 Acked Write 10 Sent Done Done false
6068 2503 Acked Read 1 Sent Done Done false
6069 2504 Acked Read 3 Sent Done Done false
6070 2505 Acked Read 9 Sent Done Done false
6071 2506 Acked Flush 0 Sent Done Done false
6072 2507 Acked Read 3 Sent Done Done false
6073 2508 Acked Write 4 Sent Done Done false
6074 STATES DS:0 DS:1 DS:2 TOTAL
6075 Sent 301 0 0 301
6076 Done 0 301 301 602
6077 Skipped 0 0 0 0
6078 Error 0 0 0 0
6079 Last Flush: Some(JobId(2207)) Some(JobId(2506)) Some(JobId(2506))
6080 Downstairs last five completed: 2207 2206 2205 2204 2203
6081 Upstairs last five completed: 2508 2507 2506 2505 2504
6082 CLIENT: Up:0 ds:301 act:2
60832025-03-17T22:20:15.830ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
60842025-03-17T22:20:16.677ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
60852025-03-17T22:20:17.678ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
60862025-03-17T22:20:18.680ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
6087 ----------------------------------------------------------------
6088 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6089 States: Offline Active Active
6090 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6091 2208 Acked Write 10 Sent Done Done false
6092 2209 Acked Write 6 Sent Done Done false
6093 2210 Acked Write 10 Sent Done Done false
6094 2211 Acked Flush 0 Sent Done Done false
6095 2212 Acked Write 4 Sent Done Done false
6096 2213 Acked Write 1 Sent Done Done false
6097 2214 Acked Write 6 Sent Done Done false
6098 2215 Acked Flush 0 Sent Done Done false
6099 2216 Acked Read 1 Sent Done Done false
6100 2217 Acked Read 3 Sent Done Done false
6101 2218 Acked Write 9 Sent Done Done false
6102 2219 Acked Write 4 Sent Done Done false
6103 2220 Acked Write 3 Sent Done Done false
6104 2221 Acked Read 7 Sent Done Done false
6105 2222 Acked Read 6 Sent Done Done false
6106 2223 Acked Write 10 Sent Done Done false
6107 2224 Acked Read 6 Sent Done Done false
6108 2225 Acked Flush 0 Sent Done Done false
6109 2226 Acked Write 7 Sent Done Done false
6110 2227 Acked Write 7 Sent Done Done false
6111 2228 Acked Read 1 Sent Done Done false
6112 2229 Acked Write 5 Sent Done Done false
6113 2230 Acked Flush 0 Sent Done Done false
6114 2231 Acked Read 7 Sent Done Done false
6115 2232 Acked Read 10 Sent Done Done false
6116 2233 Acked Read 6 Sent Done Done false
6117 2234 Acked Read 3 Sent Done Done false
6118 2235 Acked Write 7 Sent Done Done false
6119 2236 Acked Write 4 Sent Done Done false
6120 2237 Acked Read 8 Sent Done Done false
6121 2238 Acked Read 9 Sent Done Done false
6122 2239 Acked Read 2 Sent Done Done false
6123 2240 Acked Read 6 Sent Done Done false
6124 2241 Acked Read 10 Sent Done Done false
6125 2242 Acked Read 10 Sent Done Done false
6126 2243 Acked Write 10 Sent Done Done false
6127 2244 Acked Write 9 Sent Done Done false
6128 2245 Acked Write 4 Sent Done Done false
6129 2246 Acked Read 10 Sent Done Done false
6130 2247 Acked Read 8 Sent Done Done false
6131 2248 Acked Read 2 Sent Done Done false
6132 2249 Acked Write 10 Sent Done Done false
6133 2250 Acked Write 7 Sent Done Done false
6134 2251 Acked Write 1 Sent Done Done false
6135 2252 Acked Write 4 Sent Done Done false
6136 2253 Acked Read 7 Sent Done Done false
6137 2254 Acked Read 1 Sent Done Done false
6138 2255 Acked Read 3 Sent Done Done false
6139 2256 Acked Read 2 Sent Done Done false
6140 2257 Acked Read 9 Sent Done Done false
6141 2258 Acked Write 3 Sent Done Done false
6142 2259 Acked Read 10 Sent Done Done false
6143 2260 Acked Read 5 Sent Done Done false
6144 2261 Acked Write 8 Sent Done Done false
6145 2262 Acked Read 9 Sent Done Done false
6146 2263 Acked Flush 0 Sent Done Done false
6147 2264 Acked Write 7 Sent Done Done false
6148 2265 Acked Write 5 Sent Done Done false
6149 2266 Acked Read 9 Sent Done Done false
6150 2267 Acked Write 5 Sent Done Done false
6151 2268 Acked Write 6 Sent Done Done false
6152 2269 Acked Write 3 Sent Done Done false
6153 2270 Acked Read 10 Sent Done Done false
6154 2271 Acked Read 10 Sent Done Done false
6155 2272 Acked Read 5 Sent Done Done false
6156 2273 Acked Read 5 Sent Done Done false
6157 2274 Acked Write 10 Sent Done Done false
6158 2275 Acked Read 6 Sent Done Done false
6159 2276 Acked Write 3 Sent Done Done false
6160 2277 Acked Write 9 Sent Done Done false
6161 2278 Acked Write 10 Sent Done Done false
6162 2279 Acked Read 3 Sent Done Done false
6163 2280 Acked Read 2 Sent Done Done false
6164 2281 Acked Read 7 Sent Done Done false
6165 2282 Acked Flush 0 Sent Done Done false
6166 2283 Acked Write 6 Sent Done Done false
6167 2284 Acked Read 3 Sent Done Done false
6168 2285 Acked Read 2 Sent Done Done false
6169 2286 Acked Read 5 Sent Done Done false
6170 2287 Acked Read 4 Sent Done Done false
6171 2288 Acked Read 3 Sent Done Done false
6172 2289 Acked Flush 0 Sent Done Done false
6173 2290 Acked Read 4 Sent Done Done false
6174 2291 Acked Read 3 Sent Done Done false
6175 2292 Acked Flush 0 Sent Done Done false
6176 2293 Acked Read 6 Sent Done Done false
6177 2294 Acked Write 1 Sent Done Done false
6178 2295 Acked Write 1 Sent Done Done false
6179 2296 Acked Write 9 Sent Done Done false
6180 2297 Acked Read 7 Sent Done Done false
6181 2298 Acked Read 2 Sent Done Done false
6182 2299 Acked Read 5 Sent Done Done false
6183 2300 Acked Flush 0 Sent Done Done false
6184 2301 Acked Read 6 Sent Done Done false
6185 2302 Acked Read 10 Sent Done Done false
6186 2303 Acked Write 4 Sent Done Done false
6187 2304 Acked Write 7 Sent Done Done false
6188 2305 Acked Read 9 Sent Done Done false
6189 2306 Acked Read 4 Sent Done Done false
6190 2307 Acked Read 2 Sent Done Done false
6191 2308 Acked Read 7 Sent Done Done false
6192 2309 Acked Flush 0 Sent Done Done false
6193 2310 Acked Read 5 Sent Done Done false
6194 2311 Acked Write 1 Sent Done Done false
6195 2312 Acked Write 6 Sent Done Done false
6196 2313 Acked Read 9 Sent Done Done false
6197 2314 Acked Write 1 Sent Done Done false
6198 2315 Acked Read 6 Sent Done Done false
6199 2316 Acked Read 8 Sent Done Done false
6200 2317 Acked Write 10 Sent Done Done false
6201 2318 Acked Write 3 Sent Done Done false
6202 2319 Acked Read 3 Sent Done Done false
6203 2320 Acked Read 10 Sent Done Done false
6204 2321 Acked Read 8 Sent Done Done false
6205 2322 Acked Read 3 Sent Done Done false
6206 2323 Acked Read 5 Sent Done Done false
6207 2324 Acked Write 1 Sent Done Done false
6208 2325 Acked Read 3 Sent Done Done false
6209 2326 Acked Flush 0 Sent Done Done false
6210 2327 Acked Write 6 Sent Done Done false
6211 2328 Acked Read 9 Sent Done Done false
6212 2329 Acked Read 2 Sent Done Done false
6213 2330 Acked Flush 0 Sent Done Done false
6214 2331 Acked Read 7 Sent Done Done false
6215 2332 Acked Write 1 Sent Done Done false
6216 2333 Acked Write 5 Sent Done Done false
6217 2334 Acked Read 7 Sent Done Done false
6218 2335 Acked Read 6 Sent Done Done false
6219 2336 Acked Flush 0 Sent Done Done false
6220 2337 Acked Write 2 Sent Done Done false
6221 2338 Acked Read 6 Sent Done Done false
6222 2339 Acked Flush 0 Sent Done Done false
6223 2340 Acked Read 3 Sent Done Done false
6224 2341 Acked Read 5 Sent Done Done false
6225 2342 Acked Write 3 Sent Done Done false
6226 2343 Acked Write 3 Sent Done Done false
6227 2344 Acked Read 9 Sent Done Done false
6228 2345 Acked Read 9 Sent Done Done false
6229 2346 Acked Read 7 Sent Done Done false
6230 2347 Acked Flush 0 Sent Done Done false
6231 2348 Acked Write 1 Sent Done Done false
6232 2349 Acked Write 5 Sent Done Done false
6233 2350 Acked Read 1 Sent Done Done false
6234 2351 Acked Write 4 Sent Done Done false
6235 2352 Acked Write 6 Sent Done Done false
6236 2353 Acked Read 9 Sent Done Done false
6237 2354 Acked Flush 0 Sent Done Done false
6238 2355 Acked Write 1 Sent Done Done false
6239 2356 Acked Read 6 Sent Done Done false
6240 2357 Acked Flush 0 Sent Done Done false
6241 2358 Acked Read 1 Sent Done Done false
6242 2359 Acked Read 1 Sent Done Done false
6243 2360 Acked Write 8 Sent Done Done false
6244 2361 Acked Flush 0 Sent Done Done false
6245 2362 Acked Read 10 Sent Done Done false
6246 2363 Acked Read 5 Sent Done Done false
6247 2364 Acked Write 1 Sent Done Done false
6248 2365 Acked Write 4 Sent Done Done false
6249 2366 Acked Write 9 Sent Done Done false
6250 2367 Acked Read 9 Sent Done Done false
6251 2368 Acked Flush 0 Sent Done Done false
6252 2369 Acked Read 2 Sent Done Done false
6253 2370 Acked Read 8 Sent Done Done false
6254 2371 Acked Write 5 Sent Done Done false
6255 2372 Acked Write 1 Sent Done Done false
6256 2373 Acked Write 1 Sent Done Done false
6257 2374 Acked Read 9 Sent Done Done false
6258 2375 Acked Read 10 Sent Done Done false
6259 2376 Acked Write 7 Sent Done Done false
6260 2377 Acked Write 4 Sent Done Done false
6261 2378 Acked Write 7 Sent Done Done false
6262 2379 Acked Read 4 Sent Done Done false
6263 2380 Acked Flush 0 Sent Done Done false
6264 2381 Acked Write 6 Sent Done Done false
6265 2382 Acked Read 8 Sent Done Done false
6266 2383 Acked Read 10 Sent Done Done false
6267 2384 Acked Write 4 Sent Done Done false
6268 2385 Acked Read 9 Sent Done Done false
6269 2386 Acked Write 5 Sent Done Done false
6270 2387 Acked Write 4 Sent Done Done false
6271 2388 Acked Read 6 Sent Done Done false
6272 2389 Acked Write 3 Sent Done Done false
6273 2390 Acked Read 9 Sent Done Done false
6274 2391 Acked Read 4 Sent Done Done false
6275 2392 Acked Write 5 Sent Done Done false
6276 2393 Acked Flush 0 Sent Done Done false
6277 2394 Acked Write 2 Sent Done Done false
6278 2395 Acked Read 2 Sent Done Done false
6279 2396 Acked Write 6 Sent Done Done false
6280 2397 Acked Flush 0 Sent Done Done false
6281 2398 Acked Read 5 Sent Done Done false
6282 2399 Acked Read 5 Sent Done Done false
6283 2400 Acked Write 9 Sent Done Done false
6284 2401 Acked Write 9 Sent Done Done false
6285 2402 Acked Flush 0 Sent Done Done false
6286 2403 Acked Read 3 Sent Done Done false
6287 2404 Acked Write 7 Sent Done Done false
6288 2405 Acked Flush 0 Sent Done Done false
6289 2406 Acked Read 1 Sent Done Done false
6290 2407 Acked Read 6 Sent Done Done false
6291 2408 Acked Read 2 Sent Done Done false
6292 2409 Acked Write 9 Sent Done Done false
6293 2410 Acked Write 1 Sent Done Done false
6294 2411 Acked Write 4 Sent Done Done false
6295 2412 Acked Read 7 Sent Done Done false
6296 2413 Acked Read 3 Sent Done Done false
6297 2414 Acked Flush 0 Sent Done Done false
6298 2415 Acked Write 7 Sent Done Done false
6299 2416 Acked Write 10 Sent Done Done false
6300 2417 Acked Read 1 Sent Done Done false
6301 2418 Acked Read 7 Sent Done Done false
6302 2419 Acked Read 8 Sent Done Done false
6303 2420 Acked Read 2 Sent Done Done false
6304 2421 Acked Write 2 Sent Done Done false
6305 2422 Acked Read 5 Sent Done Done false
6306 2423 Acked Read 8 Sent Done Done false
6307 2424 Acked Read 4 Sent Done Done false
6308 2425 Acked Read 9 Sent Done Done false
6309 2426 Acked Read 1 Sent Done Done false
6310 2427 Acked Read 2 Sent Done Done false
6311 2428 Acked Read 7 Sent Done Done false
6312 2429 Acked Write 7 Sent Done Done false
6313 2430 Acked Flush 0 Sent Done Done false
6314 2431 Acked Read 1 Sent Done Done false
6315 2432 Acked Read 1 Sent Done Done false
6316 2433 Acked Read 6 Sent Done Done false
6317 2434 Acked Write 2 Sent Done Done false
6318 2435 Acked Read 4 Sent Done Done false
6319 2436 Acked Write 3 Sent Done Done false
6320 2437 Acked Read 5 Sent Done Done false
6321 2438 Acked Read 7 Sent Done Done false
6322 2439 Acked Write 1 Sent Done Done false
6323 2440 Acked Read 2 Sent Done Done false
6324 2441 Acked Read 7 Sent Done Done false
6325 2442 Acked Write 9 Sent Done Done false
6326 2443 Acked Write 6 Sent Done Done false
6327 2444 Acked Write 9 Sent Done Done false
6328 2445 Acked Write 4 Sent Done Done false
6329 2446 Acked Read 3 Sent Done Done false
6330 2447 Acked Read 7 Sent Done Done false
6331 2448 Acked Write 8 Sent Done Done false
6332 2449 Acked Write 8 Sent Done Done false
6333 2450 Acked Flush 0 Sent Done Done false
6334 2451 Acked Write 8 Sent Done Done false
6335 2452 Acked Write 2 Sent Done Done false
6336 2453 Acked Write 9 Sent Done Done false
6337 2454 Acked Write 2 Sent Done Done false
6338 2455 Acked Flush 0 Sent Done Done false
6339 2456 Acked Write 2 Sent Done Done false
6340 2457 Acked Write 5 Sent Done Done false
6341 2458 Acked Read 3 Sent Done Done false
6342 2459 Acked Read 3 Sent Done Done false
6343 2460 Acked Read 1 Sent Done Done false
6344 2461 Acked Read 7 Sent Done Done false
6345 2462 Acked Write 4 Sent Done Done false
6346 2463 Acked Read 4 Sent Done Done false
6347 2464 Acked Flush 0 Sent Done Done false
6348 2465 Acked Read 9 Sent Done Done false
6349 2466 Acked Write 8 Sent Done Done false
6350 2467 Acked Read 1 Sent Done Done false
6351 2468 Acked Write 9 Sent Done Done false
6352 2469 Acked Write 5 Sent Done Done false
6353 2470 Acked Read 3 Sent Done Done false
6354 2471 Acked Write 5 Sent Done Done false
6355 2472 Acked Flush 0 Sent Done Done false
6356 2473 Acked Read 6 Sent Done Done false
6357 2474 Acked Write 6 Sent Done Done false
6358 2475 Acked Write 7 Sent Done Done false
6359 2476 Acked Read 10 Sent Done Done false
6360 2477 Acked Write 6 Sent Done Done false
6361 2478 Acked Flush 0 Sent Done Done false
6362 2479 Acked Write 9 Sent Done Done false
6363 2480 Acked Read 7 Sent Done Done false
6364 2481 Acked Flush 0 Sent Done Done false
6365 2482 Acked Write 7 Sent Done Done false
6366 2483 Acked Read 8 Sent Done Done false
6367 2484 Acked Read 1 Sent Done Done false
6368 2485 Acked Read 9 Sent Done Done false
6369 2486 Acked Read 10 Sent Done Done false
6370 2487 Acked Write 7 Sent Done Done false
6371 2488 Acked Write 7 Sent Done Done false
6372 2489 Acked Read 1 Sent Done Done false
6373 2490 Acked Write 6 Sent Done Done false
6374 2491 Acked Flush 0 Sent Done Done false
6375 2492 Acked Read 5 Sent Done Done false
6376 2493 Acked Read 6 Sent Done Done false
6377 2494 Acked Read 6 Sent Done Done false
6378 2495 Acked Write 5 Sent Done Done false
6379 2496 Acked Write 7 Sent Done Done false
6380 2497 Acked Write 2 Sent Done Done false
6381 2498 Acked Write 7 Sent Done Done false
6382 2499 Acked Read 6 Sent Done Done false
6383 2500 Acked Write 5 Sent Done Done false
6384 2501 Acked Write 3 Sent Done Done false
6385 2502 Acked Write 10 Sent Done Done false
6386 2503 Acked Read 1 Sent Done Done false
6387 2504 Acked Read 3 Sent Done Done false
6388 2505 Acked Read 9 Sent Done Done false
6389 2506 Acked Flush 0 Sent Done Done false
6390 2507 Acked Read 3 Sent Done Done false
6391 2508 Acked Write 4 Sent Done Done false
6392 STATES DS:0 DS:1 DS:2 TOTAL
6393 Sent 301 0 0 301
6394 Done 0 301 301 602
6395 Skipped 0 0 0 0
6396 Error 0 0 0 0
6397 Last Flush: Some(JobId(2207)) Some(JobId(2506)) Some(JobId(2506))
6398 Downstairs last five completed: 2207 2206 2205 2204 2203
6399 Upstairs last five completed: 2508 2507 2506 2505 2504
6400 CLIENT: Up:0 ds:301 act:2
64012025-03-17T22:20:20.108ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
64022025-03-17T22:20:20.684ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
64032025-03-17T22:20:21.086ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
64042025-03-17T22:20:21.087ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
64052025-03-17T22:20:21.087ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
64062025-03-17T22:20:21.087ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
64072025-03-17T22:20:21.087ZINFOcrucible: client task is exiting = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
6408 [8810] got start action
6409 [8810] I am going to restart
6410 Starting downstairs at port 8810
6411 Make output file at "/tmp/dsc/downstairs-8810.txt"
6412 [8810][0] reports Starting
6413 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1216)
6414 [8810][0] reports Running
64152025-03-17T22:20:21.686ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
64162025-03-17T22:20:22.687ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
64172025-03-17T22:20:23.188ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
6418 ----------------------------------------------------------------
6419 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6420 States: Offline Active Active
6421 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6422 2208 Acked Write 10 Sent Done Done false
6423 2209 Acked Write 6 Sent Done Done false
6424 2210 Acked Write 10 Sent Done Done false
6425 2211 Acked Flush 0 Sent Done Done false
6426 2212 Acked Write 4 Sent Done Done false
6427 2213 Acked Write 1 Sent Done Done false
6428 2214 Acked Write 6 Sent Done Done false
6429 2215 Acked Flush 0 Sent Done Done false
6430 2216 Acked Read 1 Sent Done Done false
6431 2217 Acked Read 3 Sent Done Done false
6432 2218 Acked Write 9 Sent Done Done false
6433 2219 Acked Write 4 Sent Done Done false
6434 2220 Acked Write 3 Sent Done Done false
6435 2221 Acked Read 7 Sent Done Done false
6436 2222 Acked Read 6 Sent Done Done false
6437 2223 Acked Write 10 Sent Done Done false
6438 2224 Acked Read 6 Sent Done Done false
6439 2225 Acked Flush 0 Sent Done Done false
6440 2226 Acked Write 7 Sent Done Done false
6441 2227 Acked Write 7 Sent Done Done false
6442 2228 Acked Read 1 Sent Done Done false
6443 2229 Acked Write 5 Sent Done Done false
6444 2230 Acked Flush 0 Sent Done Done false
6445 2231 Acked Read 7 Sent Done Done false
6446 2232 Acked Read 10 Sent Done Done false
6447 2233 Acked Read 6 Sent Done Done false
6448 2234 Acked Read 3 Sent Done Done false
6449 2235 Acked Write 7 Sent Done Done false
6450 2236 Acked Write 4 Sent Done Done false
6451 2237 Acked Read 8 Sent Done Done false
6452 2238 Acked Read 9 Sent Done Done false
6453 2239 Acked Read 2 Sent Done Done false
6454 2240 Acked Read 6 Sent Done Done false
6455 2241 Acked Read 10 Sent Done Done false
6456 2242 Acked Read 10 Sent Done Done false
6457 2243 Acked Write 10 Sent Done Done false
6458 2244 Acked Write 9 Sent Done Done false
6459 2245 Acked Write 4 Sent Done Done false
6460 2246 Acked Read 10 Sent Done Done false
6461 2247 Acked Read 8 Sent Done Done false
6462 2248 Acked Read 2 Sent Done Done false
6463 2249 Acked Write 10 Sent Done Done false
6464 2250 Acked Write 7 Sent Done Done false
6465 2251 Acked Write 1 Sent Done Done false
6466 2252 Acked Write 4 Sent Done Done false
6467 2253 Acked Read 7 Sent Done Done false
6468 2254 Acked Read 1 Sent Done Done false
6469 2255 Acked Read 3 Sent Done Done false
6470 2256 Acked Read 2 Sent Done Done false
6471 2257 Acked Read 9 Sent Done Done false
6472 2258 Acked Write 3 Sent Done Done false
6473 2259 Acked Read 10 Sent Done Done false
6474 2260 Acked Read 5 Sent Done Done false
6475 2261 Acked Write 8 Sent Done Done false
6476 2262 Acked Read 9 Sent Done Done false
6477 2263 Acked Flush 0 Sent Done Done false
6478 2264 Acked Write 7 Sent Done Done false
6479 2265 Acked Write 5 Sent Done Done false
6480 2266 Acked Read 9 Sent Done Done false
6481 2267 Acked Write 5 Sent Done Done false
6482 2268 Acked Write 6 Sent Done Done false
6483 2269 Acked Write 3 Sent Done Done false
6484 2270 Acked Read 10 Sent Done Done false
6485 2271 Acked Read 10 Sent Done Done false
6486 2272 Acked Read 5 Sent Done Done false
6487 2273 Acked Read 5 Sent Done Done false
6488 2274 Acked Write 10 Sent Done Done false
6489 2275 Acked Read 6 Sent Done Done false
6490 2276 Acked Write 3 Sent Done Done false
6491 2277 Acked Write 9 Sent Done Done false
6492 2278 Acked Write 10 Sent Done Done false
6493 2279 Acked Read 3 Sent Done Done false
6494 2280 Acked Read 2 Sent Done Done false
6495 2281 Acked Read 7 Sent Done Done false
6496 2282 Acked Flush 0 Sent Done Done false
6497 2283 Acked Write 6 Sent Done Done false
6498 2284 Acked Read 3 Sent Done Done false
6499 2285 Acked Read 2 Sent Done Done false
6500 2286 Acked Read 5 Sent Done Done false
6501 2287 Acked Read 4 Sent Done Done false
6502 2288 Acked Read 3 Sent Done Done false
6503 2289 Acked Flush 0 Sent Done Done false
6504 2290 Acked Read 4 Sent Done Done false
6505 2291 Acked Read 3 Sent Done Done false
6506 2292 Acked Flush 0 Sent Done Done false
6507 2293 Acked Read 6 Sent Done Done false
6508 2294 Acked Write 1 Sent Done Done false
6509 2295 Acked Write 1 Sent Done Done false
6510 2296 Acked Write 9 Sent Done Done false
6511 2297 Acked Read 7 Sent Done Done false
6512 2298 Acked Read 2 Sent Done Done false
6513 2299 Acked Read 5 Sent Done Done false
6514 2300 Acked Flush 0 Sent Done Done false
6515 2301 Acked Read 6 Sent Done Done false
6516 2302 Acked Read 10 Sent Done Done false
6517 2303 Acked Write 4 Sent Done Done false
6518 2304 Acked Write 7 Sent Done Done false
6519 2305 Acked Read 9 Sent Done Done false
6520 2306 Acked Read 4 Sent Done Done false
6521 2307 Acked Read 2 Sent Done Done false
6522 2308 Acked Read 7 Sent Done Done false
6523 2309 Acked Flush 0 Sent Done Done false
6524 2310 Acked Read 5 Sent Done Done false
6525 2311 Acked Write 1 Sent Done Done false
6526 2312 Acked Write 6 Sent Done Done false
6527 2313 Acked Read 9 Sent Done Done false
6528 2314 Acked Write 1 Sent Done Done false
6529 2315 Acked Read 6 Sent Done Done false
6530 2316 Acked Read 8 Sent Done Done false
6531 2317 Acked Write 10 Sent Done Done false
6532 2318 Acked Write 3 Sent Done Done false
6533 2319 Acked Read 3 Sent Done Done false
6534 2320 Acked Read 10 Sent Done Done false
6535 2321 Acked Read 8 Sent Done Done false
6536 2322 Acked Read 3 Sent Done Done false
6537 2323 Acked Read 5 Sent Done Done false
6538 2324 Acked Write 1 Sent Done Done false
6539 2325 Acked Read 3 Sent Done Done false
6540 2326 Acked Flush 0 Sent Done Done false
6541 2327 Acked Write 6 Sent Done Done false
6542 2328 Acked Read 9 Sent Done Done false
6543 2329 Acked Read 2 Sent Done Done false
6544 2330 Acked Flush 0 Sent Done Done false
6545 2331 Acked Read 7 Sent Done Done false
6546 2332 Acked Write 1 Sent Done Done false
6547 2333 Acked Write 5 Sent Done Done false
6548 2334 Acked Read 7 Sent Done Done false
6549 2335 Acked Read 6 Sent Done Done false
6550 2336 Acked Flush 0 Sent Done Done false
6551 2337 Acked Write 2 Sent Done Done false
6552 2338 Acked Read 6 Sent Done Done false
6553 2339 Acked Flush 0 Sent Done Done false
6554 2340 Acked Read 3 Sent Done Done false
6555 2341 Acked Read 5 Sent Done Done false
6556 2342 Acked Write 3 Sent Done Done false
6557 2343 Acked Write 3 Sent Done Done false
6558 2344 Acked Read 9 Sent Done Done false
6559 2345 Acked Read 9 Sent Done Done false
6560 2346 Acked Read 7 Sent Done Done false
6561 2347 Acked Flush 0 Sent Done Done false
6562 2348 Acked Write 1 Sent Done Done false
6563 2349 Acked Write 5 Sent Done Done false
6564 2350 Acked Read 1 Sent Done Done false
6565 2351 Acked Write 4 Sent Done Done false
6566 2352 Acked Write 6 Sent Done Done false
6567 2353 Acked Read 9 Sent Done Done false
6568 2354 Acked Flush 0 Sent Done Done false
6569 2355 Acked Write 1 Sent Done Done false
6570 2356 Acked Read 6 Sent Done Done false
6571 2357 Acked Flush 0 Sent Done Done false
6572 2358 Acked Read 1 Sent Done Done false
6573 2359 Acked Read 1 Sent Done Done false
6574 2360 Acked Write 8 Sent Done Done false
6575 2361 Acked Flush 0 Sent Done Done false
6576 2362 Acked Read 10 Sent Done Done false
6577 2363 Acked Read 5 Sent Done Done false
6578 2364 Acked Write 1 Sent Done Done false
6579 2365 Acked Write 4 Sent Done Done false
6580 2366 Acked Write 9 Sent Done Done false
6581 2367 Acked Read 9 Sent Done Done false
6582 2368 Acked Flush 0 Sent Done Done false
6583 2369 Acked Read 2 Sent Done Done false
6584 2370 Acked Read 8 Sent Done Done false
6585 2371 Acked Write 5 Sent Done Done false
6586 2372 Acked Write 1 Sent Done Done false
6587 2373 Acked Write 1 Sent Done Done false
6588 2374 Acked Read 9 Sent Done Done false
6589 2375 Acked Read 10 Sent Done Done false
6590 2376 Acked Write 7 Sent Done Done false
6591 2377 Acked Write 4 Sent Done Done false
6592 2378 Acked Write 7 Sent Done Done false
6593 2379 Acked Read 4 Sent Done Done false
6594 2380 Acked Flush 0 Sent Done Done false
6595 2381 Acked Write 6 Sent Done Done false
6596 2382 Acked Read 8 Sent Done Done false
6597 2383 Acked Read 10 Sent Done Done false
6598 2384 Acked Write 4 Sent Done Done false
6599 2385 Acked Read 9 Sent Done Done false
6600 2386 Acked Write 5 Sent Done Done false
6601 2387 Acked Write 4 Sent Done Done false
6602 2388 Acked Read 6 Sent Done Done false
6603 2389 Acked Write 3 Sent Done Done false
6604 2390 Acked Read 9 Sent Done Done false
6605 2391 Acked Read 4 Sent Done Done false
6606 2392 Acked Write 5 Sent Done Done false
6607 2393 Acked Flush 0 Sent Done Done false
6608 2394 Acked Write 2 Sent Done Done false
6609 2395 Acked Read 2 Sent Done Done false
6610 2396 Acked Write 6 Sent Done Done false
6611 2397 Acked Flush 0 Sent Done Done false
6612 2398 Acked Read 5 Sent Done Done false
6613 2399 Acked Read 5 Sent Done Done false
6614 2400 Acked Write 9 Sent Done Done false
6615 2401 Acked Write 9 Sent Done Done false
6616 2402 Acked Flush 0 Sent Done Done false
6617 2403 Acked Read 3 Sent Done Done false
6618 2404 Acked Write 7 Sent Done Done false
6619 2405 Acked Flush 0 Sent Done Done false
6620 2406 Acked Read 1 Sent Done Done false
6621 2407 Acked Read 6 Sent Done Done false
6622 2408 Acked Read 2 Sent Done Done false
6623 2409 Acked Write 9 Sent Done Done false
6624 2410 Acked Write 1 Sent Done Done false
6625 2411 Acked Write 4 Sent Done Done false
6626 2412 Acked Read 7 Sent Done Done false
6627 2413 Acked Read 3 Sent Done Done false
6628 2414 Acked Flush 0 Sent Done Done false
6629 2415 Acked Write 7 Sent Done Done false
6630 2416 Acked Write 10 Sent Done Done false
6631 2417 Acked Read 1 Sent Done Done false
6632 2418 Acked Read 7 Sent Done Done false
6633 2419 Acked Read 8 Sent Done Done false
6634 2420 Acked Read 2 Sent Done Done false
6635 2421 Acked Write 2 Sent Done Done false
6636 2422 Acked Read 5 Sent Done Done false
6637 2423 Acked Read 8 Sent Done Done false
6638 2424 Acked Read 4 Sent Done Done false
6639 2425 Acked Read 9 Sent Done Done false
6640 2426 Acked Read 1 Sent Done Done false
6641 2427 Acked Read 2 Sent Done Done false
6642 2428 Acked Read 7 Sent Done Done false
6643 2429 Acked Write 7 Sent Done Done false
6644 2430 Acked Flush 0 Sent Done Done false
6645 2431 Acked Read 1 Sent Done Done false
6646 2432 Acked Read 1 Sent Done Done false
6647 2433 Acked Read 6 Sent Done Done false
6648 2434 Acked Write 2 Sent Done Done false
6649 2435 Acked Read 4 Sent Done Done false
6650 2436 Acked Write 3 Sent Done Done false
6651 2437 Acked Read 5 Sent Done Done false
6652 2438 Acked Read 7 Sent Done Done false
6653 2439 Acked Write 1 Sent Done Done false
6654 2440 Acked Read 2 Sent Done Done false
6655 2441 Acked Read 7 Sent Done Done false
6656 2442 Acked Write 9 Sent Done Done false
6657 2443 Acked Write 6 Sent Done Done false
6658 2444 Acked Write 9 Sent Done Done false
6659 2445 Acked Write 4 Sent Done Done false
6660 2446 Acked Read 3 Sent Done Done false
6661 2447 Acked Read 7 Sent Done Done false
6662 2448 Acked Write 8 Sent Done Done false
6663 2449 Acked Write 8 Sent Done Done false
6664 2450 Acked Flush 0 Sent Done Done false
6665 2451 Acked Write 8 Sent Done Done false
6666 2452 Acked Write 2 Sent Done Done false
6667 2453 Acked Write 9 Sent Done Done false
6668 2454 Acked Write 2 Sent Done Done false
6669 2455 Acked Flush 0 Sent Done Done false
6670 2456 Acked Write 2 Sent Done Done false
6671 2457 Acked Write 5 Sent Done Done false
6672 2458 Acked Read 3 Sent Done Done false
6673 2459 Acked Read 3 Sent Done Done false
6674 2460 Acked Read 1 Sent Done Done false
6675 2461 Acked Read 7 Sent Done Done false
6676 2462 Acked Write 4 Sent Done Done false
6677 2463 Acked Read 4 Sent Done Done false
6678 2464 Acked Flush 0 Sent Done Done false
6679 2465 Acked Read 9 Sent Done Done false
6680 2466 Acked Write 8 Sent Done Done false
6681 2467 Acked Read 1 Sent Done Done false
6682 2468 Acked Write 9 Sent Done Done false
6683 2469 Acked Write 5 Sent Done Done false
6684 2470 Acked Read 3 Sent Done Done false
6685 2471 Acked Write 5 Sent Done Done false
6686 2472 Acked Flush 0 Sent Done Done false
6687 2473 Acked Read 6 Sent Done Done false
6688 2474 Acked Write 6 Sent Done Done false
6689 2475 Acked Write 7 Sent Done Done false
6690 2476 Acked Read 10 Sent Done Done false
6691 2477 Acked Write 6 Sent Done Done false
6692 2478 Acked Flush 0 Sent Done Done false
6693 2479 Acked Write 9 Sent Done Done false
6694 2480 Acked Read 7 Sent Done Done false
6695 2481 Acked Flush 0 Sent Done Done false
6696 2482 Acked Write 7 Sent Done Done false
6697 2483 Acked Read 8 Sent Done Done false
6698 2484 Acked Read 1 Sent Done Done false
6699 2485 Acked Read 9 Sent Done Done false
6700 2486 Acked Read 10 Sent Done Done false
6701 2487 Acked Write 7 Sent Done Done false
6702 2488 Acked Write 7 Sent Done Done false
6703 2489 Acked Read 1 Sent Done Done false
6704 2490 Acked Write 6 Sent Done Done false
6705 2491 Acked Flush 0 Sent Done Done false
6706 2492 Acked Read 5 Sent Done Done false
6707 2493 Acked Read 6 Sent Done Done false
6708 2494 Acked Read 6 Sent Done Done false
6709 2495 Acked Write 5 Sent Done Done false
6710 2496 Acked Write 7 Sent Done Done false
6711 2497 Acked Write 2 Sent Done Done false
6712 2498 Acked Write 7 Sent Done Done false
6713 2499 Acked Read 6 Sent Done Done false
6714 2500 Acked Write 5 Sent Done Done false
6715 2501 Acked Write 3 Sent Done Done false
6716 2502 Acked Write 10 Sent Done Done false
6717 2503 Acked Read 1 Sent Done Done false
6718 2504 Acked Read 3 Sent Done Done false
6719 2505 Acked Read 9 Sent Done Done false
6720 2506 Acked Flush 0 Sent Done Done false
6721 2507 Acked Read 3 Sent Done Done false
6722 2508 Acked Write 4 Sent Done Done false
6723 STATES DS:0 DS:1 DS:2 TOTAL
6724 Sent 301 0 0 301
6725 Done 0 301 301 602
6726 Skipped 0 0 0 0
6727 Error 0 0 0 0
6728 Last Flush: Some(JobId(2207)) Some(JobId(2506)) Some(JobId(2506))
6729 Downstairs last five completed: 2207 2206 2205 2204 2203
6730 Upstairs last five completed: 2508 2507 2506 2505 2504
6731 CLIENT: Up:0 ds:301 act:2
67322025-03-17T22:20:23.834ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
67332025-03-17T22:20:25.108ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
67342025-03-17T22:20:25.693ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
67352025-03-17T22:20:26.695ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
6736 ----------------------------------------------------------------
6737 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6738 States: Offline Active Active
6739 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6740 2208 Acked Write 10 Sent Done Done false
6741 2209 Acked Write 6 Sent Done Done false
6742 2210 Acked Write 10 Sent Done Done false
6743 2211 Acked Flush 0 Sent Done Done false
6744 2212 Acked Write 4 Sent Done Done false
6745 2213 Acked Write 1 Sent Done Done false
6746 2214 Acked Write 6 Sent Done Done false
6747 2215 Acked Flush 0 Sent Done Done false
6748 2216 Acked Read 1 Sent Done Done false
6749 2217 Acked Read 3 Sent Done Done false
6750 2218 Acked Write 9 Sent Done Done false
6751 2219 Acked Write 4 Sent Done Done false
6752 2220 Acked Write 3 Sent Done Done false
6753 2221 Acked Read 7 Sent Done Done false
6754 2222 Acked Read 6 Sent Done Done false
6755 2223 Acked Write 10 Sent Done Done false
6756 2224 Acked Read 6 Sent Done Done false
6757 2225 Acked Flush 0 Sent Done Done false
6758 2226 Acked Write 7 Sent Done Done false
6759 2227 Acked Write 7 Sent Done Done false
6760 2228 Acked Read 1 Sent Done Done false
6761 2229 Acked Write 5 Sent Done Done false
6762 2230 Acked Flush 0 Sent Done Done false
6763 2231 Acked Read 7 Sent Done Done false
6764 2232 Acked Read 10 Sent Done Done false
6765 2233 Acked Read 6 Sent Done Done false
6766 2234 Acked Read 3 Sent Done Done false
6767 2235 Acked Write 7 Sent Done Done false
6768 2236 Acked Write 4 Sent Done Done false
6769 2237 Acked Read 8 Sent Done Done false
6770 2238 Acked Read 9 Sent Done Done false
6771 2239 Acked Read 2 Sent Done Done false
6772 2240 Acked Read 6 Sent Done Done false
6773 2241 Acked Read 10 Sent Done Done false
6774 2242 Acked Read 10 Sent Done Done false
6775 2243 Acked Write 10 Sent Done Done false
6776 2244 Acked Write 9 Sent Done Done false
6777 2245 Acked Write 4 Sent Done Done false
6778 2246 Acked Read 10 Sent Done Done false
6779 2247 Acked Read 8 Sent Done Done false
6780 2248 Acked Read 2 Sent Done Done false
6781 2249 Acked Write 10 Sent Done Done false
6782 2250 Acked Write 7 Sent Done Done false
6783 2251 Acked Write 1 Sent Done Done false
6784 2252 Acked Write 4 Sent Done Done false
6785 2253 Acked Read 7 Sent Done Done false
6786 2254 Acked Read 1 Sent Done Done false
6787 2255 Acked Read 3 Sent Done Done false
6788 2256 Acked Read 2 Sent Done Done false
6789 2257 Acked Read 9 Sent Done Done false
6790 2258 Acked Write 3 Sent Done Done false
6791 2259 Acked Read 10 Sent Done Done false
6792 2260 Acked Read 5 Sent Done Done false
6793 2261 Acked Write 8 Sent Done Done false
6794 2262 Acked Read 9 Sent Done Done false
6795 2263 Acked Flush 0 Sent Done Done false
6796 2264 Acked Write 7 Sent Done Done false
6797 2265 Acked Write 5 Sent Done Done false
6798 2266 Acked Read 9 Sent Done Done false
6799 2267 Acked Write 5 Sent Done Done false
6800 2268 Acked Write 6 Sent Done Done false
6801 2269 Acked Write 3 Sent Done Done false
6802 2270 Acked Read 10 Sent Done Done false
6803 2271 Acked Read 10 Sent Done Done false
6804 2272 Acked Read 5 Sent Done Done false
6805 2273 Acked Read 5 Sent Done Done false
6806 2274 Acked Write 10 Sent Done Done false
6807 2275 Acked Read 6 Sent Done Done false
6808 2276 Acked Write 3 Sent Done Done false
6809 2277 Acked Write 9 Sent Done Done false
6810 2278 Acked Write 10 Sent Done Done false
6811 2279 Acked Read 3 Sent Done Done false
6812 2280 Acked Read 2 Sent Done Done false
6813 2281 Acked Read 7 Sent Done Done false
6814 2282 Acked Flush 0 Sent Done Done false
6815 2283 Acked Write 6 Sent Done Done false
6816 2284 Acked Read 3 Sent Done Done false
6817 2285 Acked Read 2 Sent Done Done false
6818 2286 Acked Read 5 Sent Done Done false
6819 2287 Acked Read 4 Sent Done Done false
6820 2288 Acked Read 3 Sent Done Done false
6821 2289 Acked Flush 0 Sent Done Done false
6822 2290 Acked Read 4 Sent Done Done false
6823 2291 Acked Read 3 Sent Done Done false
6824 2292 Acked Flush 0 Sent Done Done false
6825 2293 Acked Read 6 Sent Done Done false
6826 2294 Acked Write 1 Sent Done Done false
6827 2295 Acked Write 1 Sent Done Done false
6828 2296 Acked Write 9 Sent Done Done false
6829 2297 Acked Read 7 Sent Done Done false
6830 2298 Acked Read 2 Sent Done Done false
6831 2299 Acked Read 5 Sent Done Done false
6832 2300 Acked Flush 0 Sent Done Done false
6833 2301 Acked Read 6 Sent Done Done false
6834 2302 Acked Read 10 Sent Done Done false
6835 2303 Acked Write 4 Sent Done Done false
6836 2304 Acked Write 7 Sent Done Done false
6837 2305 Acked Read 9 Sent Done Done false
6838 2306 Acked Read 4 Sent Done Done false
6839 2307 Acked Read 2 Sent Done Done false
6840 2308 Acked Read 7 Sent Done Done false
6841 2309 Acked Flush 0 Sent Done Done false
6842 2310 Acked Read 5 Sent Done Done false
6843 2311 Acked Write 1 Sent Done Done false
6844 2312 Acked Write 6 Sent Done Done false
6845 2313 Acked Read 9 Sent Done Done false
6846 2314 Acked Write 1 Sent Done Done false
6847 2315 Acked Read 6 Sent Done Done false
6848 2316 Acked Read 8 Sent Done Done false
6849 2317 Acked Write 10 Sent Done Done false
6850 2318 Acked Write 3 Sent Done Done false
6851 2319 Acked Read 3 Sent Done Done false
6852 2320 Acked Read 10 Sent Done Done false
6853 2321 Acked Read 8 Sent Done Done false
6854 2322 Acked Read 3 Sent Done Done false
6855 2323 Acked Read 5 Sent Done Done false
6856 2324 Acked Write 1 Sent Done Done false
6857 2325 Acked Read 3 Sent Done Done false
6858 2326 Acked Flush 0 Sent Done Done false
6859 2327 Acked Write 6 Sent Done Done false
6860 2328 Acked Read 9 Sent Done Done false
6861 2329 Acked Read 2 Sent Done Done false
6862 2330 Acked Flush 0 Sent Done Done false
6863 2331 Acked Read 7 Sent Done Done false
6864 2332 Acked Write 1 Sent Done Done false
6865 2333 Acked Write 5 Sent Done Done false
6866 2334 Acked Read 7 Sent Done Done false
6867 2335 Acked Read 6 Sent Done Done false
6868 2336 Acked Flush 0 Sent Done Done false
6869 2337 Acked Write 2 Sent Done Done false
6870 2338 Acked Read 6 Sent Done Done false
6871 2339 Acked Flush 0 Sent Done Done false
6872 2340 Acked Read 3 Sent Done Done false
6873 2341 Acked Read 5 Sent Done Done false
6874 2342 Acked Write 3 Sent Done Done false
6875 2343 Acked Write 3 Sent Done Done false
6876 2344 Acked Read 9 Sent Done Done false
6877 2345 Acked Read 9 Sent Done Done false
6878 2346 Acked Read 7 Sent Done Done false
6879 2347 Acked Flush 0 Sent Done Done false
6880 2348 Acked Write 1 Sent Done Done false
6881 2349 Acked Write 5 Sent Done Done false
6882 2350 Acked Read 1 Sent Done Done false
6883 2351 Acked Write 4 Sent Done Done false
6884 2352 Acked Write 6 Sent Done Done false
6885 2353 Acked Read 9 Sent Done Done false
6886 2354 Acked Flush 0 Sent Done Done false
6887 2355 Acked Write 1 Sent Done Done false
6888 2356 Acked Read 6 Sent Done Done false
6889 2357 Acked Flush 0 Sent Done Done false
6890 2358 Acked Read 1 Sent Done Done false
6891 2359 Acked Read 1 Sent Done Done false
6892 2360 Acked Write 8 Sent Done Done false
6893 2361 Acked Flush 0 Sent Done Done false
6894 2362 Acked Read 10 Sent Done Done false
6895 2363 Acked Read 5 Sent Done Done false
6896 2364 Acked Write 1 Sent Done Done false
6897 2365 Acked Write 4 Sent Done Done false
6898 2366 Acked Write 9 Sent Done Done false
6899 2367 Acked Read 9 Sent Done Done false
6900 2368 Acked Flush 0 Sent Done Done false
6901 2369 Acked Read 2 Sent Done Done false
6902 2370 Acked Read 8 Sent Done Done false
6903 2371 Acked Write 5 Sent Done Done false
6904 2372 Acked Write 1 Sent Done Done false
6905 2373 Acked Write 1 Sent Done Done false
6906 2374 Acked Read 9 Sent Done Done false
6907 2375 Acked Read 10 Sent Done Done false
6908 2376 Acked Write 7 Sent Done Done false
6909 2377 Acked Write 4 Sent Done Done false
6910 2378 Acked Write 7 Sent Done Done false
6911 2379 Acked Read 4 Sent Done Done false
6912 2380 Acked Flush 0 Sent Done Done false
6913 2381 Acked Write 6 Sent Done Done false
6914 2382 Acked Read 8 Sent Done Done false
6915 2383 Acked Read 10 Sent Done Done false
6916 2384 Acked Write 4 Sent Done Done false
6917 2385 Acked Read 9 Sent Done Done false
6918 2386 Acked Write 5 Sent Done Done false
6919 2387 Acked Write 4 Sent Done Done false
6920 2388 Acked Read 6 Sent Done Done false
6921 2389 Acked Write 3 Sent Done Done false
6922 2390 Acked Read 9 Sent Done Done false
6923 2391 Acked Read 4 Sent Done Done false
6924 2392 Acked Write 5 Sent Done Done false
6925 2393 Acked Flush 0 Sent Done Done false
6926 2394 Acked Write 2 Sent Done Done false
6927 2395 Acked Read 2 Sent Done Done false
6928 2396 Acked Write 6 Sent Done Done false
6929 2397 Acked Flush 0 Sent Done Done false
6930 2398 Acked Read 5 Sent Done Done false
6931 2399 Acked Read 5 Sent Done Done false
6932 2400 Acked Write 9 Sent Done Done false
6933 2401 Acked Write 9 Sent Done Done false
6934 2402 Acked Flush 0 Sent Done Done false
6935 2403 Acked Read 3 Sent Done Done false
6936 2404 Acked Write 7 Sent Done Done false
6937 2405 Acked Flush 0 Sent Done Done false
6938 2406 Acked Read 1 Sent Done Done false
6939 2407 Acked Read 6 Sent Done Done false
6940 2408 Acked Read 2 Sent Done Done false
6941 2409 Acked Write 9 Sent Done Done false
6942 2410 Acked Write 1 Sent Done Done false
6943 2411 Acked Write 4 Sent Done Done false
6944 2412 Acked Read 7 Sent Done Done false
6945 2413 Acked Read 3 Sent Done Done false
6946 2414 Acked Flush 0 Sent Done Done false
6947 2415 Acked Write 7 Sent Done Done false
6948 2416 Acked Write 10 Sent Done Done false
6949 2417 Acked Read 1 Sent Done Done false
6950 2418 Acked Read 7 Sent Done Done false
6951 2419 Acked Read 8 Sent Done Done false
6952 2420 Acked Read 2 Sent Done Done false
6953 2421 Acked Write 2 Sent Done Done false
6954 2422 Acked Read 5 Sent Done Done false
6955 2423 Acked Read 8 Sent Done Done false
6956 2424 Acked Read 4 Sent Done Done false
6957 2425 Acked Read 9 Sent Done Done false
6958 2426 Acked Read 1 Sent Done Done false
6959 2427 Acked Read 2 Sent Done Done false
6960 2428 Acked Read 7 Sent Done Done false
6961 2429 Acked Write 7 Sent Done Done false
6962 2430 Acked Flush 0 Sent Done Done false
6963 2431 Acked Read 1 Sent Done Done false
6964 2432 Acked Read 1 Sent Done Done false
6965 2433 Acked Read 6 Sent Done Done false
6966 2434 Acked Write 2 Sent Done Done false
6967 2435 Acked Read 4 Sent Done Done false
6968 2436 Acked Write 3 Sent Done Done false
6969 2437 Acked Read 5 Sent Done Done false
6970 2438 Acked Read 7 Sent Done Done false
6971 2439 Acked Write 1 Sent Done Done false
6972 2440 Acked Read 2 Sent Done Done false
6973 2441 Acked Read 7 Sent Done Done false
6974 2442 Acked Write 9 Sent Done Done false
6975 2443 Acked Write 6 Sent Done Done false
6976 2444 Acked Write 9 Sent Done Done false
6977 2445 Acked Write 4 Sent Done Done false
6978 2446 Acked Read 3 Sent Done Done false
6979 2447 Acked Read 7 Sent Done Done false
6980 2448 Acked Write 8 Sent Done Done false
6981 2449 Acked Write 8 Sent Done Done false
6982 2450 Acked Flush 0 Sent Done Done false
6983 2451 Acked Write 8 Sent Done Done false
6984 2452 Acked Write 2 Sent Done Done false
6985 2453 Acked Write 9 Sent Done Done false
6986 2454 Acked Write 2 Sent Done Done false
6987 2455 Acked Flush 0 Sent Done Done false
6988 2456 Acked Write 2 Sent Done Done false
6989 2457 Acked Write 5 Sent Done Done false
6990 2458 Acked Read 3 Sent Done Done false
6991 2459 Acked Read 3 Sent Done Done false
6992 2460 Acked Read 1 Sent Done Done false
6993 2461 Acked Read 7 Sent Done Done false
6994 2462 Acked Write 4 Sent Done Done false
6995 2463 Acked Read 4 Sent Done Done false
6996 2464 Acked Flush 0 Sent Done Done false
6997 2465 Acked Read 9 Sent Done Done false
6998 2466 Acked Write 8 Sent Done Done false
6999 2467 Acked Read 1 Sent Done Done false
7000 2468 Acked Write 9 Sent Done Done false
7001 2469 Acked Write 5 Sent Done Done false
7002 2470 Acked Read 3 Sent Done Done false
7003 2471 Acked Write 5 Sent Done Done false
7004 2472 Acked Flush 0 Sent Done Done false
7005 2473 Acked Read 6 Sent Done Done false
7006 2474 Acked Write 6 Sent Done Done false
7007 2475 Acked Write 7 Sent Done Done false
7008 2476 Acked Read 10 Sent Done Done false
7009 2477 Acked Write 6 Sent Done Done false
7010 2478 Acked Flush 0 Sent Done Done false
7011 2479 Acked Write 9 Sent Done Done false
7012 2480 Acked Read 7 Sent Done Done false
7013 2481 Acked Flush 0 Sent Done Done false
7014 2482 Acked Write 7 Sent Done Done false
7015 2483 Acked Read 8 Sent Done Done false
7016 2484 Acked Read 1 Sent Done Done false
7017 2485 Acked Read 9 Sent Done Done false
7018 2486 Acked Read 10 Sent Done Done false
7019 2487 Acked Write 7 Sent Done Done false
7020 2488 Acked Write 7 Sent Done Done false
7021 2489 Acked Read 1 Sent Done Done false
7022 2490 Acked Write 6 Sent Done Done false
7023 2491 Acked Flush 0 Sent Done Done false
7024 2492 Acked Read 5 Sent Done Done false
7025 2493 Acked Read 6 Sent Done Done false
7026 2494 Acked Read 6 Sent Done Done false
7027 2495 Acked Write 5 Sent Done Done false
7028 2496 Acked Write 7 Sent Done Done false
7029 2497 Acked Write 2 Sent Done Done false
7030 2498 Acked Write 7 Sent Done Done false
7031 2499 Acked Read 6 Sent Done Done false
7032 2500 Acked Write 5 Sent Done Done false
7033 2501 Acked Write 3 Sent Done Done false
7034 2502 Acked Write 10 Sent Done Done false
7035 2503 Acked Read 1 Sent Done Done false
7036 2504 Acked Read 3 Sent Done Done false
7037 2505 Acked Read 9 Sent Done Done false
7038 2506 Acked Flush 0 Sent Done Done false
7039 2507 Acked Read 3 Sent Done Done false
7040 2508 Acked Write 4 Sent Done Done false
7041 STATES DS:0 DS:1 DS:2 TOTAL
7042 Sent 301 0 0 301
7043 Done 0 301 301 602
7044 Skipped 0 0 0 0
7045 Error 0 0 0 0
7046 Last Flush: Some(JobId(2207)) Some(JobId(2506)) Some(JobId(2506))
7047 Downstairs last five completed: 2207 2206 2205 2204 2203
7048 Upstairs last five completed: 2508 2507 2506 2505 2504
7049 CLIENT: Up:0 ds:301 act:2
70502025-03-17T22:20:27.836ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
70512025-03-17T22:20:28.699ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
70522025-03-17T22:20:30.110ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
70532025-03-17T22:20:30.702ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
70542025-03-17T22:20:31.088ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
70552025-03-17T22:20:31.089ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:33327) = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
70562025-03-17T22:20:31.090ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
70572025-03-17T22:20:31.090ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
70582025-03-17T22:20:31.090ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
70592025-03-17T22:20:31.090ZINFOcrucible: send last flush ID to this DS: Some(JobId(2207)) = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
70602025-03-17T22:20:31.090ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
70612025-03-17T22:20:31.090ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(2207)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
70622025-03-17T22:20:31.091ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(2207)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
7063 ----------------------------------------------------------------
7064 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:2
7065 States: Active Active Active
7066 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7067 2507 Acked Read 3 Done Done Done true
7068 2508 Acked Write 4 Done Done Done true
7069 STATES DS:0 DS:1 DS:2 TOTAL
7070 Sent 0 0 0 0
7071 Done 2 2 2 6
7072 Skipped 0 0 0 0
7073 Error 0 0 0 0
7074 Last Flush: Some(JobId(2506)) Some(JobId(2506)) Some(JobId(2506))
7075 Downstairs last five completed: 2506 2505 2504 2503 2502
7076 Upstairs last five completed: 2508 2507 2506 2505 2504
7077 CLIENT: Up:0 ds:2 act:3
7078 ----------------------------------------------------------------
7079 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
7080 Downstairs last five completed: 2509 2508 2507 2506 2505
7081 Upstairs last five completed: 2509 2508 2507 2506 2505
7082 CLIENT: Up:0 ds:0 act:3
7083 Replay: All jobs finished
7084 Main task has work to do, go find it
7085 got dsc Stop(1)
7086 stop 1
7087 Mar 17 22:20:35.339 INFO request completed, latency_us: 331, response_code: 204, uri: /stop/cid/1, method: POST, req_id: [8820] Got stop action so:false kr:false
7088 3e930db4-06fa-4cdc-a13a-332bd3c8be80, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
70892025-03-17T22:20:35.340ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
7090 Mar 17 22:20:35.340{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:20:35.340755446Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
7091 INFO request completed, latency_us: 114, response_code: 200, uri: /state/cid/1, method: GET{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:20:35.340786893Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
7092 , req_id: ff1274aa-b131-4f96-83f5-3650e8b10c9e, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
70932025-03-17T22:20:35.340ZINFOcrucible: client task is exiting = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
7094 [8820] Exited with: ExitStatus(unix_wait_status(9))
7095 [8820][1] reports Exit
7096 Mar 17 22:20:39.343 INFO request completed, latency_us: 211, response_code: 200, uri: /state/cid/1, method: GET, req_id: 607d036a-11a4-4bc3-96ae-096beb8d0e2f, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
7097 001/300 Write block 4550 len 20480 data: 4 4 3 2 3
7098 002/300 Read block 2567 len 32768
7099 003/300 Write block 2582 len 32768 data: 3 4 4 4 4 2 2 2
7100 004/300 Flush
7101 005/300 Write block 494 len 36864 data: 3 3 2 2 2 2 2 2 2
7102 006/300 Read block 311 len 16384
7103 007/300 Write block 3665 len 4096 data: 3
7104 008/300 Read block 4701 len 12288
7105 009/300 Write block 4204 len 32768 data: 2 2 2 2 2 2 3 2
7106 010/300 Read block 3803 len 24576
7107 011/300 Write block 2221 len 32768 data: 3 3 3 4 3 3 4 3
7108 012/300 Write block 802 len 32768 data: 2 2 2 2 2 2 3 4
7109 013/300 Read block 1249 len 32768
7110 014/300 Write block 126 len 28672 data: 2 2 2 2 2 3 3
7111 015/300 Read block 218 len 12288
7112 016/300 Write block 1957 len 16384 data: 2 2 2 2
7113 017/300 Read block 2717 len 36864
7114 018/300 Read block 4353 len 36864
7115 019/300 Write block 1136 len 40960 data: 3 3 3 3 3 3 3 2 2 2
7116 020/300 Write block 849 len 36864 data: 2 2 2 2 2 2 2 2 3
7117 021/300 Read block 3667 len 28672
7118 022/300 Read block 4197 len 8192
7119 023/300 Flush
7120 024/300 Write block 2735 len 12288 data: 4 3 4
7121 025/300 Read block 2169 len 24576
7122 026/300 Read block 416 len 28672
7123 027/300 Write block 3495 len 28672 data: 3 3 2 2 3 3 3
7124 028/300 Read block 4741 len 40960
7125 029/300 Write block 1906 len 20480 data: 2 3 4 4 4
7126 030/300 Flush
7127 031/300 Read block 3753 len 36864
7128 032/300 Flush
7129 033/300 Write block 3828 len 8192 data: 2 2
7130 034/300 Read block 3235 len 28672
7131 035/300 Read block 1081 len 20480
7132 036/300 Flush
7133 037/300 Read block 598 len 16384
7134 038/300 Flush
7135 039/300 Write block 2086 len 32768 data: 2 2 2 2 2 2 2 2
7136 040/300 Write block 2709 len 28672 data: 3 2 2 2 2 2 2
7137 041/300 Read block 2126 len 4096
7138 042/300 Write block 4833 len 4096 data: 2
7139 043/300 Read block 2699 len 4096
7140 044/300 Write block 2237 len 16384 data: 2 2 2 2
7141 045/300 Read block 3535 len 12288
7142 046/300 Write block 4820 len 24576 data: 3 3 3 3 3 2
7143 047/300 Read block 2782 len 28672
7144 048/300 Read block 1235 len 16384
7145 049/300 Write block 3635 len 24576 data: 2 2 2 2 2 3
7146 050/300 Write block 154 len 8192 data: 5 5
7147 051/300 Write block 944 len 24576 data: 3 4 4 4 4 4
7148 052/300 Read block 1694 len 36864
7149 053/300 Write block 3849 len 8192 data: 3 3
7150 054/300 Read block 4757 len 36864
7151 055/300 Write block 3489 len 36864 data: 3 4 4 3 3 3 4 4 3
7152 056/300 Read block 1246 len 8192
7153 057/300 Write block 4081 len 20480 data: 2 3 4 4 4
7154 058/300 Write block 294 len 20480 data: 3 3 3 3 2
7155 059/300 Read block 1777 len 40960
7156 060/300 Write block 435 len 12288 data: 3 3 2
7157 061/300 Write block 3768 len 40960 data: 4 4 3 2 2 3 4 4 4 3
7158 062/300 Read block 4873 len 32768
7159 063/300 Read block 2633 len 28672
7160 064/300 Read block 3672 len 20480
7161 065/300 Flush
7162 066/300 Read block 195 len 28672
7163 067/300 Write block 2881 len 36864 data: 4 3 3 3 2 2 2 2 2
7164 068/300 Read block 2825 len 32768
7165 069/300 Read block 4002 len 4096
7166 070/300 Read block 1548 len 28672
7167 071/300 Write block 2372 len 36864 data: 2 2 2 2 2 2 2 2 2
7168 072/300 Write block 3056 len 28672 data: 5 5 4 3 3 3 3
7169 073/300 Write block 3806 len 40960 data: 2 2 2 2 2 2 2 2 2 2
7170 074/300 Write block 1391 len 8192 data: 2 2
7171 075/300 Read block 2724 len 8192
7172 076/300 Write block 3336 len 20480 data: 2 2 2 2 2
7173 077/300 Write block 1995 len 12288 data: 4 3 3
7174 078/300 Read block 692 len 28672
7175 079/300 Read block 3701 len 24576
7176 080/300 Read block 678 len 40960
7177 081/300 Read block 293 len 12288
7178 082/300 Write block 569 len 40960 data: 2 2 2 3 4 4 4 4 4 4
7179 083/300 Write block 2531 len 12288 data: 3 3 2
7180 084/300 Read block 2449 len 12288
7181 085/300 Write block 2675 len 4096 data: 2
7182 086/300 Flush
7183 087/300 Write block 2313 len 12288 data: 6 7 6
7184 088/300 Write block 405 len 12288 data: 3 2 2
7185 089/300 Read block 2817 len 8192
7186 090/300 Read block 2505 len 24576
7187 091/300 Read block 4397 len 32768
7188 092/300 Flush
7189 093/300 Write block 4710 len 32768 data: 2 3 2 2 3 3 3 3
7190 094/300 Read block 320 len 32768
7191 095/300 Read block 4470 len 16384
7192 096/300 Flush
7193 097/300 Read block 2481 len 28672
7194 098/300 Write block 3192 len 20480 data: 2 3 3 3 3
7195 099/300 Write block 1577 len 32768 data: 2 3 4 3 2 2 3 3
7196 100/300 Write block 2992 len 12288 data: 2 2 2
7197 101/300 Read block 588 len 40960
7198 102/300 Read block 4122 len 8192
7199 103/300 Read block 2495 len 8192
7200 104/300 Write block 4151 len 28672 data: 4 3 3 2 2 2 2
7201 105/300 Read block 4437 len 28672
7202 106/300 Flush
7203 107/300 Read block 1588 len 32768
7204 108/300 Write block 3549 len 16384 data: 4 4 4 3
7205 109/300 Write block 1869 len 20480 data: 4 4 4 4 2
7206 110/300 Write block 2178 len 16384 data: 2 2 2 2
7207 111/300 Read block 1518 len 36864
7208 112/300 Read block 2832 len 12288
7209 113/300 Read block 2471 len 36864
7210 114/300 Write block 1837 len 4096 data: 3
7211 115/300 Write block 551 len 4096 data: 2
7212 116/300 Read block 1189 len 12288
7213 117/300 Write block 846 len 20480 data: 3 3 2 3 3
7214 118/300 Write block 4666 len 40960 data: 2 2 2 3 3 3 2 2 2 2
7215 119/300 Read block 1520 len 16384
7216 120/300 Write block 4503 len 8192 data: 4 2
7217 121/300 Write block 4811 len 20480 data: 2 2 2 2 3
7218 122/300 Read block 1977 len 32768
7219 123/300 Flush
7220 124/300 Write block 1509 len 4096 data: 3
7221 125/300 Read block 4158 len 36864
7222 126/300 Read block 1025 len 20480
7223 127/300 Flush
7224 128/300 Read block 2734 len 28672
7225 129/300 Write block 2454 len 4096 data: 2
7226 130/300 Read block 327 len 24576
7227 131/300 Read block 3194 len 16384
7228 132/300 Read block 4954 len 12288
7229 133/300 Flush
7230 134/300 Write block 1667 len 8192 data: 3 2
7231 135/300 Write block 2908 len 8192 data: 2 2
7232 136/300 Read block 1512 len 20480
7233 137/300 Read block 1016 len 4096
7234 138/300 Read block 3914 len 32768
7235 139/300 Flush
7236 140/300 Read block 3681 len 12288
7237 141/300 Write block 4073 len 12288 data: 3 4 5
7238 142/300 Write block 1166 len 12288 data: 2 3 3
7239 143/300 Read block 472 len 16384
7240 144/300 Write block 2592 len 36864 data: 2 2 2 2 2 2 2 2 2
7241 145/300 Write block 4196 len 16384 data: 4 4 4 3
7242 146/300 Write block 3799 len 4096 data: 2
7243 147/300 Read block 3129 len 32768
7244 148/300 Write block 140 len 16384 data: 2 2 2 3
7245 149/300 Write block 1979 len 32768 data: 3 3 3 3 3 3 3 3
7246 150/300 Write block 1229 len 24576 data: 2 2 2 2 2 2
7247 151/300 Write block 4972 len 28672 data: 2 2 2 2 2 2 2
7248 152/300 Write block 4557 len 32768 data: 3 3 3 2 2 2 2 2
7249 153/300 Write block 2924 len 24576 data: 3 3 2 2 2 2
7250 154/300 Write block 281 len 36864 data: 3 3 3 3 3 3 3 3 4
7251 155/300 Read block 2014 len 36864
7252 156/300 Read block 2080 len 4096
7253 157/300 Read block 2886 len 36864
7254 158/300 Read block 2602 len 8192
7255 159/300 Read block 547 len 8192
7256 160/300 Write block 3583 len 36864 data: 2 2 2 2 2 2 3 3 3
7257 161/300 Write block 1696 len 40960 data: 3 3 3 3 3 2 2 2 2 2
7258 162/300 Read block 4301 len 12288
7259 163/300 Write block 710 len 20480 data: 2 2 2 2 2
7260 164/300 Write block 2446 len 20480 data: 4 4 4 5 4
7261 165/300 Write block 3344 len 32768 data: 3 4 3 4 3 3 3 2
7262 166/300 Read block 1648 len 36864
7263 167/300 Read block 942 len 32768
7264 168/300 Flush
7265 169/300 Write block 4368 len 36864 data: 3 3 3 2 2 2 2 2 2
7266 170/300 Read block 2943 len 40960
7267 171/300 Read block 3928 len 16384
7268 172/300 Flush
7269 173/300 Write block 1372 len 12288 data: 2 2 4
7270 174/300 Write block 4326 len 8192 data: 2 2
7271 175/300 Write block 4751 len 28672 data: 3 3 3 3 3 3 3
7272 176/300 Read block 4419 len 28672
7273 177/300 Flush
7274 178/300 Write block 4254 len 32768 data: 2 2 3 4 4 5 4 4
7275 179/300 Flush
7276 180/300 Write block 3538 len 40960 data: 4 3 3 3 4 4 3 4 4 4
7277 181/300 Write block 587 len 8192 data: 2 2
7278 182/300 Write block 4760 len 12288 data: 3 3 3
7279 183/300 Read block 1282 len 20480
7280 184/300 Flush
7281 185/300 Write block 4135 len 4096 data: 3
7282 186/300 Write block 201 len 40960 data: 3 4 4 3 3 3 3 3 3 2
7283 187/300 Write block 943 len 8192 data: 2 4
7284 188/300 Flush
7285 189/300 Read block 4530 len 16384
7286 190/300 Read block 4533 len 8192
7287 191/300 Read block 4982 len 4096
7288 192/300 Read block 26 len 36864
7289 193/300 Flush
7290 194/300 Read block 2092 len 20480
7291 195/300 Flush
7292 196/300 Write block 1414 len 16384 data: 2 2 2 2
7293 197/300 Write block 3076 len 36864 data: 2 2 3 4 5 5 5 2 2
7294 198/300 Read block 4052 len 32768
7295 199/300 Read block 3688 len 20480
7296 200/300 Read block 3028 len 24576
7297 201/300 Flush
7298 202/300 Read block 4458 len 36864
7299 203/300 Read block 2047 len 12288
7300 204/300 Read block 264 len 8192
7301 205/300 Write block 4090 len 28672 data: 2 2 2 2 2 2 2
7302 206/300 Read block 418 len 12288
7303 207/300 Read block 4636 len 4096
7304 208/300 Write block 4161 len 28672 data: 3 3 3 2 2 3 3
7305 209/300 Read block 1529 len 24576
7306 210/300 Write block 3348 len 12288 data: 4 4 4
7307 211/300 Read block 1561 len 36864
7308 212/300 Read block 2336 len 12288
7309 213/300 Read block 1194 len 24576
7310 214/300 Flush
7311 215/300 Read block 2260 len 24576
7312 216/300 Write block 1831 len 40960 data: 3 3 3 2 2 3 4 3 4 4
7313 217/300 Read block 2381 len 24576
7314 218/300 Read block 1483 len 28672
7315 219/300 Read block 3591 len 12288
7316 220/300 Write block 1315 len 36864 data: 2 2 2 2 2 2 2 3 2
7317 221/300 Write block 475 len 8192 data: 3 3
7318 222/300 Read block 723 len 24576
7319 223/300 Read block 3913 len 40960
7320 224/300 Flush
7321 225/300 Read block 2203 len 32768
7322 226/300 Write block 2763 len 16384 data: 2 3 3 2
7323 227/300 Write block 1553 len 32768 data: 4 4 3 2 2 2 3 3
7324 228/300 Read block 2366 len 8192
7325 229/300 Read block 226 len 36864
7326 230/300 Write block 2053 len 4096 data: 2
7327 231/300 Read block 965 len 32768
7328 232/300 Read block 2683 len 36864
7329 233/300 Flush
7330 234/300 Read block 1927 len 4096
7331 235/300 Read block 1877 len 16384
7332 236/300 Write block 3197 len 40960 data: 5 4 4 4 5 4 4 4 3 3
7333 237/300 Write block 2526 len 20480 data: 2 3 3 3 3
7334 238/300 Write block 2338 len 36864 data: 2 2 2 2 2 2 2 2 2
7335 239/300 Write block 351 len 24576 data: 2 2 2 2 2 2
7336 240/300 Read block 3929 len 32768
7337 241/300 Read block 3057 len 12288
7338 242/300 Write block 1871 len 8192 data: 5 5
7339 243/300 Flush
7340 244/300 Write block 2385 len 12288 data: 3 3 3
7341 245/300 Write block 4543 len 24576 data: 2 2 2 2 3 4
7342 246/300 Write block 4538 len 12288 data: 4 3 3
7343 247/300 Read block 4858 len 16384
7344 248/300 Read block 4030 len 20480
7345 249/300 Read block 865 len 28672
7346 250/300 Flush
7347 251/300 Read block 1402 len 16384
7348 252/300 Read block 3259 len 24576
7349 253/300 Read block 3600 len 8192
7350 254/300 Write block 1239 len 16384 data: 2 2 2 2
7351 255/300 Read block 1305 len 8192
7352 256/300 Read block 1930 len 16384
7353 257/300 Read block 1528 len 32768
7354 258/300 Write block 3507 len 12288 data: 3 3 2
7355 259/300 Write block 2036 len 24576 data: 2 2 2 2 3 3
7356 260/300 Write block 3302 len 8192 data: 3 3
7357 261/300 Write block 3384 len 8192 data: 2 2
7358 262/300 Read block 4325 len 12288
7359 263/300 Flush
7360 264/300 Read block 4575 len 20480
7361 265/300 Write block 3925 len 4096 data: 2
7362 266/300 Flush
7363 267/300 Flush
7364 268/300 Write block 2192 len 12288 data: 4 4 4
7365 269/300 Read block 3647 len 28672
7366 270/300 Flush
7367 271/300 Flush
7368 272/300 Write block 3891 len 8192 data: 2 2
7369 273/300 Read block 4849 len 20480
7370 274/300 Read block 4585 len 32768
7371 275/300 Write block 4482 len 4096 data: 6
7372 276/300 Write block 4444 len 4096 data: 2
7373 277/300 Read block 4522 len 28672
7374 278/300 Read block 1167 len 16384
7375 279/300 Read block 3953 len 32768
7376 280/300 Read block 3670 len 12288
7377 281/300 Write block 4768 len 12288 data: 3 3 3
7378 282/300 Write block 366 len 4096 data: 3
7379 283/300 Read block 3961 len 40960
7380 284/300 Write block 3341 len 40960 data: 2 2 3 4 5 4 5 5 5 5
7381 285/300 Write block 4510 len 28672 data: 3 4 4 4 4 4 4
7382 286/300 Write block 3400 len 32768 data: 4 4 4 4 3 3 4 3
7383 287/300 Write block 3112 len 20480 data: 3 4 4 3 3
7384 288/300 Read block 4039 len 8192
7385 289/300 Read block 2132 len 8192
7386 290/300 Write block 2290 len 8192 data: 3 3
7387 291/300 Write block 1934 len 24576 data: 3 3 3 2 2 2
7388 292/300 Read block 728 len 16384
7389 293/300 Read block 4 len 16384
7390 294/300 Write block 371 len 12288 data: 4 4 4
7391 295/300 Read block 3345 len 4096
7392 296/300 Read block 4473 len 36864
7393 297/300 Read block 1731 len 12288
7394 298/300 Read block 4663 len 12288
7395 299/300 Read block 1067 len 20480
7396 300/300 Flush
7397 301/300 Read block 422 len 8192
7398 Main task has work to do, go find it
7399 got dsc Start(1)
7400 start 1
7401 Mar 17 22:20:39.578 INFO request completed, latency_us: 197, response_code: 204, uri: /start/cid/1, method: POST, req_id: 3d76eb2d-c789-45f2-a022-6580fdf09d61, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
7402 Replay: started 1, returned:Ok(())
7403 ----------------------------------------------------------------
7404 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7405 States: Active Offline Active
7406 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7407 2510 Acked Write 5 Done Sent Done false
7408 2511 Acked Read 8 Done Sent Done false
7409 2512 Acked Write 8 Done Sent Done false
7410 2513 Acked Flush 0 Done Sent Done false
7411 2514 Acked Write 9 Done Sent Done false
7412 2515 Acked Read 4 Done Sent Done false
7413 2516 Acked Write 1 Done Sent Done false
7414 2517 Acked Read 3 Done Sent Done false
7415 2518 Acked Write 8 Done Sent Done false
7416 2519 Acked Read 6 Done Sent Done false
7417 2520 Acked Write 8 Done Sent Done false
7418 2521 Acked Write 8 Done Sent Done false
7419 2522 Acked Read 8 Done Sent Done false
7420 2523 Acked Write 7 Done Sent Done false
7421 2524 Acked Read 3 Done Sent Done false
7422 2525 Acked Write 4 Done Sent Done false
7423 2526 Acked Read 9 Done Sent Done false
7424 2527 Acked Read 9 Done Sent Done false
7425 2528 Acked Write 10 Done Sent Done false
7426 2529 Acked Write 9 Done Sent Done false
7427 2530 Acked Read 7 Done Sent Done false
7428 2531 Acked Read 2 Done Sent Done false
7429 2532 Acked Flush 0 Done Sent Done false
7430 2533 Acked Write 3 Done Sent Done false
7431 2534 Acked Read 6 Done Sent Done false
7432 2535 Acked Read 7 Done Sent Done false
7433 2536 Acked Write 7 Done Sent Done false
7434 2537 Acked Read 10 Done Sent Done false
7435 2538 Acked Write 5 Done Sent Done false
7436 2539 Acked Flush 0 Done Sent Done false
7437 2540 Acked Read 9 Done Sent Done false
7438 2541 Acked Flush 0 Done Sent Done false
7439 2542 Acked Write 2 Done Sent Done false
7440 2543 Acked Read 7 Done Sent Done false
7441 2544 Acked Read 5 Done Sent Done false
7442 2545 Acked Flush 0 Done Sent Done false
7443 2546 Acked Read 4 Done Sent Done false
7444 2547 Acked Flush 0 Done Sent Done false
7445 2548 Acked Write 8 Done Sent Done false
7446 2549 Acked Write 7 Done Sent Done false
7447 2550 Acked Read 1 Done Sent Done false
7448 2551 Acked Write 1 Done Sent Done false
7449 2552 Acked Read 1 Done Sent Done false
7450 2553 Acked Write 4 Done Sent Done false
7451 2554 Acked Read 3 Done Sent Done false
7452 2555 Acked Write 6 Done Sent Done false
7453 2556 Acked Read 7 Done Sent Done false
7454 2557 Acked Read 4 Done Sent Done false
7455 2558 Acked Write 6 Done Sent Done false
7456 2559 Acked Write 2 Done Sent Done false
7457 2560 Acked Write 6 Done Sent Done false
7458 2561 Acked Read 9 Done Sent Done false
7459 2562 Acked Write 2 Done Sent Done false
7460 2563 Acked Read 9 Done Sent Done false
7461 2564 Acked Write 9 Done Sent Done false
7462 2565 Acked Read 2 Done Sent Done false
7463 2566 Acked Write 5 Done Sent Done false
7464 2567 Acked Write 5 Done Sent Done false
7465 2568 Acked Read 10 Done Sent Done false
7466 2569 Acked Write 3 Done Sent Done false
7467 2570 Acked Write 10 Done Sent Done false
7468 2571 Acked Read 8 Done Sent Done false
7469 2572 Acked Read 7 Done Sent Done false
7470 2573 Acked Read 5 Done Sent Done false
7471 2574 Acked Flush 0 Done Sent Done false
7472 2575 Acked Read 7 Done Sent Done false
7473 2576 Acked Write 9 Done Sent Done false
7474 2577 Acked Read 8 Done Sent Done false
7475 2578 Acked Read 1 Done Sent Done false
7476 2579 Acked Read 7 Done Sent Done false
7477 2580 Acked Write 9 Done Sent Done false
7478 2581 Acked Write 7 Done Sent Done false
7479 2582 Acked Write 10 Done Sent Done false
7480 2583 Acked Write 2 Done Sent Done false
7481 2584 Acked Read 2 Done Sent Done false
7482 2585 Acked Write 5 Done Sent Done false
7483 2586 Acked Write 3 Done Sent Done false
7484 2587 Acked Read 7 Done Sent Done false
7485 2588 Acked Read 6 Done Sent Done false
7486 2589 Acked Read 10 Done Sent Done false
7487 2590 Acked Read 3 Done Sent Done false
7488 2591 Acked Write 10 Done Sent Done false
7489 2592 Acked Write 3 Done Sent Done false
7490 2593 Acked Read 3 Done Sent Done false
7491 2594 Acked Write 1 Done Sent Done false
7492 2595 Acked Flush 0 Done Sent Done false
7493 2596 Acked Write 3 Done Sent Done false
7494 2597 Acked Write 3 Done Sent Done false
7495 2598 Acked Read 2 Done Sent Done false
7496 2599 Acked Read 6 Done Sent Done false
7497 2600 Acked Read 8 Done Sent Done false
7498 2601 Acked Flush 0 Done Sent Done false
7499 2602 Acked Write 8 Done Sent Done false
7500 2603 Acked Read 8 Done Sent Done false
7501 2604 Acked Read 4 Done Sent Done false
7502 2605 Acked Flush 0 Done Sent Done false
7503 2606 Acked Read 7 Done Sent Done false
7504 2607 Acked Write 5 Done Sent Done false
7505 2608 Acked Write 8 Done Sent Done false
7506 2609 Acked Write 3 Done Sent Done false
7507 2610 Acked Read 10 Done Sent Done false
7508 2611 Acked Read 2 Done Sent Done false
7509 2612 Acked Read 2 Done Sent Done false
7510 2613 Acked Write 7 Done Sent Done false
7511 2614 Acked Read 7 Done Sent Done false
7512 2615 Acked Flush 0 Done Sent Done false
7513 2616 Acked Read 8 Done Sent Done false
7514 2617 Acked Write 4 Done Sent Done false
7515 2618 Acked Write 5 Done Sent Done false
7516 2619 Acked Write 4 Done Sent Done false
7517 2620 Acked Read 9 Done Sent Done false
7518 2621 Acked Read 3 Done Sent Done false
7519 2622 Acked Read 9 Done Sent Done false
7520 2623 Acked Write 1 Done Sent Done false
7521 2624 Acked Write 1 Done Sent Done false
7522 2625 Acked Read 3 Done Sent Done false
7523 2626 Acked Write 5 Done Sent Done false
7524 2627 Acked Write 10 Done Sent Done false
7525 2628 Acked Read 4 Done Sent Done false
7526 2629 Acked Write 2 Done Sent Done false
7527 2630 Acked Write 5 Done Sent Done false
7528 2631 Acked Read 8 Done Sent Done false
7529 2632 Acked Flush 0 Done Sent Done false
7530 2633 Acked Write 1 Done Sent Done false
7531 2634 Acked Read 9 Done Sent Done false
7532 2635 Acked Read 5 Done Sent Done false
7533 2636 Acked Flush 0 Done Sent Done false
7534 2637 Acked Read 7 Done Sent Done false
7535 2638 Acked Write 1 Done Sent Done false
7536 2639 Acked Read 6 Done Sent Done false
7537 2640 Acked Read 4 Done Sent Done false
7538 2641 Acked Read 3 Done Sent Done false
7539 2642 Acked Flush 0 Done Sent Done false
7540 2643 Acked Write 2 Done Sent Done false
7541 2644 Acked Write 2 Done Sent Done false
7542 2645 Acked Read 5 Done Sent Done false
7543 2646 Acked Read 1 Done Sent Done false
7544 2647 Acked Read 8 Done Sent Done false
7545 2648 Acked Flush 0 Done Sent Done false
7546 2649 Acked Read 3 Done Sent Done false
7547 2650 Acked Write 3 Done Sent Done false
7548 2651 Acked Write 3 Done Sent Done false
7549 2652 Acked Read 4 Done Sent Done false
7550 2653 Acked Write 9 Done Sent Done false
7551 2654 Acked Write 4 Done Sent Done false
7552 2655 Acked Write 1 Done Sent Done false
7553 2656 Acked Read 8 Done Sent Done false
7554 2657 Acked Write 4 Done Sent Done false
7555 2658 Acked Write 8 Done Sent Done false
7556 2659 Acked Write 6 Done Sent Done false
7557 2660 Acked Write 7 Done Sent Done false
7558 2661 Acked Write 8 Done Sent Done false
7559 2662 Acked Write 6 Done Sent Done false
7560 2663 Acked Write 9 Done Sent Done false
7561 2664 Acked Read 9 Done Sent Done false
7562 2665 Acked Read 1 Done Sent Done false
7563 2666 Acked Read 9 Done Sent Done false
7564 2667 Acked Read 2 Done Sent Done false
7565 2668 Acked Read 2 Done Sent Done false
7566 2669 Acked Write 9 Done Sent Done false
7567 2670 Acked Write 10 Done Sent Done false
7568 2671 Acked Read 3 Done Sent Done false
7569 2672 Acked Write 5 Done Sent Done false
7570 2673 Acked Write 5 Done Sent Done false
7571 2674 Acked Write 8 Done Sent Done false
7572 2675 Acked Read 9 Done Sent Done false
7573 2676 Acked Read 8 Done Sent Done false
7574 2677 Acked Flush 0 Done Sent Done false
7575 2678 Acked Write 9 Done Sent Done false
7576 2679 Acked Read 10 Done Sent Done false
7577 2680 Acked Read 4 Done Sent Done false
7578 2681 Acked Flush 0 Done Sent Done false
7579 2682 Acked Write 3 Done Sent Done false
7580 2683 Acked Write 2 Done Sent Done false
7581 2684 Acked Write 7 Done Sent Done false
7582 2685 Acked Read 7 Done Sent Done false
7583 2686 Acked Flush 0 Done Sent Done false
7584 2687 Acked Write 8 Done Sent Done false
7585 2688 Acked Flush 0 Done Sent Done false
7586 2689 Acked Write 10 Done Sent Done false
7587 2690 Acked Write 2 Done Sent Done false
7588 2691 Acked Write 3 Done Sent Done false
7589 2692 Acked Read 5 Done Sent Done false
7590 2693 Acked Flush 0 Done Sent Done false
7591 2694 Acked Write 1 Done Sent Done false
7592 2695 Acked Write 10 Done Sent Done false
7593 2696 Acked Write 2 Done Sent Done false
7594 2697 Acked Flush 0 Done Sent Done false
7595 2698 Acked Read 4 Done Sent Done false
7596 2699 Acked Read 2 Done Sent Done false
7597 2700 Acked Read 1 Done Sent Done false
7598 2701 Acked Read 9 Done Sent Done false
7599 2702 Acked Flush 0 Done Sent Done false
7600 2703 Acked Read 5 Done Sent Done false
7601 2704 Acked Flush 0 Done Sent Done false
7602 2705 Acked Write 4 Done Sent Done false
7603 2706 Acked Write 9 Done Sent Done false
7604 2707 Acked Read 8 Done Sent Done false
7605 2708 Acked Read 5 Done Sent Done false
7606 2709 Acked Read 6 Done Sent Done false
7607 2710 Acked Flush 0 Done Sent Done false
7608 2711 Acked Read 9 Done Sent Done false
7609 2712 Acked Read 3 Done Sent Done false
7610 2713 Acked Read 2 Done Sent Done false
7611 2714 Acked Write 7 Done Sent Done false
7612 2715 Acked Read 3 Done Sent Done false
7613 2716 Acked Read 1 Done Sent Done false
7614 2717 Acked Write 7 Done Sent Done false
7615 2718 Acked Read 6 Done Sent Done false
7616 2719 Acked Write 3 Done Sent Done false
7617 2720 Acked Read 9 Done Sent Done false
7618 2721 Acked Read 3 Done Sent Done false
7619 2722 Acked Read 6 Done Sent Done false
7620 2723 Acked Flush 0 Done Sent Done false
7621 2724 Acked Read 6 Done Sent Done false
7622 2725 Acked Write 10 Done Sent Done false
7623 2726 Acked Read 6 Done Sent Done false
7624 2727 Acked Read 7 Done Sent Done false
7625 2728 Acked Read 3 Done Sent Done false
7626 2729 Acked Write 9 Done Sent Done false
7627 2730 Acked Write 2 Done Sent Done false
7628 2731 Acked Read 6 Done Sent Done false
7629 2732 Acked Read 10 Done Sent Done false
7630 2733 Acked Flush 0 Done Sent Done false
7631 2734 Acked Read 8 Done Sent Done false
7632 2735 Acked Write 4 Done Sent Done false
7633 2736 Acked Write 8 Done Sent Done false
7634 2737 Acked Read 2 Done Sent Done false
7635 2738 Acked Read 9 Done Sent Done false
7636 2739 Acked Write 1 Done Sent Done false
7637 2740 Acked Read 8 Done Sent Done false
7638 2741 Acked Read 9 Done Sent Done false
7639 2742 Acked Flush 0 Done Sent Done false
7640 2743 Acked Read 1 Done Sent Done false
7641 2744 Acked Read 4 Done Sent Done false
7642 2745 Acked Write 10 Done Sent Done false
7643 2746 Acked Write 5 Done Sent Done false
7644 2747 Acked Write 9 Done Sent Done false
7645 2748 Acked Write 6 Done Sent Done false
7646 2749 Acked Read 8 Done Sent Done false
7647 2750 Acked Read 3 Done Sent Done false
7648 2751 Acked Write 2 Done Sent Done false
7649 2752 Acked Flush 0 Done Sent Done false
7650 2753 Acked Write 3 Done Sent Done false
7651 2754 Acked Write 6 Done Sent Done false
7652 2755 Acked Write 3 Done Sent Done false
7653 2756 Acked Read 4 Done Sent Done false
7654 2757 Acked Read 5 Done Sent Done false
7655 2758 Acked Read 7 Done Sent Done false
7656 2759 Acked Flush 0 Done Sent Done false
7657 2760 Acked Read 4 Done Sent Done false
7658 2761 Acked Read 6 Done Sent Done false
7659 2762 Acked Read 2 Done Sent Done false
7660 2763 Acked Write 4 Done Sent Done false
7661 2764 Acked Read 2 Done Sent Done false
7662 2765 Acked Read 4 Done Sent Done false
7663 2766 Acked Read 8 Done Sent Done false
7664 2767 Acked Write 3 Done Sent Done false
7665 2768 Acked Write 6 Done Sent Done false
7666 2769 Acked Write 2 Done Sent Done false
7667 2770 Acked Write 2 Done Sent Done false
7668 2771 Acked Read 3 Done Sent Done false
7669 2772 Acked Flush 0 Done Sent Done false
7670 2773 Acked Read 5 Done Sent Done false
7671 2774 Acked Write 1 Done Sent Done false
7672 2775 Acked Flush 0 Done Sent Done false
7673 2776 Acked Flush 0 Done Sent Done false
7674 2777 Acked Write 3 Done Sent Done false
7675 2778 Acked Read 7 Done Sent Done false
7676 2779 Acked Flush 0 Done Sent Done false
7677 2780 Acked Flush 0 Done Sent Done false
7678 2781 Acked Write 2 Done Sent Done false
7679 2782 Acked Read 5 Done Sent Done false
7680 2783 Acked Read 8 Done Sent Done false
7681 2784 Acked Write 1 Done Sent Done false
7682 2785 Acked Write 1 Done Sent Done false
7683 2786 Acked Read 7 Done Sent Done false
7684 2787 Acked Read 4 Done Sent Done false
7685 2788 Acked Read 8 Done Sent Done false
7686 2789 Acked Read 3 Done Sent Done false
7687 2790 Acked Write 3 Done Sent Done false
7688 2791 Acked Write 1 Done Sent Done false
7689 2792 Acked Read 10 Done Sent Done false
7690 2793 Acked Write 10 Done Sent Done false
7691 2794 Acked Write 7 Done Sent Done false
7692 2795 Acked Write 8 Done Sent Done false
7693 2796 Acked Write 5 Done Sent Done false
7694 2797 Acked Read 2 Done Sent Done false
7695 2798 Acked Read 2 Done Sent Done false
7696 2799 Acked Write 2 Done Sent Done false
7697 2800 Acked Write 6 Done Sent Done false
7698 2801 Acked Read 4 Done Sent Done false
7699 2802 Acked Read 4 Done Sent Done false
7700 2803 Acked Write 3 Done Sent Done false
7701 2804 Acked Read 1 Done Sent Done false
7702 2805 Acked Read 9 Done Sent Done false
7703 2806 Acked Read 3 Done Sent Done false
7704 2807 Acked Read 3 Done Sent Done false
7705 2808 Acked Read 5 Done Sent Done false
7706 2809 Acked Flush 0 Done Sent Done false
7707 2810 Acked Read 2 Done Sent Done false
7708 STATES DS:0 DS:1 DS:2 TOTAL
7709 Sent 0 301 0 301
7710 Done 301 0 301 602
7711 Skipped 0 0 0 0
7712 Error 0 0 0 0
7713 Last Flush: Some(JobId(2809)) Some(JobId(2509)) Some(JobId(2809))
7714 Downstairs last five completed: 2509 2508 2507 2506 2505
7715 Upstairs last five completed: 2810 2809 2808 2807 2806
7716 CLIENT: Up:0 ds:301 act:2
77172025-03-17T22:20:40.112ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
77182025-03-17T22:20:40.718ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
77192025-03-17T22:20:41.720ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
77202025-03-17T22:20:42.721ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
7721 ----------------------------------------------------------------
7722 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7723 States: Active Offline Active
7724 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7725 2510 Acked Write 5 Done Sent Done false
7726 2511 Acked Read 8 Done Sent Done false
7727 2512 Acked Write 8 Done Sent Done false
7728 2513 Acked Flush 0 Done Sent Done false
7729 2514 Acked Write 9 Done Sent Done false
7730 2515 Acked Read 4 Done Sent Done false
7731 2516 Acked Write 1 Done Sent Done false
7732 2517 Acked Read 3 Done Sent Done false
7733 2518 Acked Write 8 Done Sent Done false
7734 2519 Acked Read 6 Done Sent Done false
7735 2520 Acked Write 8 Done Sent Done false
7736 2521 Acked Write 8 Done Sent Done false
7737 2522 Acked Read 8 Done Sent Done false
7738 2523 Acked Write 7 Done Sent Done false
7739 2524 Acked Read 3 Done Sent Done false
7740 2525 Acked Write 4 Done Sent Done false
7741 2526 Acked Read 9 Done Sent Done false
7742 2527 Acked Read 9 Done Sent Done false
7743 2528 Acked Write 10 Done Sent Done false
7744 2529 Acked Write 9 Done Sent Done false
7745 2530 Acked Read 7 Done Sent Done false
7746 2531 Acked Read 2 Done Sent Done false
7747 2532 Acked Flush 0 Done Sent Done false
7748 2533 Acked Write 3 Done Sent Done false
7749 2534 Acked Read 6 Done Sent Done false
7750 2535 Acked Read 7 Done Sent Done false
7751 2536 Acked Write 7 Done Sent Done false
7752 2537 Acked Read 10 Done Sent Done false
7753 2538 Acked Write 5 Done Sent Done false
7754 2539 Acked Flush 0 Done Sent Done false
7755 2540 Acked Read 9 Done Sent Done false
7756 2541 Acked Flush 0 Done Sent Done false
7757 2542 Acked Write 2 Done Sent Done false
7758 2543 Acked Read 7 Done Sent Done false
7759 2544 Acked Read 5 Done Sent Done false
7760 2545 Acked Flush 0 Done Sent Done false
7761 2546 Acked Read 4 Done Sent Done false
7762 2547 Acked Flush 0 Done Sent Done false
7763 2548 Acked Write 8 Done Sent Done false
7764 2549 Acked Write 7 Done Sent Done false
7765 2550 Acked Read 1 Done Sent Done false
7766 2551 Acked Write 1 Done Sent Done false
7767 2552 Acked Read 1 Done Sent Done false
7768 2553 Acked Write 4 Done Sent Done false
7769 2554 Acked Read 3 Done Sent Done false
7770 2555 Acked Write 6 Done Sent Done false
7771 2556 Acked Read 7 Done Sent Done false
7772 2557 Acked Read 4 Done Sent Done false
7773 2558 Acked Write 6 Done Sent Done false
7774 2559 Acked Write 2 Done Sent Done false
7775 2560 Acked Write 6 Done Sent Done false
7776 2561 Acked Read 9 Done Sent Done false
7777 2562 Acked Write 2 Done Sent Done false
7778 2563 Acked Read 9 Done Sent Done false
7779 2564 Acked Write 9 Done Sent Done false
7780 2565 Acked Read 2 Done Sent Done false
7781 2566 Acked Write 5 Done Sent Done false
7782 2567 Acked Write 5 Done Sent Done false
7783 2568 Acked Read 10 Done Sent Done false
7784 2569 Acked Write 3 Done Sent Done false
7785 2570 Acked Write 10 Done Sent Done false
7786 2571 Acked Read 8 Done Sent Done false
7787 2572 Acked Read 7 Done Sent Done false
7788 2573 Acked Read 5 Done Sent Done false
7789 2574 Acked Flush 0 Done Sent Done false
7790 2575 Acked Read 7 Done Sent Done false
7791 2576 Acked Write 9 Done Sent Done false
7792 2577 Acked Read 8 Done Sent Done false
7793 2578 Acked Read 1 Done Sent Done false
7794 2579 Acked Read 7 Done Sent Done false
7795 2580 Acked Write 9 Done Sent Done false
7796 2581 Acked Write 7 Done Sent Done false
7797 2582 Acked Write 10 Done Sent Done false
7798 2583 Acked Write 2 Done Sent Done false
7799 2584 Acked Read 2 Done Sent Done false
7800 2585 Acked Write 5 Done Sent Done false
7801 2586 Acked Write 3 Done Sent Done false
7802 2587 Acked Read 7 Done Sent Done false
7803 2588 Acked Read 6 Done Sent Done false
7804 2589 Acked Read 10 Done Sent Done false
7805 2590 Acked Read 3 Done Sent Done false
7806 2591 Acked Write 10 Done Sent Done false
7807 2592 Acked Write 3 Done Sent Done false
7808 2593 Acked Read 3 Done Sent Done false
7809 2594 Acked Write 1 Done Sent Done false
7810 2595 Acked Flush 0 Done Sent Done false
7811 2596 Acked Write 3 Done Sent Done false
7812 2597 Acked Write 3 Done Sent Done false
7813 2598 Acked Read 2 Done Sent Done false
7814 2599 Acked Read 6 Done Sent Done false
7815 2600 Acked Read 8 Done Sent Done false
7816 2601 Acked Flush 0 Done Sent Done false
7817 2602 Acked Write 8 Done Sent Done false
7818 2603 Acked Read 8 Done Sent Done false
7819 2604 Acked Read 4 Done Sent Done false
7820 2605 Acked Flush 0 Done Sent Done false
7821 2606 Acked Read 7 Done Sent Done false
7822 2607 Acked Write 5 Done Sent Done false
7823 2608 Acked Write 8 Done Sent Done false
7824 2609 Acked Write 3 Done Sent Done false
7825 2610 Acked Read 10 Done Sent Done false
7826 2611 Acked Read 2 Done Sent Done false
7827 2612 Acked Read 2 Done Sent Done false
7828 2613 Acked Write 7 Done Sent Done false
7829 2614 Acked Read 7 Done Sent Done false
7830 2615 Acked Flush 0 Done Sent Done false
7831 2616 Acked Read 8 Done Sent Done false
7832 2617 Acked Write 4 Done Sent Done false
7833 2618 Acked Write 5 Done Sent Done false
7834 2619 Acked Write 4 Done Sent Done false
7835 2620 Acked Read 9 Done Sent Done false
7836 2621 Acked Read 3 Done Sent Done false
7837 2622 Acked Read 9 Done Sent Done false
7838 2623 Acked Write 1 Done Sent Done false
7839 2624 Acked Write 1 Done Sent Done false
7840 2625 Acked Read 3 Done Sent Done false
7841 2626 Acked Write 5 Done Sent Done false
7842 2627 Acked Write 10 Done Sent Done false
7843 2628 Acked Read 4 Done Sent Done false
7844 2629 Acked Write 2 Done Sent Done false
7845 2630 Acked Write 5 Done Sent Done false
7846 2631 Acked Read 8 Done Sent Done false
7847 2632 Acked Flush 0 Done Sent Done false
7848 2633 Acked Write 1 Done Sent Done false
7849 2634 Acked Read 9 Done Sent Done false
7850 2635 Acked Read 5 Done Sent Done false
7851 2636 Acked Flush 0 Done Sent Done false
7852 2637 Acked Read 7 Done Sent Done false
7853 2638 Acked Write 1 Done Sent Done false
7854 2639 Acked Read 6 Done Sent Done false
7855 2640 Acked Read 4 Done Sent Done false
7856 2641 Acked Read 3 Done Sent Done false
7857 2642 Acked Flush 0 Done Sent Done false
7858 2643 Acked Write 2 Done Sent Done false
7859 2644 Acked Write 2 Done Sent Done false
7860 2645 Acked Read 5 Done Sent Done false
7861 2646 Acked Read 1 Done Sent Done false
7862 2647 Acked Read 8 Done Sent Done false
7863 2648 Acked Flush 0 Done Sent Done false
7864 2649 Acked Read 3 Done Sent Done false
7865 2650 Acked Write 3 Done Sent Done false
7866 2651 Acked Write 3 Done Sent Done false
7867 2652 Acked Read 4 Done Sent Done false
7868 2653 Acked Write 9 Done Sent Done false
7869 2654 Acked Write 4 Done Sent Done false
7870 2655 Acked Write 1 Done Sent Done false
7871 2656 Acked Read 8 Done Sent Done false
7872 2657 Acked Write 4 Done Sent Done false
7873 2658 Acked Write 8 Done Sent Done false
7874 2659 Acked Write 6 Done Sent Done false
7875 2660 Acked Write 7 Done Sent Done false
7876 2661 Acked Write 8 Done Sent Done false
7877 2662 Acked Write 6 Done Sent Done false
7878 2663 Acked Write 9 Done Sent Done false
7879 2664 Acked Read 9 Done Sent Done false
7880 2665 Acked Read 1 Done Sent Done false
7881 2666 Acked Read 9 Done Sent Done false
7882 2667 Acked Read 2 Done Sent Done false
7883 2668 Acked Read 2 Done Sent Done false
7884 2669 Acked Write 9 Done Sent Done false
7885 2670 Acked Write 10 Done Sent Done false
7886 2671 Acked Read 3 Done Sent Done false
7887 2672 Acked Write 5 Done Sent Done false
7888 2673 Acked Write 5 Done Sent Done false
7889 2674 Acked Write 8 Done Sent Done false
7890 2675 Acked Read 9 Done Sent Done false
7891 2676 Acked Read 8 Done Sent Done false
7892 2677 Acked Flush 0 Done Sent Done false
7893 2678 Acked Write 9 Done Sent Done false
7894 2679 Acked Read 10 Done Sent Done false
7895 2680 Acked Read 4 Done Sent Done false
7896 2681 Acked Flush 0 Done Sent Done false
7897 2682 Acked Write 3 Done Sent Done false
7898 2683 Acked Write 2 Done Sent Done false
7899 2684 Acked Write 7 Done Sent Done false
7900 2685 Acked Read 7 Done Sent Done false
7901 2686 Acked Flush 0 Done Sent Done false
7902 2687 Acked Write 8 Done Sent Done false
7903 2688 Acked Flush 0 Done Sent Done false
7904 2689 Acked Write 10 Done Sent Done false
7905 2690 Acked Write 2 Done Sent Done false
7906 2691 Acked Write 3 Done Sent Done false
7907 2692 Acked Read 5 Done Sent Done false
7908 2693 Acked Flush 0 Done Sent Done false
7909 2694 Acked Write 1 Done Sent Done false
7910 2695 Acked Write 10 Done Sent Done false
7911 2696 Acked Write 2 Done Sent Done false
7912 2697 Acked Flush 0 Done Sent Done false
7913 2698 Acked Read 4 Done Sent Done false
7914 2699 Acked Read 2 Done Sent Done false
7915 2700 Acked Read 1 Done Sent Done false
7916 2701 Acked Read 9 Done Sent Done false
7917 2702 Acked Flush 0 Done Sent Done false
7918 2703 Acked Read 5 Done Sent Done false
7919 2704 Acked Flush 0 Done Sent Done false
7920 2705 Acked Write 4 Done Sent Done false
7921 2706 Acked Write 9 Done Sent Done false
7922 2707 Acked Read 8 Done Sent Done false
7923 2708 Acked Read 5 Done Sent Done false
7924 2709 Acked Read 6 Done Sent Done false
7925 2710 Acked Flush 0 Done Sent Done false
7926 2711 Acked Read 9 Done Sent Done false
7927 2712 Acked Read 3 Done Sent Done false
7928 2713 Acked Read 2 Done Sent Done false
7929 2714 Acked Write 7 Done Sent Done false
7930 2715 Acked Read 3 Done Sent Done false
7931 2716 Acked Read 1 Done Sent Done false
7932 2717 Acked Write 7 Done Sent Done false
7933 2718 Acked Read 6 Done Sent Done false
7934 2719 Acked Write 3 Done Sent Done false
7935 2720 Acked Read 9 Done Sent Done false
7936 2721 Acked Read 3 Done Sent Done false
7937 2722 Acked Read 6 Done Sent Done false
7938 2723 Acked Flush 0 Done Sent Done false
7939 2724 Acked Read 6 Done Sent Done false
7940 2725 Acked Write 10 Done Sent Done false
7941 2726 Acked Read 6 Done Sent Done false
7942 2727 Acked Read 7 Done Sent Done false
7943 2728 Acked Read 3 Done Sent Done false
7944 2729 Acked Write 9 Done Sent Done false
7945 2730 Acked Write 2 Done Sent Done false
7946 2731 Acked Read 6 Done Sent Done false
7947 2732 Acked Read 10 Done Sent Done false
7948 2733 Acked Flush 0 Done Sent Done false
7949 2734 Acked Read 8 Done Sent Done false
7950 2735 Acked Write 4 Done Sent Done false
7951 2736 Acked Write 8 Done Sent Done false
7952 2737 Acked Read 2 Done Sent Done false
7953 2738 Acked Read 9 Done Sent Done false
7954 2739 Acked Write 1 Done Sent Done false
7955 2740 Acked Read 8 Done Sent Done false
7956 2741 Acked Read 9 Done Sent Done false
7957 2742 Acked Flush 0 Done Sent Done false
7958 2743 Acked Read 1 Done Sent Done false
7959 2744 Acked Read 4 Done Sent Done false
7960 2745 Acked Write 10 Done Sent Done false
7961 2746 Acked Write 5 Done Sent Done false
7962 2747 Acked Write 9 Done Sent Done false
7963 2748 Acked Write 6 Done Sent Done false
7964 2749 Acked Read 8 Done Sent Done false
7965 2750 Acked Read 3 Done Sent Done false
7966 2751 Acked Write 2 Done Sent Done false
7967 2752 Acked Flush 0 Done Sent Done false
7968 2753 Acked Write 3 Done Sent Done false
7969 2754 Acked Write 6 Done Sent Done false
7970 2755 Acked Write 3 Done Sent Done false
7971 2756 Acked Read 4 Done Sent Done false
7972 2757 Acked Read 5 Done Sent Done false
7973 2758 Acked Read 7 Done Sent Done false
7974 2759 Acked Flush 0 Done Sent Done false
7975 2760 Acked Read 4 Done Sent Done false
7976 2761 Acked Read 6 Done Sent Done false
7977 2762 Acked Read 2 Done Sent Done false
7978 2763 Acked Write 4 Done Sent Done false
7979 2764 Acked Read 2 Done Sent Done false
7980 2765 Acked Read 4 Done Sent Done false
7981 2766 Acked Read 8 Done Sent Done false
7982 2767 Acked Write 3 Done Sent Done false
7983 2768 Acked Write 6 Done Sent Done false
7984 2769 Acked Write 2 Done Sent Done false
7985 2770 Acked Write 2 Done Sent Done false
7986 2771 Acked Read 3 Done Sent Done false
7987 2772 Acked Flush 0 Done Sent Done false
7988 2773 Acked Read 5 Done Sent Done false
7989 2774 Acked Write 1 Done Sent Done false
7990 2775 Acked Flush 0 Done Sent Done false
7991 2776 Acked Flush 0 Done Sent Done false
7992 2777 Acked Write 3 Done Sent Done false
7993 2778 Acked Read 7 Done Sent Done false
7994 2779 Acked Flush 0 Done Sent Done false
7995 2780 Acked Flush 0 Done Sent Done false
7996 2781 Acked Write 2 Done Sent Done false
7997 2782 Acked Read 5 Done Sent Done false
7998 2783 Acked Read 8 Done Sent Done false
7999 2784 Acked Write 1 Done Sent Done false
8000 2785 Acked Write 1 Done Sent Done false
8001 2786 Acked Read 7 Done Sent Done false
8002 2787 Acked Read 4 Done Sent Done false
8003 2788 Acked Read 8 Done Sent Done false
8004 2789 Acked Read 3 Done Sent Done false
8005 2790 Acked Write 3 Done Sent Done false
8006 2791 Acked Write 1 Done Sent Done false
8007 2792 Acked Read 10 Done Sent Done false
8008 2793 Acked Write 10 Done Sent Done false
8009 2794 Acked Write 7 Done Sent Done false
8010 2795 Acked Write 8 Done Sent Done false
8011 2796 Acked Write 5 Done Sent Done false
8012 2797 Acked Read 2 Done Sent Done false
8013 2798 Acked Read 2 Done Sent Done false
8014 2799 Acked Write 2 Done Sent Done false
8015 2800 Acked Write 6 Done Sent Done false
8016 2801 Acked Read 4 Done Sent Done false
8017 2802 Acked Read 4 Done Sent Done false
8018 2803 Acked Write 3 Done Sent Done false
8019 2804 Acked Read 1 Done Sent Done false
8020 2805 Acked Read 9 Done Sent Done false
8021 2806 Acked Read 3 Done Sent Done false
8022 2807 Acked Read 3 Done Sent Done false
8023 2808 Acked Read 5 Done Sent Done false
8024 2809 Acked Flush 0 Done Sent Done false
8025 2810 Acked Read 2 Done Sent Done false
8026 STATES DS:0 DS:1 DS:2 TOTAL
8027 Sent 0 301 0 301
8028 Done 301 0 301 602
8029 Skipped 0 0 0 0
8030 Error 0 0 0 0
8031 Last Flush: Some(JobId(2809)) Some(JobId(2509)) Some(JobId(2809))
8032 Downstairs last five completed: 2509 2508 2507 2506 2505
8033 Upstairs last five completed: 2810 2809 2808 2807 2806
8034 CLIENT: Up:0 ds:301 act:2
80352025-03-17T22:20:44.083ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
80362025-03-17T22:20:45.112ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
80372025-03-17T22:20:45.342ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
80382025-03-17T22:20:45.342ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
80392025-03-17T22:20:45.342ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
80402025-03-17T22:20:45.342ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
80412025-03-17T22:20:45.342ZINFOcrucible: client task is exiting = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
8042 [8820] got start action
8043 [8820] I am going to restart
8044 Starting downstairs at port 8820
8045 Make output file at "/tmp/dsc/downstairs-8820.txt"
8046 [8820][1] reports Starting
8047 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1217)
8048 [8820][1] reports Running
80492025-03-17T22:20:45.843ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
80502025-03-17T22:20:46.728ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
8051 ----------------------------------------------------------------
8052 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8053 States: Active Offline Active
8054 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8055 2510 Acked Write 5 Done Sent Done false
8056 2511 Acked Read 8 Done Sent Done false
8057 2512 Acked Write 8 Done Sent Done false
8058 2513 Acked Flush 0 Done Sent Done false
8059 2514 Acked Write 9 Done Sent Done false
8060 2515 Acked Read 4 Done Sent Done false
8061 2516 Acked Write 1 Done Sent Done false
8062 2517 Acked Read 3 Done Sent Done false
8063 2518 Acked Write 8 Done Sent Done false
8064 2519 Acked Read 6 Done Sent Done false
8065 2520 Acked Write 8 Done Sent Done false
8066 2521 Acked Write 8 Done Sent Done false
8067 2522 Acked Read 8 Done Sent Done false
8068 2523 Acked Write 7 Done Sent Done false
8069 2524 Acked Read 3 Done Sent Done false
8070 2525 Acked Write 4 Done Sent Done false
8071 2526 Acked Read 9 Done Sent Done false
8072 2527 Acked Read 9 Done Sent Done false
8073 2528 Acked Write 10 Done Sent Done false
8074 2529 Acked Write 9 Done Sent Done false
8075 2530 Acked Read 7 Done Sent Done false
8076 2531 Acked Read 2 Done Sent Done false
8077 2532 Acked Flush 0 Done Sent Done false
8078 2533 Acked Write 3 Done Sent Done false
8079 2534 Acked Read 6 Done Sent Done false
8080 2535 Acked Read 7 Done Sent Done false
8081 2536 Acked Write 7 Done Sent Done false
8082 2537 Acked Read 10 Done Sent Done false
8083 2538 Acked Write 5 Done Sent Done false
8084 2539 Acked Flush 0 Done Sent Done false
8085 2540 Acked Read 9 Done Sent Done false
8086 2541 Acked Flush 0 Done Sent Done false
8087 2542 Acked Write 2 Done Sent Done false
8088 2543 Acked Read 7 Done Sent Done false
8089 2544 Acked Read 5 Done Sent Done false
8090 2545 Acked Flush 0 Done Sent Done false
8091 2546 Acked Read 4 Done Sent Done false
8092 2547 Acked Flush 0 Done Sent Done false
8093 2548 Acked Write 8 Done Sent Done false
8094 2549 Acked Write 7 Done Sent Done false
8095 2550 Acked Read 1 Done Sent Done false
8096 2551 Acked Write 1 Done Sent Done false
8097 2552 Acked Read 1 Done Sent Done false
8098 2553 Acked Write 4 Done Sent Done false
8099 2554 Acked Read 3 Done Sent Done false
8100 2555 Acked Write 6 Done Sent Done false
8101 2556 Acked Read 7 Done Sent Done false
8102 2557 Acked Read 4 Done Sent Done false
8103 2558 Acked Write 6 Done Sent Done false
8104 2559 Acked Write 2 Done Sent Done false
8105 2560 Acked Write 6 Done Sent Done false
8106 2561 Acked Read 9 Done Sent Done false
8107 2562 Acked Write 2 Done Sent Done false
8108 2563 Acked Read 9 Done Sent Done false
8109 2564 Acked Write 9 Done Sent Done false
8110 2565 Acked Read 2 Done Sent Done false
8111 2566 Acked Write 5 Done Sent Done false
8112 2567 Acked Write 5 Done Sent Done false
8113 2568 Acked Read 10 Done Sent Done false
8114 2569 Acked Write 3 Done Sent Done false
8115 2570 Acked Write 10 Done Sent Done false
8116 2571 Acked Read 8 Done Sent Done false
8117 2572 Acked Read 7 Done Sent Done false
8118 2573 Acked Read 5 Done Sent Done false
8119 2574 Acked Flush 0 Done Sent Done false
8120 2575 Acked Read 7 Done Sent Done false
8121 2576 Acked Write 9 Done Sent Done false
8122 2577 Acked Read 8 Done Sent Done false
8123 2578 Acked Read 1 Done Sent Done false
8124 2579 Acked Read 7 Done Sent Done false
8125 2580 Acked Write 9 Done Sent Done false
8126 2581 Acked Write 7 Done Sent Done false
8127 2582 Acked Write 10 Done Sent Done false
8128 2583 Acked Write 2 Done Sent Done false
8129 2584 Acked Read 2 Done Sent Done false
8130 2585 Acked Write 5 Done Sent Done false
8131 2586 Acked Write 3 Done Sent Done false
8132 2587 Acked Read 7 Done Sent Done false
8133 2588 Acked Read 6 Done Sent Done false
8134 2589 Acked Read 10 Done Sent Done false
8135 2590 Acked Read 3 Done Sent Done false
8136 2591 Acked Write 10 Done Sent Done false
8137 2592 Acked Write 3 Done Sent Done false
8138 2593 Acked Read 3 Done Sent Done false
8139 2594 Acked Write 1 Done Sent Done false
8140 2595 Acked Flush 0 Done Sent Done false
8141 2596 Acked Write 3 Done Sent Done false
8142 2597 Acked Write 3 Done Sent Done false
8143 2598 Acked Read 2 Done Sent Done false
8144 2599 Acked Read 6 Done Sent Done false
8145 2600 Acked Read 8 Done Sent Done false
8146 2601 Acked Flush 0 Done Sent Done false
8147 2602 Acked Write 8 Done Sent Done false
8148 2603 Acked Read 8 Done Sent Done false
8149 2604 Acked Read 4 Done Sent Done false
8150 2605 Acked Flush 0 Done Sent Done false
8151 2606 Acked Read 7 Done Sent Done false
8152 2607 Acked Write 5 Done Sent Done false
8153 2608 Acked Write 8 Done Sent Done false
8154 2609 Acked Write 3 Done Sent Done false
8155 2610 Acked Read 10 Done Sent Done false
8156 2611 Acked Read 2 Done Sent Done false
8157 2612 Acked Read 2 Done Sent Done false
8158 2613 Acked Write 7 Done Sent Done false
8159 2614 Acked Read 7 Done Sent Done false
8160 2615 Acked Flush 0 Done Sent Done false
8161 2616 Acked Read 8 Done Sent Done false
8162 2617 Acked Write 4 Done Sent Done false
8163 2618 Acked Write 5 Done Sent Done false
8164 2619 Acked Write 4 Done Sent Done false
8165 2620 Acked Read 9 Done Sent Done false
8166 2621 Acked Read 3 Done Sent Done false
8167 2622 Acked Read 9 Done Sent Done false
8168 2623 Acked Write 1 Done Sent Done false
8169 2624 Acked Write 1 Done Sent Done false
8170 2625 Acked Read 3 Done Sent Done false
8171 2626 Acked Write 5 Done Sent Done false
8172 2627 Acked Write 10 Done Sent Done false
8173 2628 Acked Read 4 Done Sent Done false
8174 2629 Acked Write 2 Done Sent Done false
8175 2630 Acked Write 5 Done Sent Done false
8176 2631 Acked Read 8 Done Sent Done false
8177 2632 Acked Flush 0 Done Sent Done false
8178 2633 Acked Write 1 Done Sent Done false
8179 2634 Acked Read 9 Done Sent Done false
8180 2635 Acked Read 5 Done Sent Done false
8181 2636 Acked Flush 0 Done Sent Done false
8182 2637 Acked Read 7 Done Sent Done false
8183 2638 Acked Write 1 Done Sent Done false
8184 2639 Acked Read 6 Done Sent Done false
8185 2640 Acked Read 4 Done Sent Done false
8186 2641 Acked Read 3 Done Sent Done false
8187 2642 Acked Flush 0 Done Sent Done false
8188 2643 Acked Write 2 Done Sent Done false
8189 2644 Acked Write 2 Done Sent Done false
8190 2645 Acked Read 5 Done Sent Done false
8191 2646 Acked Read 1 Done Sent Done false
8192 2647 Acked Read 8 Done Sent Done false
8193 2648 Acked Flush 0 Done Sent Done false
8194 2649 Acked Read 3 Done Sent Done false
8195 2650 Acked Write 3 Done Sent Done false
8196 2651 Acked Write 3 Done Sent Done false
8197 2652 Acked Read 4 Done Sent Done false
8198 2653 Acked Write 9 Done Sent Done false
8199 2654 Acked Write 4 Done Sent Done false
8200 2655 Acked Write 1 Done Sent Done false
8201 2656 Acked Read 8 Done Sent Done false
8202 2657 Acked Write 4 Done Sent Done false
8203 2658 Acked Write 8 Done Sent Done false
8204 2659 Acked Write 6 Done Sent Done false
8205 2660 Acked Write 7 Done Sent Done false
8206 2661 Acked Write 8 Done Sent Done false
8207 2662 Acked Write 6 Done Sent Done false
8208 2663 Acked Write 9 Done Sent Done false
8209 2664 Acked Read 9 Done Sent Done false
8210 2665 Acked Read 1 Done Sent Done false
8211 2666 Acked Read 9 Done Sent Done false
8212 2667 Acked Read 2 Done Sent Done false
8213 2668 Acked Read 2 Done Sent Done false
8214 2669 Acked Write 9 Done Sent Done false
8215 2670 Acked Write 10 Done Sent Done false
8216 2671 Acked Read 3 Done Sent Done false
8217 2672 Acked Write 5 Done Sent Done false
8218 2673 Acked Write 5 Done Sent Done false
8219 2674 Acked Write 8 Done Sent Done false
8220 2675 Acked Read 9 Done Sent Done false
8221 2676 Acked Read 8 Done Sent Done false
8222 2677 Acked Flush 0 Done Sent Done false
8223 2678 Acked Write 9 Done Sent Done false
8224 2679 Acked Read 10 Done Sent Done false
8225 2680 Acked Read 4 Done Sent Done false
8226 2681 Acked Flush 0 Done Sent Done false
8227 2682 Acked Write 3 Done Sent Done false
8228 2683 Acked Write 2 Done Sent Done false
8229 2684 Acked Write 7 Done Sent Done false
8230 2685 Acked Read 7 Done Sent Done false
8231 2686 Acked Flush 0 Done Sent Done false
8232 2687 Acked Write 8 Done Sent Done false
8233 2688 Acked Flush 0 Done Sent Done false
8234 2689 Acked Write 10 Done Sent Done false
8235 2690 Acked Write 2 Done Sent Done false
8236 2691 Acked Write 3 Done Sent Done false
8237 2692 Acked Read 5 Done Sent Done false
8238 2693 Acked Flush 0 Done Sent Done false
8239 2694 Acked Write 1 Done Sent Done false
8240 2695 Acked Write 10 Done Sent Done false
8241 2696 Acked Write 2 Done Sent Done false
8242 2697 Acked Flush 0 Done Sent Done false
8243 2698 Acked Read 4 Done Sent Done false
8244 2699 Acked Read 2 Done Sent Done false
8245 2700 Acked Read 1 Done Sent Done false
8246 2701 Acked Read 9 Done Sent Done false
8247 2702 Acked Flush 0 Done Sent Done false
8248 2703 Acked Read 5 Done Sent Done false
8249 2704 Acked Flush 0 Done Sent Done false
8250 2705 Acked Write 4 Done Sent Done false
8251 2706 Acked Write 9 Done Sent Done false
8252 2707 Acked Read 8 Done Sent Done false
8253 2708 Acked Read 5 Done Sent Done false
8254 2709 Acked Read 6 Done Sent Done false
8255 2710 Acked Flush 0 Done Sent Done false
8256 2711 Acked Read 9 Done Sent Done false
8257 2712 Acked Read 3 Done Sent Done false
8258 2713 Acked Read 2 Done Sent Done false
8259 2714 Acked Write 7 Done Sent Done false
8260 2715 Acked Read 3 Done Sent Done false
8261 2716 Acked Read 1 Done Sent Done false
8262 2717 Acked Write 7 Done Sent Done false
8263 2718 Acked Read 6 Done Sent Done false
8264 2719 Acked Write 3 Done Sent Done false
8265 2720 Acked Read 9 Done Sent Done false
8266 2721 Acked Read 3 Done Sent Done false
8267 2722 Acked Read 6 Done Sent Done false
8268 2723 Acked Flush 0 Done Sent Done false
8269 2724 Acked Read 6 Done Sent Done false
8270 2725 Acked Write 10 Done Sent Done false
8271 2726 Acked Read 6 Done Sent Done false
8272 2727 Acked Read 7 Done Sent Done false
8273 2728 Acked Read 3 Done Sent Done false
8274 2729 Acked Write 9 Done Sent Done false
8275 2730 Acked Write 2 Done Sent Done false
8276 2731 Acked Read 6 Done Sent Done false
8277 2732 Acked Read 10 Done Sent Done false
8278 2733 Acked Flush 0 Done Sent Done false
8279 2734 Acked Read 8 Done Sent Done false
8280 2735 Acked Write 4 Done Sent Done false
8281 2736 Acked Write 8 Done Sent Done false
8282 2737 Acked Read 2 Done Sent Done false
8283 2738 Acked Read 9 Done Sent Done false
8284 2739 Acked Write 1 Done Sent Done false
8285 2740 Acked Read 8 Done Sent Done false
8286 2741 Acked Read 9 Done Sent Done false
8287 2742 Acked Flush 0 Done Sent Done false
8288 2743 Acked Read 1 Done Sent Done false
8289 2744 Acked Read 4 Done Sent Done false
8290 2745 Acked Write 10 Done Sent Done false
8291 2746 Acked Write 5 Done Sent Done false
8292 2747 Acked Write 9 Done Sent Done false
8293 2748 Acked Write 6 Done Sent Done false
8294 2749 Acked Read 8 Done Sent Done false
8295 2750 Acked Read 3 Done Sent Done false
8296 2751 Acked Write 2 Done Sent Done false
8297 2752 Acked Flush 0 Done Sent Done false
8298 2753 Acked Write 3 Done Sent Done false
8299 2754 Acked Write 6 Done Sent Done false
8300 2755 Acked Write 3 Done Sent Done false
8301 2756 Acked Read 4 Done Sent Done false
8302 2757 Acked Read 5 Done Sent Done false
8303 2758 Acked Read 7 Done Sent Done false
8304 2759 Acked Flush 0 Done Sent Done false
8305 2760 Acked Read 4 Done Sent Done false
8306 2761 Acked Read 6 Done Sent Done false
8307 2762 Acked Read 2 Done Sent Done false
8308 2763 Acked Write 4 Done Sent Done false
8309 2764 Acked Read 2 Done Sent Done false
8310 2765 Acked Read 4 Done Sent Done false
8311 2766 Acked Read 8 Done Sent Done false
8312 2767 Acked Write 3 Done Sent Done false
8313 2768 Acked Write 6 Done Sent Done false
8314 2769 Acked Write 2 Done Sent Done false
8315 2770 Acked Write 2 Done Sent Done false
8316 2771 Acked Read 3 Done Sent Done false
8317 2772 Acked Flush 0 Done Sent Done false
8318 2773 Acked Read 5 Done Sent Done false
8319 2774 Acked Write 1 Done Sent Done false
8320 2775 Acked Flush 0 Done Sent Done false
8321 2776 Acked Flush 0 Done Sent Done false
8322 2777 Acked Write 3 Done Sent Done false
8323 2778 Acked Read 7 Done Sent Done false
8324 2779 Acked Flush 0 Done Sent Done false
8325 2780 Acked Flush 0 Done Sent Done false
8326 2781 Acked Write 2 Done Sent Done false
8327 2782 Acked Read 5 Done Sent Done false
8328 2783 Acked Read 8 Done Sent Done false
8329 2784 Acked Write 1 Done Sent Done false
8330 2785 Acked Write 1 Done Sent Done false
8331 2786 Acked Read 7 Done Sent Done false
8332 2787 Acked Read 4 Done Sent Done false
8333 2788 Acked Read 8 Done Sent Done false
8334 2789 Acked Read 3 Done Sent Done false
8335 2790 Acked Write 3 Done Sent Done false
8336 2791 Acked Write 1 Done Sent Done false
8337 2792 Acked Read 10 Done Sent Done false
8338 2793 Acked Write 10 Done Sent Done false
8339 2794 Acked Write 7 Done Sent Done false
8340 2795 Acked Write 8 Done Sent Done false
8341 2796 Acked Write 5 Done Sent Done false
8342 2797 Acked Read 2 Done Sent Done false
8343 2798 Acked Read 2 Done Sent Done false
8344 2799 Acked Write 2 Done Sent Done false
8345 2800 Acked Write 6 Done Sent Done false
8346 2801 Acked Read 4 Done Sent Done false
8347 2802 Acked Read 4 Done Sent Done false
8348 2803 Acked Write 3 Done Sent Done false
8349 2804 Acked Read 1 Done Sent Done false
8350 2805 Acked Read 9 Done Sent Done false
8351 2806 Acked Read 3 Done Sent Done false
8352 2807 Acked Read 3 Done Sent Done false
8353 2808 Acked Read 5 Done Sent Done false
8354 2809 Acked Flush 0 Done Sent Done false
8355 2810 Acked Read 2 Done Sent Done false
8356 STATES DS:0 DS:1 DS:2 TOTAL
8357 Sent 0 301 0 301
8358 Done 301 0 301 602
8359 Skipped 0 0 0 0
8360 Error 0 0 0 0
8361 Last Flush: Some(JobId(2809)) Some(JobId(2509)) Some(JobId(2809))
8362 Downstairs last five completed: 2509 2508 2507 2506 2505
8363 Upstairs last five completed: 2810 2809 2808 2807 2806
8364 CLIENT: Up:0 ds:301 act:2
83652025-03-17T22:20:48.084ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
83662025-03-17T22:20:48.731ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
83672025-03-17T22:20:49.232ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
83682025-03-17T22:20:50.113ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
83692025-03-17T22:20:50.736ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
8370 ----------------------------------------------------------------
8371 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8372 States: Active Offline Active
8373 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8374 2510 Acked Write 5 Done Sent Done false
8375 2511 Acked Read 8 Done Sent Done false
8376 2512 Acked Write 8 Done Sent Done false
8377 2513 Acked Flush 0 Done Sent Done false
8378 2514 Acked Write 9 Done Sent Done false
8379 2515 Acked Read 4 Done Sent Done false
8380 2516 Acked Write 1 Done Sent Done false
8381 2517 Acked Read 3 Done Sent Done false
8382 2518 Acked Write 8 Done Sent Done false
8383 2519 Acked Read 6 Done Sent Done false
8384 2520 Acked Write 8 Done Sent Done false
8385 2521 Acked Write 8 Done Sent Done false
8386 2522 Acked Read 8 Done Sent Done false
8387 2523 Acked Write 7 Done Sent Done false
8388 2524 Acked Read 3 Done Sent Done false
8389 2525 Acked Write 4 Done Sent Done false
8390 2526 Acked Read 9 Done Sent Done false
8391 2527 Acked Read 9 Done Sent Done false
8392 2528 Acked Write 10 Done Sent Done false
8393 2529 Acked Write 9 Done Sent Done false
8394 2530 Acked Read 7 Done Sent Done false
8395 2531 Acked Read 2 Done Sent Done false
8396 2532 Acked Flush 0 Done Sent Done false
8397 2533 Acked Write 3 Done Sent Done false
8398 2534 Acked Read 6 Done Sent Done false
8399 2535 Acked Read 7 Done Sent Done false
8400 2536 Acked Write 7 Done Sent Done false
8401 2537 Acked Read 10 Done Sent Done false
8402 2538 Acked Write 5 Done Sent Done false
8403 2539 Acked Flush 0 Done Sent Done false
8404 2540 Acked Read 9 Done Sent Done false
8405 2541 Acked Flush 0 Done Sent Done false
8406 2542 Acked Write 2 Done Sent Done false
8407 2543 Acked Read 7 Done Sent Done false
8408 2544 Acked Read 5 Done Sent Done false
8409 2545 Acked Flush 0 Done Sent Done false
8410 2546 Acked Read 4 Done Sent Done false
8411 2547 Acked Flush 0 Done Sent Done false
8412 2548 Acked Write 8 Done Sent Done false
8413 2549 Acked Write 7 Done Sent Done false
8414 2550 Acked Read 1 Done Sent Done false
8415 2551 Acked Write 1 Done Sent Done false
8416 2552 Acked Read 1 Done Sent Done false
8417 2553 Acked Write 4 Done Sent Done false
8418 2554 Acked Read 3 Done Sent Done false
8419 2555 Acked Write 6 Done Sent Done false
8420 2556 Acked Read 7 Done Sent Done false
8421 2557 Acked Read 4 Done Sent Done false
8422 2558 Acked Write 6 Done Sent Done false
8423 2559 Acked Write 2 Done Sent Done false
8424 2560 Acked Write 6 Done Sent Done false
8425 2561 Acked Read 9 Done Sent Done false
8426 2562 Acked Write 2 Done Sent Done false
8427 2563 Acked Read 9 Done Sent Done false
8428 2564 Acked Write 9 Done Sent Done false
8429 2565 Acked Read 2 Done Sent Done false
8430 2566 Acked Write 5 Done Sent Done false
8431 2567 Acked Write 5 Done Sent Done false
8432 2568 Acked Read 10 Done Sent Done false
8433 2569 Acked Write 3 Done Sent Done false
8434 2570 Acked Write 10 Done Sent Done false
8435 2571 Acked Read 8 Done Sent Done false
8436 2572 Acked Read 7 Done Sent Done false
8437 2573 Acked Read 5 Done Sent Done false
8438 2574 Acked Flush 0 Done Sent Done false
8439 2575 Acked Read 7 Done Sent Done false
8440 2576 Acked Write 9 Done Sent Done false
8441 2577 Acked Read 8 Done Sent Done false
8442 2578 Acked Read 1 Done Sent Done false
8443 2579 Acked Read 7 Done Sent Done false
8444 2580 Acked Write 9 Done Sent Done false
8445 2581 Acked Write 7 Done Sent Done false
8446 2582 Acked Write 10 Done Sent Done false
8447 2583 Acked Write 2 Done Sent Done false
8448 2584 Acked Read 2 Done Sent Done false
8449 2585 Acked Write 5 Done Sent Done false
8450 2586 Acked Write 3 Done Sent Done false
8451 2587 Acked Read 7 Done Sent Done false
8452 2588 Acked Read 6 Done Sent Done false
8453 2589 Acked Read 10 Done Sent Done false
8454 2590 Acked Read 3 Done Sent Done false
8455 2591 Acked Write 10 Done Sent Done false
8456 2592 Acked Write 3 Done Sent Done false
8457 2593 Acked Read 3 Done Sent Done false
8458 2594 Acked Write 1 Done Sent Done false
8459 2595 Acked Flush 0 Done Sent Done false
8460 2596 Acked Write 3 Done Sent Done false
8461 2597 Acked Write 3 Done Sent Done false
8462 2598 Acked Read 2 Done Sent Done false
8463 2599 Acked Read 6 Done Sent Done false
8464 2600 Acked Read 8 Done Sent Done false
8465 2601 Acked Flush 0 Done Sent Done false
8466 2602 Acked Write 8 Done Sent Done false
8467 2603 Acked Read 8 Done Sent Done false
8468 2604 Acked Read 4 Done Sent Done false
8469 2605 Acked Flush 0 Done Sent Done false
8470 2606 Acked Read 7 Done Sent Done false
8471 2607 Acked Write 5 Done Sent Done false
8472 2608 Acked Write 8 Done Sent Done false
8473 2609 Acked Write 3 Done Sent Done false
8474 2610 Acked Read 10 Done Sent Done false
8475 2611 Acked Read 2 Done Sent Done false
8476 2612 Acked Read 2 Done Sent Done false
8477 2613 Acked Write 7 Done Sent Done false
8478 2614 Acked Read 7 Done Sent Done false
8479 2615 Acked Flush 0 Done Sent Done false
8480 2616 Acked Read 8 Done Sent Done false
8481 2617 Acked Write 4 Done Sent Done false
8482 2618 Acked Write 5 Done Sent Done false
8483 2619 Acked Write 4 Done Sent Done false
8484 2620 Acked Read 9 Done Sent Done false
8485 2621 Acked Read 3 Done Sent Done false
8486 2622 Acked Read 9 Done Sent Done false
8487 2623 Acked Write 1 Done Sent Done false
8488 2624 Acked Write 1 Done Sent Done false
8489 2625 Acked Read 3 Done Sent Done false
8490 2626 Acked Write 5 Done Sent Done false
8491 2627 Acked Write 10 Done Sent Done false
8492 2628 Acked Read 4 Done Sent Done false
8493 2629 Acked Write 2 Done Sent Done false
8494 2630 Acked Write 5 Done Sent Done false
8495 2631 Acked Read 8 Done Sent Done false
8496 2632 Acked Flush 0 Done Sent Done false
8497 2633 Acked Write 1 Done Sent Done false
8498 2634 Acked Read 9 Done Sent Done false
8499 2635 Acked Read 5 Done Sent Done false
8500 2636 Acked Flush 0 Done Sent Done false
8501 2637 Acked Read 7 Done Sent Done false
8502 2638 Acked Write 1 Done Sent Done false
8503 2639 Acked Read 6 Done Sent Done false
8504 2640 Acked Read 4 Done Sent Done false
8505 2641 Acked Read 3 Done Sent Done false
8506 2642 Acked Flush 0 Done Sent Done false
8507 2643 Acked Write 2 Done Sent Done false
8508 2644 Acked Write 2 Done Sent Done false
8509 2645 Acked Read 5 Done Sent Done false
8510 2646 Acked Read 1 Done Sent Done false
8511 2647 Acked Read 8 Done Sent Done false
8512 2648 Acked Flush 0 Done Sent Done false
8513 2649 Acked Read 3 Done Sent Done false
8514 2650 Acked Write 3 Done Sent Done false
8515 2651 Acked Write 3 Done Sent Done false
8516 2652 Acked Read 4 Done Sent Done false
8517 2653 Acked Write 9 Done Sent Done false
8518 2654 Acked Write 4 Done Sent Done false
8519 2655 Acked Write 1 Done Sent Done false
8520 2656 Acked Read 8 Done Sent Done false
8521 2657 Acked Write 4 Done Sent Done false
8522 2658 Acked Write 8 Done Sent Done false
8523 2659 Acked Write 6 Done Sent Done false
8524 2660 Acked Write 7 Done Sent Done false
8525 2661 Acked Write 8 Done Sent Done false
8526 2662 Acked Write 6 Done Sent Done false
8527 2663 Acked Write 9 Done Sent Done false
8528 2664 Acked Read 9 Done Sent Done false
8529 2665 Acked Read 1 Done Sent Done false
8530 2666 Acked Read 9 Done Sent Done false
8531 2667 Acked Read 2 Done Sent Done false
8532 2668 Acked Read 2 Done Sent Done false
8533 2669 Acked Write 9 Done Sent Done false
8534 2670 Acked Write 10 Done Sent Done false
8535 2671 Acked Read 3 Done Sent Done false
8536 2672 Acked Write 5 Done Sent Done false
8537 2673 Acked Write 5 Done Sent Done false
8538 2674 Acked Write 8 Done Sent Done false
8539 2675 Acked Read 9 Done Sent Done false
8540 2676 Acked Read 8 Done Sent Done false
8541 2677 Acked Flush 0 Done Sent Done false
8542 2678 Acked Write 9 Done Sent Done false
8543 2679 Acked Read 10 Done Sent Done false
8544 2680 Acked Read 4 Done Sent Done false
8545 2681 Acked Flush 0 Done Sent Done false
8546 2682 Acked Write 3 Done Sent Done false
8547 2683 Acked Write 2 Done Sent Done false
8548 2684 Acked Write 7 Done Sent Done false
8549 2685 Acked Read 7 Done Sent Done false
8550 2686 Acked Flush 0 Done Sent Done false
8551 2687 Acked Write 8 Done Sent Done false
8552 2688 Acked Flush 0 Done Sent Done false
8553 2689 Acked Write 10 Done Sent Done false
8554 2690 Acked Write 2 Done Sent Done false
8555 2691 Acked Write 3 Done Sent Done false
8556 2692 Acked Read 5 Done Sent Done false
8557 2693 Acked Flush 0 Done Sent Done false
8558 2694 Acked Write 1 Done Sent Done false
8559 2695 Acked Write 10 Done Sent Done false
8560 2696 Acked Write 2 Done Sent Done false
8561 2697 Acked Flush 0 Done Sent Done false
8562 2698 Acked Read 4 Done Sent Done false
8563 2699 Acked Read 2 Done Sent Done false
8564 2700 Acked Read 1 Done Sent Done false
8565 2701 Acked Read 9 Done Sent Done false
8566 2702 Acked Flush 0 Done Sent Done false
8567 2703 Acked Read 5 Done Sent Done false
8568 2704 Acked Flush 0 Done Sent Done false
8569 2705 Acked Write 4 Done Sent Done false
8570 2706 Acked Write 9 Done Sent Done false
8571 2707 Acked Read 8 Done Sent Done false
8572 2708 Acked Read 5 Done Sent Done false
8573 2709 Acked Read 6 Done Sent Done false
8574 2710 Acked Flush 0 Done Sent Done false
8575 2711 Acked Read 9 Done Sent Done false
8576 2712 Acked Read 3 Done Sent Done false
8577 2713 Acked Read 2 Done Sent Done false
8578 2714 Acked Write 7 Done Sent Done false
8579 2715 Acked Read 3 Done Sent Done false
8580 2716 Acked Read 1 Done Sent Done false
8581 2717 Acked Write 7 Done Sent Done false
8582 2718 Acked Read 6 Done Sent Done false
8583 2719 Acked Write 3 Done Sent Done false
8584 2720 Acked Read 9 Done Sent Done false
8585 2721 Acked Read 3 Done Sent Done false
8586 2722 Acked Read 6 Done Sent Done false
8587 2723 Acked Flush 0 Done Sent Done false
8588 2724 Acked Read 6 Done Sent Done false
8589 2725 Acked Write 10 Done Sent Done false
8590 2726 Acked Read 6 Done Sent Done false
8591 2727 Acked Read 7 Done Sent Done false
8592 2728 Acked Read 3 Done Sent Done false
8593 2729 Acked Write 9 Done Sent Done false
8594 2730 Acked Write 2 Done Sent Done false
8595 2731 Acked Read 6 Done Sent Done false
8596 2732 Acked Read 10 Done Sent Done false
8597 2733 Acked Flush 0 Done Sent Done false
8598 2734 Acked Read 8 Done Sent Done false
8599 2735 Acked Write 4 Done Sent Done false
8600 2736 Acked Write 8 Done Sent Done false
8601 2737 Acked Read 2 Done Sent Done false
8602 2738 Acked Read 9 Done Sent Done false
8603 2739 Acked Write 1 Done Sent Done false
8604 2740 Acked Read 8 Done Sent Done false
8605 2741 Acked Read 9 Done Sent Done false
8606 2742 Acked Flush 0 Done Sent Done false
8607 2743 Acked Read 1 Done Sent Done false
8608 2744 Acked Read 4 Done Sent Done false
8609 2745 Acked Write 10 Done Sent Done false
8610 2746 Acked Write 5 Done Sent Done false
8611 2747 Acked Write 9 Done Sent Done false
8612 2748 Acked Write 6 Done Sent Done false
8613 2749 Acked Read 8 Done Sent Done false
8614 2750 Acked Read 3 Done Sent Done false
8615 2751 Acked Write 2 Done Sent Done false
8616 2752 Acked Flush 0 Done Sent Done false
8617 2753 Acked Write 3 Done Sent Done false
8618 2754 Acked Write 6 Done Sent Done false
8619 2755 Acked Write 3 Done Sent Done false
8620 2756 Acked Read 4 Done Sent Done false
8621 2757 Acked Read 5 Done Sent Done false
8622 2758 Acked Read 7 Done Sent Done false
8623 2759 Acked Flush 0 Done Sent Done false
8624 2760 Acked Read 4 Done Sent Done false
8625 2761 Acked Read 6 Done Sent Done false
8626 2762 Acked Read 2 Done Sent Done false
8627 2763 Acked Write 4 Done Sent Done false
8628 2764 Acked Read 2 Done Sent Done false
8629 2765 Acked Read 4 Done Sent Done false
8630 2766 Acked Read 8 Done Sent Done false
8631 2767 Acked Write 3 Done Sent Done false
8632 2768 Acked Write 6 Done Sent Done false
8633 2769 Acked Write 2 Done Sent Done false
8634 2770 Acked Write 2 Done Sent Done false
8635 2771 Acked Read 3 Done Sent Done false
8636 2772 Acked Flush 0 Done Sent Done false
8637 2773 Acked Read 5 Done Sent Done false
8638 2774 Acked Write 1 Done Sent Done false
8639 2775 Acked Flush 0 Done Sent Done false
8640 2776 Acked Flush 0 Done Sent Done false
8641 2777 Acked Write 3 Done Sent Done false
8642 2778 Acked Read 7 Done Sent Done false
8643 2779 Acked Flush 0 Done Sent Done false
8644 2780 Acked Flush 0 Done Sent Done false
8645 2781 Acked Write 2 Done Sent Done false
8646 2782 Acked Read 5 Done Sent Done false
8647 2783 Acked Read 8 Done Sent Done false
8648 2784 Acked Write 1 Done Sent Done false
8649 2785 Acked Write 1 Done Sent Done false
8650 2786 Acked Read 7 Done Sent Done false
8651 2787 Acked Read 4 Done Sent Done false
8652 2788 Acked Read 8 Done Sent Done false
8653 2789 Acked Read 3 Done Sent Done false
8654 2790 Acked Write 3 Done Sent Done false
8655 2791 Acked Write 1 Done Sent Done false
8656 2792 Acked Read 10 Done Sent Done false
8657 2793 Acked Write 10 Done Sent Done false
8658 2794 Acked Write 7 Done Sent Done false
8659 2795 Acked Write 8 Done Sent Done false
8660 2796 Acked Write 5 Done Sent Done false
8661 2797 Acked Read 2 Done Sent Done false
8662 2798 Acked Read 2 Done Sent Done false
8663 2799 Acked Write 2 Done Sent Done false
8664 2800 Acked Write 6 Done Sent Done false
8665 2801 Acked Read 4 Done Sent Done false
8666 2802 Acked Read 4 Done Sent Done false
8667 2803 Acked Write 3 Done Sent Done false
8668 2804 Acked Read 1 Done Sent Done false
8669 2805 Acked Read 9 Done Sent Done false
8670 2806 Acked Read 3 Done Sent Done false
8671 2807 Acked Read 3 Done Sent Done false
8672 2808 Acked Read 5 Done Sent Done false
8673 2809 Acked Flush 0 Done Sent Done false
8674 2810 Acked Read 2 Done Sent Done false
8675 STATES DS:0 DS:1 DS:2 TOTAL
8676 Sent 0 301 0 301
8677 Done 301 0 301 602
8678 Skipped 0 0 0 0
8679 Error 0 0 0 0
8680 Last Flush: Some(JobId(2809)) Some(JobId(2509)) Some(JobId(2809))
8681 Downstairs last five completed: 2509 2508 2507 2506 2505
8682 Upstairs last five completed: 2810 2809 2808 2807 2806
8683 CLIENT: Up:0 ds:301 act:2
86842025-03-17T22:20:52.087ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
86852025-03-17T22:20:52.738ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
86862025-03-17T22:20:53.740ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
86872025-03-17T22:20:55.116ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
86882025-03-17T22:20:55.343ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
86892025-03-17T22:20:55.343ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:63161) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
86902025-03-17T22:20:55.344ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
86912025-03-17T22:20:55.344ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
86922025-03-17T22:20:55.344ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
86932025-03-17T22:20:55.344ZINFOcrucible: send last flush ID to this DS: Some(JobId(2509)) = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
86942025-03-17T22:20:55.344ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
86952025-03-17T22:20:55.344ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(2509)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
86962025-03-17T22:20:55.345ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(2509)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
8697 ----------------------------------------------------------------
8698 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:1
8699 States: Active Active Active
8700 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8701 2810 Acked Read 2 Done Done Done true
8702 STATES DS:0 DS:1 DS:2 TOTAL
8703 Sent 0 0 0 0
8704 Done 1 1 1 3
8705 Skipped 0 0 0 0
8706 Error 0 0 0 0
8707 Last Flush: Some(JobId(2809)) Some(JobId(2809)) Some(JobId(2809))
8708 Downstairs last five completed: 2809 2808 2807 2806 2805
8709 Upstairs last five completed: 2810 2809 2808 2807 2806
8710 CLIENT: Up:0 ds:1 act:3
8711 ----------------------------------------------------------------
8712 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
8713 Downstairs last five completed: 2811 2810 2809 2808 2807
8714 Upstairs last five completed: 2811 2810 2809 2808 2807
8715 CLIENT: Up:0 ds:0 act:3
8716 Replay: All jobs finished
8717 Main task has work to do, go find it
8718 got dsc Stop(1)
8719 stop 1
8720 Mar 17 22:20:59.590 INFO request completed[8820] Got stop action so:false kr:false
8721 , latency_us: 355, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 6e5854e0-6a70-4e3b-999b-a22edbf75a5d, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
87222025-03-17T22:20:59.591ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
87232025-03-17T22:20:59.591ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
87242025-03-17T22:20:59.591ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
8725 Mar 17 22:20:59.591 INFO request completed, latency_us: 166, response_code: 200, uri: /state/cid/1, method: GET, req_id: fc4df614-f5e7-45bd-b660-9093342d676c, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
87262025-03-17T22:20:59.591ZINFOcrucible: client task is exiting = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
8727 [8820] Exited with: ExitStatus(unix_wait_status(9))
8728 [8820][1] reports Exit
8729 Mar 17 22:21:03.593 INFO request completed, latency_us: 192, response_code: 200, uri: /state/cid/1, method: GET, req_id: 0e592cba-ffc6-4e2b-b6f4-9d3c59f6d99b, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
8730 001/300 Read block 2367 len 36864
8731 002/300 Read block 1417 len 20480
8732 003/300 Read block 846 len 40960
8733 004/300 Write block 4139 len 20480 data: 2 2 2 3 4
8734 005/300 Read block 4347 len 36864
8735 006/300 Write block 1133 len 4096 data: 3
8736 007/300 Flush
8737 008/300 Flush
8738 009/300 Read block 1203 len 28672
8739 010/300 Read block 1265 len 4096
8740 011/300 Read block 4341 len 16384
8741 012/300 Write block 1929 len 32768 data: 3 2 3 4 5 4 4 4
8742 013/300 Write block 4999 len 4096 data: 3
8743 014/300 Read block 3731 len 40960
8744 015/300 Read block 1190 len 28672
8745 016/300 Read block 4558 len 40960
8746 017/300 Read block 1978 len 36864
8747 018/300 Write block 927 len 20480 data: 2 2 2 2 2
8748 019/300 Read block 3251 len 8192
8749 020/300 Write block 3377 len 12288 data: 2 2 2
8750 021/300 Read block 2626 len 32768
8751 022/300 Write block 3114 len 28672 data: 5 4 4 3 3 3 3
8752 023/300 Write block 2521 len 8192 data: 3 3
8753 024/300 Write block 3942 len 36864 data: 3 3 3 3 3 3 3 3 3
8754 025/300 Read block 2935 len 32768
8755 026/300 Write block 2959 len 32768 data: 3 4 3 3 3 3 3 2
8756 027/300 Flush
8757 028/300 Write block 1962 len 40960 data: 2 2 2 2 3 3 3 3 3 3
8758 029/300 Write block 1531 len 24576 data: 2 2 2 2 2 2
8759 030/300 Flush
8760 031/300 Flush
8761 032/300 Write block 182 len 24576 data: 4 4 3 3 3 4
8762 033/300 Flush
8763 034/300 Read block 1828 len 8192
8764 035/300 Write block 4538 len 28672 data: 5 4 4 2 2 3 3
8765 036/300 Flush
8766 037/300 Read block 1601 len 32768
8767 038/300 Write block 3938 len 20480 data: 3 3 3 3 4
8768 039/300 Read block 53 len 16384
8769 040/300 Read block 3057 len 32768
8770 041/300 Read block 2634 len 36864
8771 042/300 Flush
8772 043/300 Read block 177 len 20480
8773 044/300 Write block 3408 len 28672 data: 3 2 2 2 2 2 2
8774 045/300 Write block 604 len 28672 data: 2 2 2 2 2 2 2
8775 046/300 Read block 1108 len 8192
8776 047/300 Read block 2111 len 16384
8777 048/300 Read block 4345 len 4096
8778 049/300 Write block 3852 len 24576 data: 3 3 3 2 2 2
8779 050/300 Read block 2338 len 24576
8780 051/300 Read block 832 len 28672
8781 052/300 Write block 3178 len 4096 data: 3
8782 053/300 Read block 241 len 4096
8783 054/300 Read block 2206 len 36864
8784 055/300 Write block 1375 len 4096 data: 3
8785 056/300 Write block 4225 len 40960 data: 4 4 4 2 2 2 3 2 2 2
8786 057/300 Write block 1424 len 8192 data: 2 2
8787 058/300 Read block 355 len 20480
8788 059/300 Write block 4917 len 40960 data: 2 2 2 2 2 2 2 2 2 2
8789 060/300 Read block 1657 len 4096
8790 061/300 Read block 1121 len 32768
8791 062/300 Read block 2649 len 12288
8792 063/300 Read block 3932 len 8192
8793 064/300 Write block 4188 len 28672 data: 2 2 3 4 3 2 3
8794 065/300 Read block 2743 len 20480
8795 066/300 Read block 3246 len 36864
8796 067/300 Write block 1358 len 8192 data: 2 2
8797 068/300 Write block 2664 len 24576 data: 2 2 2 2 2 2
8798 069/300 Write block 147 len 32768 data: 3 3 4 4 3 4 5 6
8799 070/300 Read block 4540 len 4096
8800 071/300 Read block 3745 len 28672
8801 072/300 Read block 328 len 8192
8802 073/300 Write block 3692 len 40960 data: 3 4 3 3 3 2 2 2 2 2
8803 074/300 Read block 699 len 32768
8804 075/300 Read block 4459 len 24576
8805 076/300 Write block 4829 len 32768 data: 2 2 2 2 3 2 2 2
8806 077/300 Write block 4932 len 8192 data: 3 3
8807 078/300 Read block 1940 len 4096
8808 079/300 Write block 3939 len 20480 data: 4 4 4 5 4
8809 080/300 Read block 2685 len 12288
8810 081/300 Write block 797 len 4096 data: 4
8811 082/300 Read block 525 len 40960
8812 083/300 Read block 3650 len 20480
8813 084/300 Write block 4220 len 16384 data: 4 4 4 4
8814 085/300 Write block 4910 len 16384 data: 2 2 2 2
8815 086/300 Write block 4816 len 32768 data: 3 3 3 3 4 4 4 4
8816 087/300 Write block 1926 len 36864 data: 4 3 3 4 3 4 5 6 5
8817 088/300 Write block 2338 len 32768 data: 3 3 3 3 3 3 3 3
8818 089/300 Read block 4483 len 40960
8819 090/300 Write block 4803 len 20480 data: 4 5 5 3 2
8820 091/300 Write block 791 len 24576 data: 2 2 2 2 3 4
8821 092/300 Write block 2935 len 20480 data: 2 2 2 2 3
8822 093/300 Read block 4049 len 4096
8823 094/300 Read block 592 len 8192
8824 095/300 Read block 2797 len 16384
8825 096/300 Write block 2419 len 28672 data: 4 4 4 4 3 3 3
8826 097/300 Read block 3702 len 40960
8827 098/300 Read block 1784 len 36864
8828 099/300 Write block 2281 len 12288 data: 3 2 2
8829 100/300 Read block 3476 len 12288
8830 101/300 Write block 2239 len 20480 data: 3 3 2 2 2
8831 102/300 Read block 936 len 24576
8832 103/300 Read block 3095 len 4096
8833 104/300 Write block 1903 len 28672 data: 3 2 2 3 4 5 5
8834 105/300 Write block 3044 len 40960 data: 3 3 3 3 3 3 3 3 2 2
8835 106/300 Flush
8836 107/300 Flush
8837 108/300 Read block 3314 len 32768
8838 109/300 Read block 2548 len 28672
8839 110/300 Read block 2725 len 28672
8840 111/300 Read block 2058 len 8192
8841 112/300 Write block 83 len 36864 data: 2 2 4 4 4 5 3 2 2
8842 113/300 Write block 4208 len 12288 data: 3 3 4
8843 114/300 Flush
8844 115/300 Read block 4058 len 20480
8845 116/300 Read block 4932 len 24576
8846 117/300 Write block 2536 len 20480 data: 2 2 3 3 3
8847 118/300 Read block 4835 len 20480
8848 119/300 Read block 75 len 8192
8849 120/300 Flush
8850 121/300 Write block 2703 len 28672 data: 2 2 3 3 4 4 4
8851 122/300 Flush
8852 123/300 Write block 674 len 28672 data: 2 2 3 3 3 2 2
8853 124/300 Read block 4640 len 40960
8854 125/300 Write block 2116 len 40960 data: 2 2 2 2 2 2 2 2 2 2
8855 126/300 Write block 2154 len 12288 data: 2 2 2
8856 127/300 Read block 1802 len 12288
8857 128/300 Write block 1879 len 40960 data: 3 3 3 2 3 3 3 3 2 2
8858 129/300 Read block 3005 len 12288
8859 130/300 Read block 3800 len 32768
8860 131/300 Read block 899 len 8192
8861 132/300 Read block 4503 len 24576
8862 133/300 Flush
8863 134/300 Read block 1091 len 24576
8864 135/300 Write block 4739 len 36864 data: 4 4 4 3 3 3 3 3 2
8865 136/300 Flush
8866 137/300 Read block 1125 len 16384
8867 138/300 Flush
8868 139/300 Write block 905 len 36864 data: 3 3 3 4 4 5 4 3 3
8869 140/300 Write block 4782 len 28672 data: 3 3 3 3 3 3 3
8870 141/300 Write block 3928 len 4096 data: 2
8871 142/300 Write block 4652 len 4096 data: 3
8872 143/300 Flush
8873 144/300 Read block 1701 len 32768
8874 145/300 Write block 929 len 28672 data: 3 3 3 2 2 2 2
8875 146/300 Flush
8876 147/300 Read block 1926 len 8192
8877 148/300 Read block 3754 len 20480
8878 149/300 Read block 344 len 36864
8879 150/300 Read block 2577 len 32768
8880 151/300 Read block 854 len 4096
8881 152/300 Read block 1065 len 20480
8882 153/300 Read block 2775 len 40960
8883 154/300 Write block 4803 len 4096 data: 5
8884 155/300 Read block 2034 len 36864
8885 156/300 Flush
8886 157/300 Read block 2656 len 12288
8887 158/300 Write block 3443 len 40960 data: 2 2 2 2 2 2 2 2 2 2
8888 159/300 Read block 2215 len 20480
8889 160/300 Write block 856 len 20480 data: 3 4 3 3 3
8890 161/300 Write block 2 len 40960 data: 3 3 3 3 3 2 2 2 2 2
8891 162/300 Write block 590 len 36864 data: 2 2 2 2 2 2 2 2 2
8892 163/300 Read block 3374 len 40960
8893 164/300 Write block 4035 len 4096 data: 2
8894 165/300 Read block 1944 len 20480
8895 166/300 Write block 217 len 4096 data: 2
8896 167/300 Read block 712 len 8192
8897 168/300 Write block 4014 len 20480 data: 3 3 3 3 3
8898 169/300 Read block 4706 len 36864
8899 170/300 Write block 2505 len 24576 data: 2 2 2 2 2 2
8900 171/300 Write block 3949 len 40960 data: 4 4 3 3 3 2 2 3 3 3
8901 172/300 Read block 3090 len 12288
8902 173/300 Write block 4568 len 32768 data: 2 2 2 2 2 2 2 2
8903 174/300 Write block 300 len 4096 data: 3
8904 175/300 Write block 158 len 40960 data: 3 3 3 3 3 4 4 3 3 3
8905 176/300 Write block 4238 len 20480 data: 2 2 2 2 3
8906 177/300 Write block 2425 len 8192 data: 4 2
8907 178/300 Read block 986 len 4096
8908 179/300 Read block 3281 len 24576
8909 180/300 Read block 4208 len 8192
8910 181/300 Write block 456 len 12288 data: 3 3 3
8911 182/300 Read block 1027 len 4096
8912 183/300 Write block 4860 len 40960 data: 2 2 3 3 3 3 3 2 3 3
8913 184/300 Write block 4140 len 12288 data: 3 3 4
8914 185/300 Read block 1201 len 8192
8915 186/300 Write block 686 len 28672 data: 3 3 3 3 2 2 2
8916 187/300 Flush
8917 188/300 Write block 4266 len 16384 data: 2 2 3 3
8918 189/300 Read block 506 len 12288
8919 190/300 Flush
8920 191/300 Read block 1713 len 24576
8921 192/300 Read block 2969 len 28672
8922 193/300 Write block 3976 len 40960 data: 4 3 4 4 4 3 3 2 2 2
8923 194/300 Read block 3382 len 24576
8924 195/300 Write block 4449 len 16384 data: 4 3 3 3
8925 196/300 Read block 4067 len 32768
8926 197/300 Read block 196 len 32768
8927 198/300 Read block 3886 len 40960
8928 199/300 Read block 4293 len 28672
8929 200/300 Read block 896 len 32768
8930 201/300 Read block 3650 len 4096
8931 202/300 Read block 2539 len 32768
8932 203/300 Read block 4339 len 16384
8933 204/300 Write block 3136 len 32768 data: 4 4 4 3 3 3 3 3
8934 205/300 Read block 4903 len 4096
8935 206/300 Write block 25 len 24576 data: 2 2 3 3 4 5
8936 207/300 Flush
8937 208/300 Read block 806 len 8192
8938 209/300 Read block 2255 len 8192
8939 210/300 Write block 3127 len 24576 data: 4 4 4 4 4 3
8940 211/300 Write block 634 len 28672 data: 2 2 2 2 2 2 2
8941 212/300 Write block 4569 len 4096 data: 3
8942 213/300 Write block 262 len 32768 data: 2 2 2 2 3 3 3 4
8943 214/300 Read block 3655 len 28672
8944 215/300 Read block 4493 len 36864
8945 216/300 Write block 2370 len 32768 data: 2 2 3 3 3 3 3 3
8946 217/300 Write block 341 len 12288 data: 2 2 2
8947 218/300 Read block 449 len 28672
8948 219/300 Read block 2790 len 4096
8949 220/300 Flush
8950 221/300 Write block 3406 len 40960 data: 5 4 4 3 3 3 3 3 3 2
8951 222/300 Write block 3561 len 24576 data: 2 2 2 3 3 3
8952 223/300 Write block 2942 len 4096 data: 3
8953 224/300 Read block 2097 len 12288
8954 225/300 Read block 1776 len 28672
8955 226/300 Read block 1139 len 20480
8956 227/300 Read block 3073 len 28672
8957 228/300 Read block 1005 len 40960
8958 229/300 Write block 1053 len 20480 data: 3 3 3 3 3
8959 230/300 Write block 4673 len 8192 data: 3 3
8960 231/300 Flush
8961 232/300 Read block 3588 len 36864
8962 233/300 Read block 426 len 32768
8963 234/300 Read block 934 len 28672
8964 235/300 Write block 768 len 8192 data: 2 3
8965 236/300 Write block 2045 len 20480 data: 2 2 2 2 2
8966 237/300 Read block 1951 len 28672
8967 238/300 Write block 187 len 32768 data: 5 4 4 4 3 2 2 2
8968 239/300 Write block 940 len 40960 data: 2 2 2 3 5 5 5 5 5 5
8969 240/300 Read block 470 len 36864
8970 241/300 Write block 3692 len 28672 data: 4 5 4 4 4 3 3
8971 242/300 Read block 1179 len 20480
8972 243/300 Write block 4264 len 24576 data: 4 3 3 3 4 4
8973 244/300 Flush
8974 245/300 Flush
8975 246/300 Read block 189 len 36864
8976 247/300 Read block 4663 len 20480
8977 248/300 Write block 2401 len 24576 data: 4 4 3 3 3 3
8978 249/300 Read block 2245 len 4096
8979 250/300 Read block 3250 len 40960
8980 251/300 Write block 4374 len 20480 data: 3 3 3 2 3
8981 252/300 Write block 4718 len 8192 data: 3 3
8982 253/300 Read block 4099 len 28672
8983 254/300 Read block 1606 len 36864
8984 255/300 Write block 3117 len 24576 data: 4 4 4 4 3 2
8985 256/300 Read block 893 len 40960
8986 257/300 Flush
8987 258/300 Read block 1401 len 36864
8988 259/300 Write block 2248 len 24576 data: 2 2 2 2 2 2
8989 260/300 Flush
8990 261/300 Read block 2586 len 20480
8991 262/300 Write block 1427 len 8192 data: 2 2
8992 263/300 Read block 2182 len 12288
8993 264/300 Flush
8994 265/300 Write block 4857 len 40960 data: 2 3 2 3 3 4 4 4 4 4
8995 266/300 Flush
8996 267/300 Read block 1345 len 40960
8997 268/300 Read block 2651 len 24576
8998 269/300 Read block 1909 len 28672
8999 270/300 Write block 874 len 16384 data: 3 3 3 3
9000 271/300 Read block 2925 len 12288
9001 272/300 Read block 756 len 36864
9002 273/300 Write block 3951 len 40960 data: 4 4 4 3 3 4 4 4 3 3
9003 274/300 Read block 3511 len 16384
9004 275/300 Write block 4940 len 20480 data: 3 3 3 3 2
9005 276/300 Read block 3114 len 24576
9006 277/300 Read block 2276 len 40960
9007 278/300 Read block 1239 len 40960
9008 279/300 Read block 3438 len 24576
9009 280/300 Write block 3434 len 4096 data: 2
9010 281/300 Flush
9011 282/300 Write block 349 len 4096 data: 2
9012 283/300 Write block 1330 len 8192 data: 3 4
9013 284/300 Read block 1964 len 40960
9014 285/300 Read block 1071 len 36864
9015 286/300 Read block 4110 len 20480
9016 287/300 Read block 306 len 36864
9017 288/300 Read block 4019 len 4096
9018 289/300 Write block 45 len 8192 data: 4 3
9019 290/300 Read block 2306 len 24576
9020 291/300 Read block 4407 len 20480
9021 292/300 Read block 2220 len 8192
9022 293/300 Read block 1718 len 12288
9023 294/300 Read block 76 len 40960
9024 295/300 Read block 1050 len 28672
9025 296/300 Read block 185 len 36864
9026 297/300 Read block 2020 len 28672
9027 298/300 Read block 2103 len 36864
9028 299/300 Write block 185 len 4096 data: 4
9029 300/300 Read block 893 len 28672
9030 301/300 Read block 4415 len 12288
9031 Main task has work to do, go find it
9032 got dsc Start(1)
9033 start 1
9034 Mar 17 22:21:03.846 INFO request completed, latency_us: 207, response_code: 204, uri: /start/cid/1, method: POST, req_id: 9655fd10-de55-46be-a675-bd7594611f95, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
9035 Replay: started 1, returned:Ok(())
9036 ----------------------------------------------------------------
9037 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9038 States: Active Offline Active
9039 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9040 2812 Acked Read 9 Done Sent Done false
9041 2813 Acked Read 5 Done Sent Done false
9042 2814 Acked Read 10 Done Sent Done false
9043 2815 Acked Write 5 Done Sent Done false
9044 2816 Acked Read 9 Done Sent Done false
9045 2817 Acked Write 1 Done Sent Done false
9046 2818 Acked Flush 0 Done Sent Done false
9047 2819 Acked Flush 0 Done Sent Done false
9048 2820 Acked Read 7 Done Sent Done false
9049 2821 Acked Read 1 Done Sent Done false
9050 2822 Acked Read 4 Done Sent Done false
9051 2823 Acked Write 8 Done Sent Done false
9052 2824 Acked Write 1 Done Sent Done false
9053 2825 Acked Read 10 Done Sent Done false
9054 2826 Acked Read 7 Done Sent Done false
9055 2827 Acked Read 10 Done Sent Done false
9056 2828 Acked Read 9 Done Sent Done false
9057 2829 Acked Write 5 Done Sent Done false
9058 2830 Acked Read 2 Done Sent Done false
9059 2831 Acked Write 3 Done Sent Done false
9060 2832 Acked Read 8 Done Sent Done false
9061 2833 Acked Write 7 Done Sent Done false
9062 2834 Acked Write 2 Done Sent Done false
9063 2835 Acked Write 9 Done Sent Done false
9064 2836 Acked Read 8 Done Sent Done false
9065 2837 Acked Write 8 Done Sent Done false
9066 2838 Acked Flush 0 Done Sent Done false
9067 2839 Acked Write 10 Done Sent Done false
9068 2840 Acked Write 6 Done Sent Done false
9069 2841 Acked Flush 0 Done Sent Done false
9070 2842 Acked Flush 0 Done Sent Done false
9071 2843 Acked Write 6 Done Sent Done false
9072 2844 Acked Flush 0 Done Sent Done false
9073 2845 Acked Read 2 Done Sent Done false
9074 2846 Acked Write 7 Done Sent Done false
9075 2847 Acked Flush 0 Done Sent Done false
9076 2848 Acked Read 8 Done Sent Done false
9077 2849 Acked Write 5 Done Sent Done false
9078 2850 Acked Read 4 Done Sent Done false
9079 2851 Acked Read 8 Done Sent Done false
9080 2852 Acked Read 9 Done Sent Done false
9081 2853 Acked Flush 0 Done Sent Done false
9082 2854 Acked Read 5 Done Sent Done false
9083 2855 Acked Write 7 Done Sent Done false
9084 2856 Acked Write 7 Done Sent Done false
9085 2857 Acked Read 2 Done Sent Done false
9086 2858 Acked Read 4 Done Sent Done false
9087 2859 Acked Read 1 Done Sent Done false
9088 2860 Acked Write 6 Done Sent Done false
9089 2861 Acked Read 6 Done Sent Done false
9090 2862 Acked Read 7 Done Sent Done false
9091 2863 Acked Write 1 Done Sent Done false
9092 2864 Acked Read 1 Done Sent Done false
9093 2865 Acked Read 9 Done Sent Done false
9094 2866 Acked Write 1 Done Sent Done false
9095 2867 Acked Write 10 Done Sent Done false
9096 2868 Acked Write 2 Done Sent Done false
9097 2869 Acked Read 5 Done Sent Done false
9098 2870 Acked Write 10 Done Sent Done false
9099 2871 Acked Read 1 Done Sent Done false
9100 2872 Acked Read 8 Done Sent Done false
9101 2873 Acked Read 3 Done Sent Done false
9102 2874 Acked Read 2 Done Sent Done false
9103 2875 Acked Write 7 Done Sent Done false
9104 2876 Acked Read 5 Done Sent Done false
9105 2877 Acked Read 9 Done Sent Done false
9106 2878 Acked Write 2 Done Sent Done false
9107 2879 Acked Write 6 Done Sent Done false
9108 2880 Acked Write 8 Done Sent Done false
9109 2881 Acked Read 1 Done Sent Done false
9110 2882 Acked Read 7 Done Sent Done false
9111 2883 Acked Read 2 Done Sent Done false
9112 2884 Acked Write 10 Done Sent Done false
9113 2885 Acked Read 8 Done Sent Done false
9114 2886 Acked Read 6 Done Sent Done false
9115 2887 Acked Write 8 Done Sent Done false
9116 2888 Acked Write 2 Done Sent Done false
9117 2889 Acked Read 1 Done Sent Done false
9118 2890 Acked Write 5 Done Sent Done false
9119 2891 Acked Read 3 Done Sent Done false
9120 2892 Acked Write 1 Done Sent Done false
9121 2893 Acked Read 10 Done Sent Done false
9122 2894 Acked Read 5 Done Sent Done false
9123 2895 Acked Write 4 Done Sent Done false
9124 2896 Acked Write 4 Done Sent Done false
9125 2897 Acked Write 8 Done Sent Done false
9126 2898 Acked Write 9 Done Sent Done false
9127 2899 Acked Write 8 Done Sent Done false
9128 2900 Acked Read 10 Done Sent Done false
9129 2901 Acked Write 5 Done Sent Done false
9130 2902 Acked Write 6 Done Sent Done false
9131 2903 Acked Write 5 Done Sent Done false
9132 2904 Acked Read 1 Done Sent Done false
9133 2905 Acked Read 2 Done Sent Done false
9134 2906 Acked Read 4 Done Sent Done false
9135 2907 Acked Write 7 Done Sent Done false
9136 2908 Acked Read 10 Done Sent Done false
9137 2909 Acked Read 9 Done Sent Done false
9138 2910 Acked Write 3 Done Sent Done false
9139 2911 Acked Read 3 Done Sent Done false
9140 2912 Acked Write 5 Done Sent Done false
9141 2913 Acked Read 6 Done Sent Done false
9142 2914 Acked Read 1 Done Sent Done false
9143 2915 Acked Write 7 Done Sent Done false
9144 2916 Acked Write 10 Done Sent Done false
9145 2917 Acked Flush 0 Done Sent Done false
9146 2918 Acked Flush 0 Done Sent Done false
9147 2919 Acked Read 8 Done Sent Done false
9148 2920 Acked Read 7 Done Sent Done false
9149 2921 Acked Read 7 Done Sent Done false
9150 2922 Acked Read 2 Done Sent Done false
9151 2923 Acked Write 9 Done Sent Done false
9152 2924 Acked Write 3 Done Sent Done false
9153 2925 Acked Flush 0 Done Sent Done false
9154 2926 Acked Read 5 Done Sent Done false
9155 2927 Acked Read 6 Done Sent Done false
9156 2928 Acked Write 5 Done Sent Done false
9157 2929 Acked Read 5 Done Sent Done false
9158 2930 Acked Read 2 Done Sent Done false
9159 2931 Acked Flush 0 Done Sent Done false
9160 2932 Acked Write 7 Done Sent Done false
9161 2933 Acked Flush 0 Done Sent Done false
9162 2934 Acked Write 7 Done Sent Done false
9163 2935 Acked Read 10 Done Sent Done false
9164 2936 Acked Write 10 Done Sent Done false
9165 2937 Acked Write 3 Done Sent Done false
9166 2938 Acked Read 3 Done Sent Done false
9167 2939 Acked Write 10 Done Sent Done false
9168 2940 Acked Read 3 Done Sent Done false
9169 2941 Acked Read 8 Done Sent Done false
9170 2942 Acked Read 2 Done Sent Done false
9171 2943 Acked Read 6 Done Sent Done false
9172 2944 Acked Flush 0 Done Sent Done false
9173 2945 Acked Read 6 Done Sent Done false
9174 2946 Acked Write 9 Done Sent Done false
9175 2947 Acked Flush 0 Done Sent Done false
9176 2948 Acked Read 4 Done Sent Done false
9177 2949 Acked Flush 0 Done Sent Done false
9178 2950 Acked Write 9 Done Sent Done false
9179 2951 Acked Write 7 Done Sent Done false
9180 2952 Acked Write 1 Done Sent Done false
9181 2953 Acked Write 1 Done Sent Done false
9182 2954 Acked Flush 0 Done Sent Done false
9183 2955 Acked Read 8 Done Sent Done false
9184 2956 Acked Write 7 Done Sent Done false
9185 2957 Acked Flush 0 Done Sent Done false
9186 2958 Acked Read 2 Done Sent Done false
9187 2959 Acked Read 5 Done Sent Done false
9188 2960 Acked Read 9 Done Sent Done false
9189 2961 Acked Read 8 Done Sent Done false
9190 2962 Acked Read 1 Done Sent Done false
9191 2963 Acked Read 5 Done Sent Done false
9192 2964 Acked Read 10 Done Sent Done false
9193 2965 Acked Write 1 Done Sent Done false
9194 2966 Acked Read 9 Done Sent Done false
9195 2967 Acked Flush 0 Done Sent Done false
9196 2968 Acked Read 3 Done Sent Done false
9197 2969 Acked Write 10 Done Sent Done false
9198 2970 Acked Read 5 Done Sent Done false
9199 2971 Acked Write 5 Done Sent Done false
9200 2972 Acked Write 10 Done Sent Done false
9201 2973 Acked Write 9 Done Sent Done false
9202 2974 Acked Read 10 Done Sent Done false
9203 2975 Acked Write 1 Done Sent Done false
9204 2976 Acked Read 5 Done Sent Done false
9205 2977 Acked Write 1 Done Sent Done false
9206 2978 Acked Read 2 Done Sent Done false
9207 2979 Acked Write 5 Done Sent Done false
9208 2980 Acked Read 9 Done Sent Done false
9209 2981 Acked Write 6 Done Sent Done false
9210 2982 Acked Write 10 Done Sent Done false
9211 2983 Acked Read 3 Done Sent Done false
9212 2984 Acked Write 8 Done Sent Done false
9213 2985 Acked Write 1 Done Sent Done false
9214 2986 Acked Write 10 Done Sent Done false
9215 2987 Acked Write 5 Done Sent Done false
9216 2988 Acked Write 2 Done Sent Done false
9217 2989 Acked Read 1 Done Sent Done false
9218 2990 Acked Read 6 Done Sent Done false
9219 2991 Acked Read 2 Done Sent Done false
9220 2992 Acked Write 3 Done Sent Done false
9221 2993 Acked Read 1 Done Sent Done false
9222 2994 Acked Write 10 Done Sent Done false
9223 2995 Acked Write 3 Done Sent Done false
9224 2996 Acked Read 2 Done Sent Done false
9225 2997 Acked Write 7 Done Sent Done false
9226 2998 Acked Flush 0 Done Sent Done false
9227 2999 Acked Write 4 Done Sent Done false
9228 3000 Acked Read 3 Done Sent Done false
9229 3001 Acked Flush 0 Done Sent Done false
9230 3002 Acked Read 6 Done Sent Done false
9231 3003 Acked Read 7 Done Sent Done false
9232 3004 Acked Write 10 Done Sent Done false
9233 3005 Acked Read 6 Done Sent Done false
9234 3006 Acked Write 4 Done Sent Done false
9235 3007 Acked Read 8 Done Sent Done false
9236 3008 Acked Read 8 Done Sent Done false
9237 3009 Acked Read 10 Done Sent Done false
9238 3010 Acked Read 7 Done Sent Done false
9239 3011 Acked Read 8 Done Sent Done false
9240 3012 Acked Read 1 Done Sent Done false
9241 3013 Acked Read 8 Done Sent Done false
9242 3014 Acked Read 4 Done Sent Done false
9243 3015 Acked Write 8 Done Sent Done false
9244 3016 Acked Read 1 Done Sent Done false
9245 3017 Acked Write 6 Done Sent Done false
9246 3018 Acked Flush 0 Done Sent Done false
9247 3019 Acked Read 2 Done Sent Done false
9248 3020 Acked Read 2 Done Sent Done false
9249 3021 Acked Write 6 Done Sent Done false
9250 3022 Acked Write 7 Done Sent Done false
9251 3023 Acked Write 1 Done Sent Done false
9252 3024 Acked Write 8 Done Sent Done false
9253 3025 Acked Read 7 Done Sent Done false
9254 3026 Acked Read 9 Done Sent Done false
9255 3027 Acked Write 8 Done Sent Done false
9256 3028 Acked Write 3 Done Sent Done false
9257 3029 Acked Read 7 Done Sent Done false
9258 3030 Acked Read 1 Done Sent Done false
9259 3031 Acked Flush 0 Done Sent Done false
9260 3032 Acked Write 10 Done Sent Done false
9261 3033 Acked Write 6 Done Sent Done false
9262 3034 Acked Write 1 Done Sent Done false
9263 3035 Acked Read 3 Done Sent Done false
9264 3036 Acked Read 7 Done Sent Done false
9265 3037 Acked Read 5 Done Sent Done false
9266 3038 Acked Read 7 Done Sent Done false
9267 3039 Acked Read 10 Done Sent Done false
9268 3040 Acked Write 5 Done Sent Done false
9269 3041 Acked Write 2 Done Sent Done false
9270 3042 Acked Flush 0 Done Sent Done false
9271 3043 Acked Read 9 Done Sent Done false
9272 3044 Acked Read 8 Done Sent Done false
9273 3045 Acked Read 7 Done Sent Done false
9274 3046 Acked Write 2 Done Sent Done false
9275 3047 Acked Write 5 Done Sent Done false
9276 3048 Acked Read 7 Done Sent Done false
9277 3049 Acked Write 8 Done Sent Done false
9278 3050 Acked Write 10 Done Sent Done false
9279 3051 Acked Read 9 Done Sent Done false
9280 3052 Acked Write 7 Done Sent Done false
9281 3053 Acked Read 5 Done Sent Done false
9282 3054 Acked Write 6 Done Sent Done false
9283 3055 Acked Flush 0 Done Sent Done false
9284 3056 Acked Flush 0 Done Sent Done false
9285 3057 Acked Read 9 Done Sent Done false
9286 3058 Acked Read 5 Done Sent Done false
9287 3059 Acked Write 6 Done Sent Done false
9288 3060 Acked Read 1 Done Sent Done false
9289 3061 Acked Read 10 Done Sent Done false
9290 3062 Acked Write 5 Done Sent Done false
9291 3063 Acked Write 2 Done Sent Done false
9292 3064 Acked Read 7 Done Sent Done false
9293 3065 Acked Read 9 Done Sent Done false
9294 3066 Acked Write 6 Done Sent Done false
9295 3067 Acked Read 10 Done Sent Done false
9296 3068 Acked Flush 0 Done Sent Done false
9297 3069 Acked Read 9 Done Sent Done false
9298 3070 Acked Write 6 Done Sent Done false
9299 3071 Acked Flush 0 Done Sent Done false
9300 3072 Acked Read 5 Done Sent Done false
9301 3073 Acked Write 2 Done Sent Done false
9302 3074 Acked Read 3 Done Sent Done false
9303 3075 Acked Flush 0 Done Sent Done false
9304 3076 Acked Write 10 Done Sent Done false
9305 3077 Acked Flush 0 Done Sent Done false
9306 3078 Acked Read 10 Done Sent Done false
9307 3079 Acked Read 6 Done Sent Done false
9308 3080 Acked Read 7 Done Sent Done false
9309 3081 Acked Write 4 Done Sent Done false
9310 3082 Acked Read 3 Done Sent Done false
9311 3083 Acked Read 9 Done Sent Done false
9312 3084 Acked Write 10 Done Sent Done false
9313 3085 Acked Read 4 Done Sent Done false
9314 3086 Acked Write 5 Done Sent Done false
9315 3087 Acked Read 6 Done Sent Done false
9316 3088 Acked Read 10 Done Sent Done false
9317 3089 Acked Read 10 Done Sent Done false
9318 3090 Acked Read 6 Done Sent Done false
9319 3091 Acked Write 1 Done Sent Done false
9320 3092 Acked Flush 0 Done Sent Done false
9321 3093 Acked Write 1 Done Sent Done false
9322 3094 Acked Write 2 Done Sent Done false
9323 3095 Acked Read 10 Done Sent Done false
9324 3096 Acked Read 9 Done Sent Done false
9325 3097 Acked Read 5 Done Sent Done false
9326 3098 Acked Read 9 Done Sent Done false
9327 3099 Acked Read 1 Done Sent Done false
9328 3100 Acked Write 2 Done Sent Done false
9329 3101 Acked Read 6 Done Sent Done false
9330 3102 Acked Read 5 Done Sent Done false
9331 3103 Acked Read 2 Done Sent Done false
9332 3104 Acked Read 3 Done Sent Done false
9333 3105 Acked Read 10 Done Sent Done false
9334 3106 Acked Read 7 Done Sent Done false
9335 3107 Acked Read 9 Done Sent Done false
9336 3108 Acked Read 7 Done Sent Done false
9337 3109 Acked Read 9 Done Sent Done false
9338 3110 Acked Write 1 Done Sent Done false
9339 3111 Acked Read 7 Done Sent Done false
9340 3112 Acked Read 3 Done Sent Done false
9341 STATES DS:0 DS:1 DS:2 TOTAL
9342 Sent 0 301 0 301
9343 Done 301 0 301 602
9344 Skipped 0 0 0 0
9345 Error 0 0 0 0
9346 Last Flush: Some(JobId(3092)) Some(JobId(2811)) Some(JobId(3092))
9347 Downstairs last five completed: 2811 2810 2809 2808 2807
9348 Upstairs last five completed: 3112 3111 3110 3109 3108
9349 CLIENT: Up:0 ds:301 act:2
93502025-03-17T22:21:05.118ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
9351 [8820] got start action
9352 [8820] I am going to restart
9353 Starting downstairs at port 8820
9354 Make output file at "/tmp/dsc/downstairs-8820.txt"
9355 [8820][1] reports Starting
9356 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1218)
9357 [8820][1] reports Running
93582025-03-17T22:21:05.760ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
93592025-03-17T22:21:06.761ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
93602025-03-17T22:21:07.763ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
9361 ----------------------------------------------------------------
9362 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9363 States: Active Offline Active
9364 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9365 2812 Acked Read 9 Done Sent Done false
9366 2813 Acked Read 5 Done Sent Done false
9367 2814 Acked Read 10 Done Sent Done false
9368 2815 Acked Write 5 Done Sent Done false
9369 2816 Acked Read 9 Done Sent Done false
9370 2817 Acked Write 1 Done Sent Done false
9371 2818 Acked Flush 0 Done Sent Done false
9372 2819 Acked Flush 0 Done Sent Done false
9373 2820 Acked Read 7 Done Sent Done false
9374 2821 Acked Read 1 Done Sent Done false
9375 2822 Acked Read 4 Done Sent Done false
9376 2823 Acked Write 8 Done Sent Done false
9377 2824 Acked Write 1 Done Sent Done false
9378 2825 Acked Read 10 Done Sent Done false
9379 2826 Acked Read 7 Done Sent Done false
9380 2827 Acked Read 10 Done Sent Done false
9381 2828 Acked Read 9 Done Sent Done false
9382 2829 Acked Write 5 Done Sent Done false
9383 2830 Acked Read 2 Done Sent Done false
9384 2831 Acked Write 3 Done Sent Done false
9385 2832 Acked Read 8 Done Sent Done false
9386 2833 Acked Write 7 Done Sent Done false
9387 2834 Acked Write 2 Done Sent Done false
9388 2835 Acked Write 9 Done Sent Done false
9389 2836 Acked Read 8 Done Sent Done false
9390 2837 Acked Write 8 Done Sent Done false
9391 2838 Acked Flush 0 Done Sent Done false
9392 2839 Acked Write 10 Done Sent Done false
9393 2840 Acked Write 6 Done Sent Done false
9394 2841 Acked Flush 0 Done Sent Done false
9395 2842 Acked Flush 0 Done Sent Done false
9396 2843 Acked Write 6 Done Sent Done false
9397 2844 Acked Flush 0 Done Sent Done false
9398 2845 Acked Read 2 Done Sent Done false
9399 2846 Acked Write 7 Done Sent Done false
9400 2847 Acked Flush 0 Done Sent Done false
9401 2848 Acked Read 8 Done Sent Done false
9402 2849 Acked Write 5 Done Sent Done false
9403 2850 Acked Read 4 Done Sent Done false
9404 2851 Acked Read 8 Done Sent Done false
9405 2852 Acked Read 9 Done Sent Done false
9406 2853 Acked Flush 0 Done Sent Done false
9407 2854 Acked Read 5 Done Sent Done false
9408 2855 Acked Write 7 Done Sent Done false
9409 2856 Acked Write 7 Done Sent Done false
9410 2857 Acked Read 2 Done Sent Done false
9411 2858 Acked Read 4 Done Sent Done false
9412 2859 Acked Read 1 Done Sent Done false
9413 2860 Acked Write 6 Done Sent Done false
9414 2861 Acked Read 6 Done Sent Done false
9415 2862 Acked Read 7 Done Sent Done false
9416 2863 Acked Write 1 Done Sent Done false
9417 2864 Acked Read 1 Done Sent Done false
9418 2865 Acked Read 9 Done Sent Done false
9419 2866 Acked Write 1 Done Sent Done false
9420 2867 Acked Write 10 Done Sent Done false
9421 2868 Acked Write 2 Done Sent Done false
9422 2869 Acked Read 5 Done Sent Done false
9423 2870 Acked Write 10 Done Sent Done false
9424 2871 Acked Read 1 Done Sent Done false
9425 2872 Acked Read 8 Done Sent Done false
9426 2873 Acked Read 3 Done Sent Done false
9427 2874 Acked Read 2 Done Sent Done false
9428 2875 Acked Write 7 Done Sent Done false
9429 2876 Acked Read 5 Done Sent Done false
9430 2877 Acked Read 9 Done Sent Done false
9431 2878 Acked Write 2 Done Sent Done false
9432 2879 Acked Write 6 Done Sent Done false
9433 2880 Acked Write 8 Done Sent Done false
9434 2881 Acked Read 1 Done Sent Done false
9435 2882 Acked Read 7 Done Sent Done false
9436 2883 Acked Read 2 Done Sent Done false
9437 2884 Acked Write 10 Done Sent Done false
9438 2885 Acked Read 8 Done Sent Done false
9439 2886 Acked Read 6 Done Sent Done false
9440 2887 Acked Write 8 Done Sent Done false
9441 2888 Acked Write 2 Done Sent Done false
9442 2889 Acked Read 1 Done Sent Done false
9443 2890 Acked Write 5 Done Sent Done false
9444 2891 Acked Read 3 Done Sent Done false
9445 2892 Acked Write 1 Done Sent Done false
9446 2893 Acked Read 10 Done Sent Done false
9447 2894 Acked Read 5 Done Sent Done false
9448 2895 Acked Write 4 Done Sent Done false
9449 2896 Acked Write 4 Done Sent Done false
9450 2897 Acked Write 8 Done Sent Done false
9451 2898 Acked Write 9 Done Sent Done false
9452 2899 Acked Write 8 Done Sent Done false
9453 2900 Acked Read 10 Done Sent Done false
9454 2901 Acked Write 5 Done Sent Done false
9455 2902 Acked Write 6 Done Sent Done false
9456 2903 Acked Write 5 Done Sent Done false
9457 2904 Acked Read 1 Done Sent Done false
9458 2905 Acked Read 2 Done Sent Done false
9459 2906 Acked Read 4 Done Sent Done false
9460 2907 Acked Write 7 Done Sent Done false
9461 2908 Acked Read 10 Done Sent Done false
9462 2909 Acked Read 9 Done Sent Done false
9463 2910 Acked Write 3 Done Sent Done false
9464 2911 Acked Read 3 Done Sent Done false
9465 2912 Acked Write 5 Done Sent Done false
9466 2913 Acked Read 6 Done Sent Done false
9467 2914 Acked Read 1 Done Sent Done false
9468 2915 Acked Write 7 Done Sent Done false
9469 2916 Acked Write 10 Done Sent Done false
9470 2917 Acked Flush 0 Done Sent Done false
9471 2918 Acked Flush 0 Done Sent Done false
9472 2919 Acked Read 8 Done Sent Done false
9473 2920 Acked Read 7 Done Sent Done false
9474 2921 Acked Read 7 Done Sent Done false
9475 2922 Acked Read 2 Done Sent Done false
9476 2923 Acked Write 9 Done Sent Done false
9477 2924 Acked Write 3 Done Sent Done false
9478 2925 Acked Flush 0 Done Sent Done false
9479 2926 Acked Read 5 Done Sent Done false
9480 2927 Acked Read 6 Done Sent Done false
9481 2928 Acked Write 5 Done Sent Done false
9482 2929 Acked Read 5 Done Sent Done false
9483 2930 Acked Read 2 Done Sent Done false
9484 2931 Acked Flush 0 Done Sent Done false
9485 2932 Acked Write 7 Done Sent Done false
9486 2933 Acked Flush 0 Done Sent Done false
9487 2934 Acked Write 7 Done Sent Done false
9488 2935 Acked Read 10 Done Sent Done false
9489 2936 Acked Write 10 Done Sent Done false
9490 2937 Acked Write 3 Done Sent Done false
9491 2938 Acked Read 3 Done Sent Done false
9492 2939 Acked Write 10 Done Sent Done false
9493 2940 Acked Read 3 Done Sent Done false
9494 2941 Acked Read 8 Done Sent Done false
9495 2942 Acked Read 2 Done Sent Done false
9496 2943 Acked Read 6 Done Sent Done false
9497 2944 Acked Flush 0 Done Sent Done false
9498 2945 Acked Read 6 Done Sent Done false
9499 2946 Acked Write 9 Done Sent Done false
9500 2947 Acked Flush 0 Done Sent Done false
9501 2948 Acked Read 4 Done Sent Done false
9502 2949 Acked Flush 0 Done Sent Done false
9503 2950 Acked Write 9 Done Sent Done false
9504 2951 Acked Write 7 Done Sent Done false
9505 2952 Acked Write 1 Done Sent Done false
9506 2953 Acked Write 1 Done Sent Done false
9507 2954 Acked Flush 0 Done Sent Done false
9508 2955 Acked Read 8 Done Sent Done false
9509 2956 Acked Write 7 Done Sent Done false
9510 2957 Acked Flush 0 Done Sent Done false
9511 2958 Acked Read 2 Done Sent Done false
9512 2959 Acked Read 5 Done Sent Done false
9513 2960 Acked Read 9 Done Sent Done false
9514 2961 Acked Read 8 Done Sent Done false
9515 2962 Acked Read 1 Done Sent Done false
9516 2963 Acked Read 5 Done Sent Done false
9517 2964 Acked Read 10 Done Sent Done false
9518 2965 Acked Write 1 Done Sent Done false
9519 2966 Acked Read 9 Done Sent Done false
9520 2967 Acked Flush 0 Done Sent Done false
9521 2968 Acked Read 3 Done Sent Done false
9522 2969 Acked Write 10 Done Sent Done false
9523 2970 Acked Read 5 Done Sent Done false
9524 2971 Acked Write 5 Done Sent Done false
9525 2972 Acked Write 10 Done Sent Done false
9526 2973 Acked Write 9 Done Sent Done false
9527 2974 Acked Read 10 Done Sent Done false
9528 2975 Acked Write 1 Done Sent Done false
9529 2976 Acked Read 5 Done Sent Done false
9530 2977 Acked Write 1 Done Sent Done false
9531 2978 Acked Read 2 Done Sent Done false
9532 2979 Acked Write 5 Done Sent Done false
9533 2980 Acked Read 9 Done Sent Done false
9534 2981 Acked Write 6 Done Sent Done false
9535 2982 Acked Write 10 Done Sent Done false
9536 2983 Acked Read 3 Done Sent Done false
9537 2984 Acked Write 8 Done Sent Done false
9538 2985 Acked Write 1 Done Sent Done false
9539 2986 Acked Write 10 Done Sent Done false
9540 2987 Acked Write 5 Done Sent Done false
9541 2988 Acked Write 2 Done Sent Done false
9542 2989 Acked Read 1 Done Sent Done false
9543 2990 Acked Read 6 Done Sent Done false
9544 2991 Acked Read 2 Done Sent Done false
9545 2992 Acked Write 3 Done Sent Done false
9546 2993 Acked Read 1 Done Sent Done false
9547 2994 Acked Write 10 Done Sent Done false
9548 2995 Acked Write 3 Done Sent Done false
9549 2996 Acked Read 2 Done Sent Done false
9550 2997 Acked Write 7 Done Sent Done false
9551 2998 Acked Flush 0 Done Sent Done false
9552 2999 Acked Write 4 Done Sent Done false
9553 3000 Acked Read 3 Done Sent Done false
9554 3001 Acked Flush 0 Done Sent Done false
9555 3002 Acked Read 6 Done Sent Done false
9556 3003 Acked Read 7 Done Sent Done false
9557 3004 Acked Write 10 Done Sent Done false
9558 3005 Acked Read 6 Done Sent Done false
9559 3006 Acked Write 4 Done Sent Done false
9560 3007 Acked Read 8 Done Sent Done false
9561 3008 Acked Read 8 Done Sent Done false
9562 3009 Acked Read 10 Done Sent Done false
9563 3010 Acked Read 7 Done Sent Done false
9564 3011 Acked Read 8 Done Sent Done false
9565 3012 Acked Read 1 Done Sent Done false
9566 3013 Acked Read 8 Done Sent Done false
9567 3014 Acked Read 4 Done Sent Done false
9568 3015 Acked Write 8 Done Sent Done false
9569 3016 Acked Read 1 Done Sent Done false
9570 3017 Acked Write 6 Done Sent Done false
9571 3018 Acked Flush 0 Done Sent Done false
9572 3019 Acked Read 2 Done Sent Done false
9573 3020 Acked Read 2 Done Sent Done false
9574 3021 Acked Write 6 Done Sent Done false
9575 3022 Acked Write 7 Done Sent Done false
9576 3023 Acked Write 1 Done Sent Done false
9577 3024 Acked Write 8 Done Sent Done false
9578 3025 Acked Read 7 Done Sent Done false
9579 3026 Acked Read 9 Done Sent Done false
9580 3027 Acked Write 8 Done Sent Done false
9581 3028 Acked Write 3 Done Sent Done false
9582 3029 Acked Read 7 Done Sent Done false
9583 3030 Acked Read 1 Done Sent Done false
9584 3031 Acked Flush 0 Done Sent Done false
9585 3032 Acked Write 10 Done Sent Done false
9586 3033 Acked Write 6 Done Sent Done false
9587 3034 Acked Write 1 Done Sent Done false
9588 3035 Acked Read 3 Done Sent Done false
9589 3036 Acked Read 7 Done Sent Done false
9590 3037 Acked Read 5 Done Sent Done false
9591 3038 Acked Read 7 Done Sent Done false
9592 3039 Acked Read 10 Done Sent Done false
9593 3040 Acked Write 5 Done Sent Done false
9594 3041 Acked Write 2 Done Sent Done false
9595 3042 Acked Flush 0 Done Sent Done false
9596 3043 Acked Read 9 Done Sent Done false
9597 3044 Acked Read 8 Done Sent Done false
9598 3045 Acked Read 7 Done Sent Done false
9599 3046 Acked Write 2 Done Sent Done false
9600 3047 Acked Write 5 Done Sent Done false
9601 3048 Acked Read 7 Done Sent Done false
9602 3049 Acked Write 8 Done Sent Done false
9603 3050 Acked Write 10 Done Sent Done false
9604 3051 Acked Read 9 Done Sent Done false
9605 3052 Acked Write 7 Done Sent Done false
9606 3053 Acked Read 5 Done Sent Done false
9607 3054 Acked Write 6 Done Sent Done false
9608 3055 Acked Flush 0 Done Sent Done false
9609 3056 Acked Flush 0 Done Sent Done false
9610 3057 Acked Read 9 Done Sent Done false
9611 3058 Acked Read 5 Done Sent Done false
9612 3059 Acked Write 6 Done Sent Done false
9613 3060 Acked Read 1 Done Sent Done false
9614 3061 Acked Read 10 Done Sent Done false
9615 3062 Acked Write 5 Done Sent Done false
9616 3063 Acked Write 2 Done Sent Done false
9617 3064 Acked Read 7 Done Sent Done false
9618 3065 Acked Read 9 Done Sent Done false
9619 3066 Acked Write 6 Done Sent Done false
9620 3067 Acked Read 10 Done Sent Done false
9621 3068 Acked Flush 0 Done Sent Done false
9622 3069 Acked Read 9 Done Sent Done false
9623 3070 Acked Write 6 Done Sent Done false
9624 3071 Acked Flush 0 Done Sent Done false
9625 3072 Acked Read 5 Done Sent Done false
9626 3073 Acked Write 2 Done Sent Done false
9627 3074 Acked Read 3 Done Sent Done false
9628 3075 Acked Flush 0 Done Sent Done false
9629 3076 Acked Write 10 Done Sent Done false
9630 3077 Acked Flush 0 Done Sent Done false
9631 3078 Acked Read 10 Done Sent Done false
9632 3079 Acked Read 6 Done Sent Done false
9633 3080 Acked Read 7 Done Sent Done false
9634 3081 Acked Write 4 Done Sent Done false
9635 3082 Acked Read 3 Done Sent Done false
9636 3083 Acked Read 9 Done Sent Done false
9637 3084 Acked Write 10 Done Sent Done false
9638 3085 Acked Read 4 Done Sent Done false
9639 3086 Acked Write 5 Done Sent Done false
9640 3087 Acked Read 6 Done Sent Done false
9641 3088 Acked Read 10 Done Sent Done false
9642 3089 Acked Read 10 Done Sent Done false
9643 3090 Acked Read 6 Done Sent Done false
9644 3091 Acked Write 1 Done Sent Done false
9645 3092 Acked Flush 0 Done Sent Done false
9646 3093 Acked Write 1 Done Sent Done false
9647 3094 Acked Write 2 Done Sent Done false
9648 3095 Acked Read 10 Done Sent Done false
9649 3096 Acked Read 9 Done Sent Done false
9650 3097 Acked Read 5 Done Sent Done false
9651 3098 Acked Read 9 Done Sent Done false
9652 3099 Acked Read 1 Done Sent Done false
9653 3100 Acked Write 2 Done Sent Done false
9654 3101 Acked Read 6 Done Sent Done false
9655 3102 Acked Read 5 Done Sent Done false
9656 3103 Acked Read 2 Done Sent Done false
9657 3104 Acked Read 3 Done Sent Done false
9658 3105 Acked Read 10 Done Sent Done false
9659 3106 Acked Read 7 Done Sent Done false
9660 3107 Acked Read 9 Done Sent Done false
9661 3108 Acked Read 7 Done Sent Done false
9662 3109 Acked Read 9 Done Sent Done false
9663 3110 Acked Write 1 Done Sent Done false
9664 3111 Acked Read 7 Done Sent Done false
9665 3112 Acked Read 3 Done Sent Done false
9666 STATES DS:0 DS:1 DS:2 TOTAL
9667 Sent 0 301 0 301
9668 Done 301 0 301 602
9669 Skipped 0 0 0 0
9670 Error 0 0 0 0
9671 Last Flush: Some(JobId(3092)) Some(JobId(2811)) Some(JobId(3092))
9672 Downstairs last five completed: 2811 2810 2809 2808 2807
9673 Upstairs last five completed: 3112 3111 3110 3109 3108
9674 CLIENT: Up:0 ds:301 act:2
96752025-03-17T22:21:08.765ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
96762025-03-17T22:21:09.593ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
96772025-03-17T22:21:09.593ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:39625) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
96782025-03-17T22:21:09.594ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
96792025-03-17T22:21:09.595ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
96802025-03-17T22:21:09.595ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
96812025-03-17T22:21:09.595ZINFOcrucible: send last flush ID to this DS: Some(JobId(2811)) = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
96822025-03-17T22:21:09.595ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
96832025-03-17T22:21:09.595ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(2811)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
96842025-03-17T22:21:09.595ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(2811)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
9685 ----------------------------------------------------------------
9686 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
9687 Downstairs last five completed: 3113 3112 3111 3110 3109
9688 Upstairs last five completed: 3113 3112 3111 3110 3109
9689 CLIENT: Up:0 ds:0 act:3
9690 Replay: All jobs finished
9691 Main task has work to do, go find it
9692 got dsc Stop(2)
9693 stop 2
9694 Mar 17 22:21:11.850 INFO request completed, latency_us: 340, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 1e59cff2-8a49-4992-ab2a-4395755e638f, remote_addr: 127.0.0.1:[8830] Got stop action so:false kr:false
9695 58317, local_addr: 127.0.0.1:9998
9696 Mar 17 22:21:11.851 INFO request completed, latency_us{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:21:11.85153356Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
9697 : 141, response_code{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:21:11.851618801Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
9698 : 200, uri: /state/cid/2, method: {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:21:11.851651597Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
9699 GET, req_id: {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:11.851688873Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
9700 ef5a5c39-a64a-4043-8e42-a1841ee9e464, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
9701 [8830] Exited with: ExitStatus(unix_wait_status(9))
9702 [8830][2] reports Exit
9703 Mar 17 22:21:15.853 INFO request completed, latency_us: 212, response_code: 200, uri: /state/cid/2, method: GET, req_id: 4a19d0a8-c80f-407b-aa57-8600f491a67e, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
9704 001/300 Read block 636 len 40960
9705 002/300 Write block 530 len 4096 data: 2
9706 003/300 Write block 2490 len 20480 data: 2 3 3 3 3
9707 004/300 Read block 1941 len 8192
9708 005/300 Read block 2328 len 4096
9709 006/300 Read block 1151 len 4096
9710 007/300 Write block 4965 len 12288 data: 3 3 3
9711 008/300 Write block 2668 len 8192 data: 3 3
9712 009/300 Flush
9713 010/300 Write block 486 len 16384 data: 3 3 3 3
9714 011/300 Read block 4568 len 28672
9715 012/300 Flush
9716 013/300 Write block 3618 len 24576 data: 2 2 2 2 3 3
9717 014/300 Flush
9718 015/300 Write block 4494 len 20480 data: 2 2 3 3 3
9719 016/300 Read block 895 len 20480
9720 017/300 Write block 4114 len 40960 data: 2 2 2 2 2 2 2 2 2 3
9721 018/300 Read block 4296 len 20480
9722 019/300 Write block 1733 len 4096 data: 2
9723 020/300 Write block 1521 len 16384 data: 2 2 2 2
9724 021/300 Read block 4678 len 40960
9725 022/300 Write block 1053 len 28672 data: 4 4 4 4 4 3 3
9726 023/300 Flush
9727 024/300 Read block 1493 len 20480
9728 025/300 Read block 4797 len 12288
9729 026/300 Read block 4743 len 36864
9730 027/300 Write block 2079 len 8192 data: 2 2
9731 028/300 Read block 344 len 28672
9732 029/300 Read block 1858 len 36864
9733 030/300 Read block 1952 len 20480
9734 031/300 Write block 1449 len 16384 data: 3 3 2 2
9735 032/300 Write block 2898 len 20480 data: 2 2 2 2 2
9736 033/300 Read block 3035 len 28672
9737 034/300 Write block 4931 len 28672 data: 3 4 4 2 2 2 2
9738 035/300 Read block 231 len 12288
9739 036/300 Read block 3386 len 4096
9740 037/300 Read block 3415 len 16384
9741 038/300 Write block 1478 len 24576 data: 2 2 2 2 2 2
9742 039/300 Read block 234 len 28672
9743 040/300 Write block 1150 len 40960 data: 2 2 2 2 2 2 2 2 3 3
9744 041/300 Read block 3630 len 24576
9745 042/300 Read block 813 len 24576
9746 043/300 Write block 2446 len 20480 data: 5 5 5 6 5
9747 044/300 Read block 3826 len 28672
9748 045/300 Write block 4208 len 28672 data: 4 4 5 3 2 2 2
9749 046/300 Read block 209 len 4096
9750 047/300 Write block 646 len 8192 data: 3 2
9751 048/300 Read block 2298 len 28672
9752 049/300 Write block 1412 len 36864 data: 2 2 3 3 3 3 2 2 2
9753 050/300 Read block 2481 len 24576
9754 051/300 Read block 3195 len 28672
9755 052/300 Write block 2553 len 24576 data: 2 2 2 2 2 2
9756 053/300 Read block 4910 len 24576
9757 054/300 Write block 2470 len 24576 data: 4 4 4 3 4 4
9758 055/300 Read block 4314 len 4096
9759 056/300 Read block 521 len 8192
9760 057/300 Write block 1782 len 12288 data: 2 2 2
9761 058/300 Write block 4321 len 32768 data: 2 2 2 2 2 3 3 3
9762 059/300 Flush
9763 060/300 Write block 1965 len 28672 data: 3 4 4 4 4 4 4
9764 061/300 Read block 4090 len 24576
9765 062/300 Read block 4292 len 40960
9766 063/300 Read block 2108 len 8192
9767 064/300 Write block 1722 len 20480 data: 3 3 3 3 3
9768 065/300 Write block 364 len 36864 data: 2 2 4 3 3 3 3 5 5
9769 066/300 Flush
9770 067/300 Write block 2584 len 8192 data: 5 5
9771 068/300 Write block 375 len 40960 data: 4 4 3 3 2 2 2 2 2 2
9772 069/300 Read block 4501 len 36864
9773 070/300 Read block 1126 len 32768
9774 071/300 Read block 3040 len 36864
9775 072/300 Read block 3700 len 40960
9776 073/300 Read block 3495 len 16384
9777 074/300 Read block 3067 len 32768
9778 075/300 Write block 699 len 28672 data: 2 3 3 3 3 2 2
9779 076/300 Write block 3574 len 32768 data: 2 3 3 3 4 4 3 3
9780 077/300 Read block 1359 len 8192
9781 078/300 Read block 749 len 36864
9782 079/300 Write block 1360 len 16384 data: 2 3 3 2
9783 080/300 Read block 4873 len 16384
9784 081/300 Read block 1207 len 24576
9785 082/300 Read block 2598 len 40960
9786 083/300 Read block 3628 len 40960
9787 084/300 Flush
9788 085/300 Write block 78 len 40960 data: 2 2 2 2 2 3 3 5 5 5
9789 086/300 Write block 4212 len 28672 data: 3 3 3 2 2 2 3
9790 087/300 Read block 3035 len 24576
9791 088/300 Flush
9792 089/300 Read block 3944 len 12288
9793 090/300 Write block 3304 len 28672 data: 3 3 3 3 3 3 3
9794 091/300 Write block 708 len 4096 data: 2
9795 092/300 Read block 4518 len 16384
9796 093/300 Read block 1013 len 4096
9797 094/300 Write block 964 len 28672 data: 2 2 2 2 2 2 3
9798 095/300 Read block 2429 len 32768
9799 096/300 Read block 4788 len 4096
9800 097/300 Read block 3120 len 24576
9801 098/300 Flush
9802 099/300 Write block 4137 len 16384 data: 3 2 3 4
9803 100/300 Flush
9804 101/300 Flush
9805 102/300 Read block 2789 len 8192
9806 103/300 Write block 2571 len 32768 data: 3 3 3 3 3 3 4 4
9807 104/300 Write block 592 len 16384 data: 3 3 3 3
9808 105/300 Read block 752 len 40960
9809 106/300 Write block 3611 len 12288 data: 4 4 3
9810 107/300 Read block 3385 len 32768
9811 108/300 Read block 500 len 32768
9812 109/300 Read block 3773 len 16384
9813 110/300 Read block 2676 len 40960
9814 111/300 Read block 626 len 20480
9815 112/300 Write block 412 len 40960 data: 3 4 4 4 4 4 5 4 5 5
9816 113/300 Write block 2519 len 8192 data: 2 2
9817 114/300 Flush
9818 115/300 Write block 4935 len 32768 data: 3 3 3 2 3 4 4 4
9819 116/300 Read block 1299 len 28672
9820 117/300 Read block 300 len 28672
9821 118/300 Read block 3055 len 40960
9822 119/300 Read block 1708 len 12288
9823 120/300 Read block 861 len 16384
9824 121/300 Read block 4383 len 36864
9825 122/300 Flush
9826 123/300 Write block 887 len 32768 data: 3 3 3 3 2 2 2 3
9827 124/300 Read block 3260 len 16384
9828 125/300 Write block 1508 len 40960 data: 3 4 3 3 3 3 2 2 2 2
9829 126/300 Read block 2846 len 40960
9830 127/300 Write block 1523 len 36864 data: 3 3 3 3 3 2 2 2 3
9831 128/300 Write block 4265 len 4096 data: 4
9832 129/300 Read block 4403 len 36864
9833 130/300 Flush
9834 131/300 Flush
9835 132/300 Write block 3702 len 4096 data: 2
9836 133/300 Write block 1802 len 40960 data: 3 3 3 2 2 2 2 3 3 3
9837 134/300 Write block 3258 len 28672 data: 3 3 3 3 3 2 2
9838 135/300 Read block 624 len 28672
9839 136/300 Read block 2231 len 20480
9840 137/300 Read block 1706 len 4096
9841 138/300 Read block 1059 len 36864
9842 139/300 Read block 1573 len 24576
9843 140/300 Read block 943 len 24576
9844 141/300 Read block 4567 len 8192
9845 142/300 Read block 707 len 32768
9846 143/300 Read block 28 len 24576
9847 144/300 Read block 1778 len 36864
9848 145/300 Read block 4375 len 28672
9849 146/300 Read block 1251 len 40960
9850 147/300 Read block 4966 len 32768
9851 148/300 Write block 336 len 4096 data: 2
9852 149/300 Write block 3924 len 20480 data: 2 3 2 2 3
9853 150/300 Write block 567 len 8192 data: 2 2
9854 151/300 Write block 1471 len 40960 data: 2 2 3 3 3 2 2 3 3 3
9855 152/300 Write block 3173 len 24576 data: 3 3 3 3 3 4
9856 153/300 Write block 2338 len 20480 data: 4 4 4 4 4
9857 154/300 Write block 3087 len 12288 data: 3 3 4
9858 155/300 Read block 2452 len 12288
9859 156/300 Read block 363 len 16384
9860 157/300 Read block 2504 len 12288
9861 158/300 Read block 2205 len 28672
9862 159/300 Read block 2510 len 32768
9863 160/300 Write block 4427 len 36864 data: 2 2 2 2 2 2 2 2 2
9864 161/300 Read block 1922 len 20480
9865 162/300 Write block 3136 len 20480 data: 5 5 5 4 4
9866 163/300 Read block 4426 len 40960
9867 164/300 Write block 1997 len 4096 data: 4
9868 165/300 Write block 265 len 8192 data: 3 4
9869 166/300 Write block 3656 len 8192 data: 3 3
9870 167/300 Write block 4944 len 12288 data: 3 2 3
9871 168/300 Read block 2537 len 40960
9872 169/300 Read block 406 len 36864
9873 170/300 Read block 2592 len 4096
9874 171/300 Write block 1358 len 8192 data: 3 3
9875 172/300 Write block 2639 len 12288 data: 4 4 3
9876 173/300 Read block 1143 len 32768
9877 174/300 Read block 4678 len 40960
9878 175/300 Write block 4473 len 40960 data: 2 2 2 2 2 3 4 5 5 7
9879 176/300 Read block 4946 len 8192
9880 177/300 Write block 1316 len 24576 data: 3 3 3 3 3 3
9881 178/300 Write block 1017 len 8192 data: 3 3
9882 179/300 Read block 1870 len 24576
9883 180/300 Read block 1930 len 24576
9884 181/300 Write block 2681 len 36864 data: 2 2 2 2 2 2 2 3 3
9885 182/300 Write block 4376 len 12288 data: 4 3 4
9886 183/300 Read block 4655 len 4096
9887 184/300 Write block 4058 len 36864 data: 2 2 2 2 2 2 2 2 2
9888 185/300 Read block 2008 len 20480
9889 186/300 Read block 2090 len 32768
9890 187/300 Read block 3201 len 12288
9891 188/300 Write block 797 len 12288 data: 5 4 3
9892 189/300 Write block 3126 len 24576 data: 4 5 5 5 5 5
9893 190/300 Read block 1284 len 12288
9894 191/300 Write block 3578 len 36864 data: 5 5 4 4 3 3 3 3 3
9895 192/300 Read block 4662 len 24576
9896 193/300 Read block 2272 len 24576
9897 194/300 Read block 4930 len 12288
9898 195/300 Read block 3155 len 36864
9899 196/300 Flush
9900 197/300 Read block 273 len 32768
9901 198/300 Read block 888 len 12288
9902 199/300 Write block 3955 len 36864 data: 4 5 5 5 4 4 4 4 3
9903 200/300 Read block 4619 len 20480
9904 201/300 Write block 4626 len 24576 data: 2 2 2 2 2 3
9905 202/300 Flush
9906 203/300 Write block 287 len 28672 data: 4 4 5 3 3 3 3
9907 204/300 Read block 423 len 36864
9908 205/300 Flush
9909 206/300 Write block 1558 len 28672 data: 3 4 4 3 3 4 4
9910 207/300 Write block 1156 len 16384 data: 3 3 4 4
9911 208/300 Read block 1779 len 12288
9912 209/300 Write block 883 len 8192 data: 3 3
9913 210/300 Read block 742 len 16384
9914 211/300 Read block 1069 len 4096
9915 212/300 Write block 1283 len 28672 data: 4 4 5 6 6 5 5
9916 213/300 Read block 3231 len 4096
9917 214/300 Read block 1267 len 24576
9918 215/300 Read block 3606 len 20480
9919 216/300 Write block 945 len 4096 data: 6
9920 217/300 Write block 3326 len 8192 data: 3 4
9921 218/300 Read block 3312 len 32768
9922 219/300 Read block 2049 len 36864
9923 220/300 Flush
9924 221/300 Read block 25 len 40960
9925 222/300 Read block 2013 len 40960
9926 223/300 Write block 4874 len 20480 data: 3 3 3 3 2
9927 224/300 Read block 4522 len 32768
9928 225/300 Write block 3525 len 4096 data: 3
9929 226/300 Write block 4551 len 28672 data: 5 4 3 4 3 3 4
9930 227/300 Flush
9931 228/300 Read block 3100 len 16384
9932 229/300 Write block 3147 len 28672 data: 3 3 2 2 2 2 3
9933 230/300 Read block 3827 len 20480
9934 231/300 Write block 275 len 32768 data: 3 3 2 2 2 3 4 4
9935 232/300 Write block 976 len 28672 data: 3 2 2 2 2 2 3
9936 233/300 Read block 3415 len 8192
9937 234/300 Flush
9938 235/300 Write block 749 len 32768 data: 3 2 2 2 3 3 3 3
9939 236/300 Read block 2297 len 4096
9940 237/300 Write block 2366 len 12288 data: 2 2 2
9941 238/300 Read block 719 len 20480
9942 239/300 Write block 1286 len 12288 data: 7 7 6
9943 240/300 Write block 365 len 32768 data: 3 5 4 4 4 4 6 6
9944 241/300 Read block 4659 len 36864
9945 242/300 Flush
9946 243/300 Read block 3113 len 20480
9947 244/300 Write block 4012 len 24576 data: 2 3 4 4 4 4
9948 245/300 Write block 410 len 40960 data: 2 2 4 5 5 5 5 5 6 5
9949 246/300 Write block 419 len 28672 data: 6 6 6 4 4 3 3
9950 247/300 Read block 4685 len 4096
9951 248/300 Flush
9952 249/300 Write block 1458 len 12288 data: 2 2 2
9953 250/300 Read block 4677 len 20480
9954 251/300 Read block 1594 len 20480
9955 252/300 Read block 983 len 28672
9956 253/300 Write block 3938 len 4096 data: 4
9957 254/300 Read block 1056 len 32768
9958 255/300 Read block 4797 len 12288
9959 256/300 Write block 4646 len 16384 data: 5 5 4 2
9960 257/300 Read block 2917 len 36864
9961 258/300 Write block 4161 len 36864 data: 4 4 4 3 3 4 4 3 3
9962 259/300 Read block 4299 len 8192
9963 260/300 Read block 2275 len 12288
9964 261/300 Read block 1529 len 28672
9965 262/300 Read block 1683 len 12288
9966 263/300 Write block 3661 len 20480 data: 2 3 3 3 4
9967 264/300 Read block 1242 len 8192
9968 265/300 Write block 2860 len 12288 data: 3 3 2
9969 266/300 Read block 3939 len 4096
9970 267/300 Read block 3678 len 40960
9971 268/300 Flush
9972 269/300 Read block 505 len 40960
9973 270/300 Write block 4163 len 28672 data: 5 4 4 5 5 4 4
9974 271/300 Write block 2439 len 40960 data: 2 2 3 3 3 3 3 6 6 6
9975 272/300 Read block 2672 len 24576
9976 273/300 Read block 4736 len 40960
9977 274/300 Flush
9978 275/300 Read block 1622 len 24576
9979 276/300 Read block 4375 len 24576
9980 277/300 Read block 1784 len 40960
9981 278/300 Write block 1283 len 24576 data: 5 5 6 8 8 7
9982 279/300 Read block 2150 len 4096
9983 280/300 Write block 2907 len 32768 data: 2 3 3 2 2 2 2 3
9984 281/300 Write block 4247 len 24576 data: 3 3 3 2 2 2
9985 282/300 Read block 1382 len 12288
9986 283/300 Read block 4836 len 16384
9987 284/300 Read block 4859 len 24576
9988 285/300 Write block 838 len 4096 data: 2
9989 286/300 Read block 1712 len 16384
9990 287/300 Read block 2968 len 28672
9991 288/300 Read block 4969 len 36864
9992 289/300 Read block 3218 len 28672
9993 290/300 Read block 1224 len 8192
9994 291/300 Read block 3780 len 12288
9995 292/300 Read block 3546 len 12288
9996 293/300 Write block 3392 len 36864 data: 2 2 2 2 3 3 3 4 5
9997 294/300 Write block 4424 len 36864 data: 3 3 3 3 3 3 3 3 3
9998 295/300 Write block 301 len 8192 data: 3 3
9999 296/300 Read block 1122 len 36864
10000 297/300 Read block 631 len 12288
10001 298/300 Read block 4561 len 4096
10002 299/300 Read block 479 len 16384
10003 300/300 Write block 1229 len 4096 data: 3
10004 301/300 Flush
10005 Main task has work to do, go find it
10006 got dsc Start(2)
10007 start 2
10008 Mar 17 22:21:16.107 INFO request completed, latency_us: 209, response_code: 204, uri: /start/cid/2, method: POST, req_id: a0c36d6e-3f50-4d5d-a5eb-b08f4ebf0ea0, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
10009 Replay: started 2, returned:Ok(())
10010 ----------------------------------------------------------------
10011 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10012 States: Active Active Offline
10013 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10014 3114 Acked Read 10 Done Done Sent false
10015 3115 Acked Write 1 Done Done Sent false
10016 3116 Acked Write 5 Done Done Sent false
10017 3117 Acked Read 2 Done Done Sent false
10018 3118 Acked Read 1 Done Done Sent false
10019 3119 Acked Read 1 Done Done Sent false
10020 3120 Acked Write 3 Done Done Sent false
10021 3121 Acked Write 2 Done Done Sent false
10022 3122 Acked Flush 0 Done Done Sent false
10023 3123 Acked Write 4 Done Done Sent false
10024 3124 Acked Read 7 Done Done Sent false
10025 3125 Acked Flush 0 Done Done Sent false
10026 3126 Acked Write 6 Done Done Sent false
10027 3127 Acked Flush 0 Done Done Sent false
10028 3128 Acked Write 5 Done Done Sent false
10029 3129 Acked Read 5 Done Done Sent false
10030 3130 Acked Write 10 Done Done Sent false
10031 3131 Acked Read 5 Done Done Sent false
10032 3132 Acked Write 1 Done Done Sent false
10033 3133 Acked Write 4 Done Done Sent false
10034 3134 Acked Read 10 Done Done Sent false
10035 3135 Acked Write 7 Done Done Sent false
10036 3136 Acked Flush 0 Done Done Sent false
10037 3137 Acked Read 5 Done Done Sent false
10038 3138 Acked Read 3 Done Done Sent false
10039 3139 Acked Read 9 Done Done Sent false
10040 3140 Acked Write 2 Done Done Sent false
10041 3141 Acked Read 7 Done Done Sent false
10042 3142 Acked Read 9 Done Done Sent false
10043 3143 Acked Read 5 Done Done Sent false
10044 3144 Acked Write 4 Done Done Sent false
10045 3145 Acked Write 5 Done Done Sent false
10046 3146 Acked Read 7 Done Done Sent false
10047 3147 Acked Write 7 Done Done Sent false
10048 3148 Acked Read 3 Done Done Sent false
10049 3149 Acked Read 1 Done Done Sent false
10050 3150 Acked Read 4 Done Done Sent false
10051 3151 Acked Write 6 Done Done Sent false
10052 3152 Acked Read 7 Done Done Sent false
10053 3153 Acked Write 10 Done Done Sent false
10054 3154 Acked Read 6 Done Done Sent false
10055 3155 Acked Read 6 Done Done Sent false
10056 3156 Acked Write 5 Done Done Sent false
10057 3157 Acked Read 7 Done Done Sent false
10058 3158 Acked Write 7 Done Done Sent false
10059 3159 Acked Read 1 Done Done Sent false
10060 3160 Acked Write 2 Done Done Sent false
10061 3161 Acked Read 7 Done Done Sent false
10062 3162 Acked Write 9 Done Done Sent false
10063 3163 Acked Read 6 Done Done Sent false
10064 3164 Acked Read 7 Done Done Sent false
10065 3165 Acked Write 6 Done Done Sent false
10066 3166 Acked Read 6 Done Done Sent false
10067 3167 Acked Write 6 Done Done Sent false
10068 3168 Acked Read 1 Done Done Sent false
10069 3169 Acked Read 2 Done Done Sent false
10070 3170 Acked Write 3 Done Done Sent false
10071 3171 Acked Write 8 Done Done Sent false
10072 3172 Acked Flush 0 Done Done Sent false
10073 3173 Acked Write 7 Done Done Sent false
10074 3174 Acked Read 6 Done Done Sent false
10075 3175 Acked Read 10 Done Done Sent false
10076 3176 Acked Read 2 Done Done Sent false
10077 3177 Acked Write 5 Done Done Sent false
10078 3178 Acked Write 9 Done Done Sent false
10079 3179 Acked Flush 0 Done Done Sent false
10080 3180 Acked Write 2 Done Done Sent false
10081 3181 Acked Write 10 Done Done Sent false
10082 3182 Acked Read 9 Done Done Sent false
10083 3183 Acked Read 8 Done Done Sent false
10084 3184 Acked Read 9 Done Done Sent false
10085 3185 Acked Read 10 Done Done Sent false
10086 3186 Acked Read 4 Done Done Sent false
10087 3187 Acked Read 8 Done Done Sent false
10088 3188 Acked Write 7 Done Done Sent false
10089 3189 Acked Write 8 Done Done Sent false
10090 3190 Acked Read 2 Done Done Sent false
10091 3191 Acked Read 9 Done Done Sent false
10092 3192 Acked Write 4 Done Done Sent false
10093 3193 Acked Read 4 Done Done Sent false
10094 3194 Acked Read 6 Done Done Sent false
10095 3195 Acked Read 10 Done Done Sent false
10096 3196 Acked Read 10 Done Done Sent false
10097 3197 Acked Flush 0 Done Done Sent false
10098 3198 Acked Write 10 Done Done Sent false
10099 3199 Acked Write 7 Done Done Sent false
10100 3200 Acked Read 6 Done Done Sent false
10101 3201 Acked Flush 0 Done Done Sent false
10102 3202 Acked Read 3 Done Done Sent false
10103 3203 Acked Write 7 Done Done Sent false
10104 3204 Acked Write 1 Done Done Sent false
10105 3205 Acked Read 4 Done Done Sent false
10106 3206 Acked Read 1 Done Done Sent false
10107 3207 Acked Write 7 Done Done Sent false
10108 3208 Acked Read 8 Done Done Sent false
10109 3209 Acked Read 1 Done Done Sent false
10110 3210 Acked Read 6 Done Done Sent false
10111 3211 Acked Flush 0 Done Done Sent false
10112 3212 Acked Write 4 Done Done Sent false
10113 3213 Acked Flush 0 Done Done Sent false
10114 3214 Acked Flush 0 Done Done Sent false
10115 3215 Acked Read 2 Done Done Sent false
10116 3216 Acked Write 8 Done Done Sent false
10117 3217 Acked Write 4 Done Done Sent false
10118 3218 Acked Read 10 Done Done Sent false
10119 3219 Acked Write 3 Done Done Sent false
10120 3220 Acked Read 8 Done Done Sent false
10121 3221 Acked Read 8 Done Done Sent false
10122 3222 Acked Read 4 Done Done Sent false
10123 3223 Acked Read 10 Done Done Sent false
10124 3224 Acked Read 5 Done Done Sent false
10125 3225 Acked Write 10 Done Done Sent false
10126 3226 Acked Write 2 Done Done Sent false
10127 3227 Acked Flush 0 Done Done Sent false
10128 3228 Acked Write 8 Done Done Sent false
10129 3229 Acked Read 7 Done Done Sent false
10130 3230 Acked Read 7 Done Done Sent false
10131 3231 Acked Read 10 Done Done Sent false
10132 3232 Acked Read 3 Done Done Sent false
10133 3233 Acked Read 4 Done Done Sent false
10134 3234 Acked Read 9 Done Done Sent false
10135 3235 Acked Flush 0 Done Done Sent false
10136 3236 Acked Write 8 Done Done Sent false
10137 3237 Acked Read 4 Done Done Sent false
10138 3238 Acked Write 10 Done Done Sent false
10139 3239 Acked Read 10 Done Done Sent false
10140 3240 Acked Write 9 Done Done Sent false
10141 3241 Acked Write 1 Done Done Sent false
10142 3242 Acked Read 9 Done Done Sent false
10143 3243 Acked Flush 0 Done Done Sent false
10144 3244 Acked Flush 0 Done Done Sent false
10145 3245 Acked Write 1 Done Done Sent false
10146 3246 Acked Write 10 Done Done Sent false
10147 3247 Acked Write 7 Done Done Sent false
10148 3248 Acked Read 7 Done Done Sent false
10149 3249 Acked Read 5 Done Done Sent false
10150 3250 Acked Read 1 Done Done Sent false
10151 3251 Acked Read 9 Done Done Sent false
10152 3252 Acked Read 6 Done Done Sent false
10153 3253 Acked Read 6 Done Done Sent false
10154 3254 Acked Read 2 Done Done Sent false
10155 3255 Acked Read 8 Done Done Sent false
10156 3256 Acked Read 6 Done Done Sent false
10157 3257 Acked Read 9 Done Done Sent false
10158 3258 Acked Read 7 Done Done Sent false
10159 3259 Acked Read 10 Done Done Sent false
10160 3260 Acked Read 8 Done Done Sent false
10161 3261 Acked Write 1 Done Done Sent false
10162 3262 Acked Write 5 Done Done Sent false
10163 3263 Acked Write 2 Done Done Sent false
10164 3264 Acked Write 10 Done Done Sent false
10165 3265 Acked Write 6 Done Done Sent false
10166 3266 Acked Write 5 Done Done Sent false
10167 3267 Acked Write 3 Done Done Sent false
10168 3268 Acked Read 3 Done Done Sent false
10169 3269 Acked Read 4 Done Done Sent false
10170 3270 Acked Read 3 Done Done Sent false
10171 3271 Acked Read 7 Done Done Sent false
10172 3272 Acked Read 8 Done Done Sent false
10173 3273 Acked Write 9 Done Done Sent false
10174 3274 Acked Read 5 Done Done Sent false
10175 3275 Acked Write 5 Done Done Sent false
10176 3276 Acked Read 10 Done Done Sent false
10177 3277 Acked Write 1 Done Done Sent false
10178 3278 Acked Write 2 Done Done Sent false
10179 3279 Acked Write 2 Done Done Sent false
10180 3280 Acked Write 3 Done Done Sent false
10181 3281 Acked Read 10 Done Done Sent false
10182 3282 Acked Read 9 Done Done Sent false
10183 3283 Acked Read 1 Done Done Sent false
10184 3284 Acked Write 2 Done Done Sent false
10185 3285 Acked Write 3 Done Done Sent false
10186 3286 Acked Read 8 Done Done Sent false
10187 3287 Acked Read 10 Done Done Sent false
10188 3288 Acked Write 10 Done Done Sent false
10189 3289 Acked Read 2 Done Done Sent false
10190 3290 Acked Write 6 Done Done Sent false
10191 3291 Acked Write 2 Done Done Sent false
10192 3292 Acked Read 6 Done Done Sent false
10193 3293 Acked Read 6 Done Done Sent false
10194 3294 Acked Write 9 Done Done Sent false
10195 3295 Acked Write 3 Done Done Sent false
10196 3296 Acked Read 1 Done Done Sent false
10197 3297 Acked Write 9 Done Done Sent false
10198 3298 Acked Read 5 Done Done Sent false
10199 3299 Acked Read 8 Done Done Sent false
10200 3300 Acked Read 3 Done Done Sent false
10201 3301 Acked Write 3 Done Done Sent false
10202 3302 Acked Write 6 Done Done Sent false
10203 3303 Acked Read 3 Done Done Sent false
10204 3304 Acked Write 9 Done Done Sent false
10205 3305 Acked Read 6 Done Done Sent false
10206 3306 Acked Read 6 Done Done Sent false
10207 3307 Acked Read 3 Done Done Sent false
10208 3308 Acked Read 9 Done Done Sent false
10209 3309 Acked Flush 0 Done Done Sent false
10210 3310 Acked Read 8 Done Done Sent false
10211 3311 Acked Read 3 Done Done Sent false
10212 3312 Acked Write 9 Done Done Sent false
10213 3313 Acked Read 5 Done Done Sent false
10214 3314 Acked Write 6 Done Done Sent false
10215 3315 Acked Flush 0 Done Done Sent false
10216 3316 Acked Write 7 Done Done Sent false
10217 3317 Acked Read 9 Done Done Sent false
10218 3318 Acked Flush 0 Done Done Sent false
10219 3319 Acked Write 7 Done Done Sent false
10220 3320 Acked Write 4 Done Done Sent false
10221 3321 Acked Read 3 Done Done Sent false
10222 3322 Acked Write 2 Done Done Sent false
10223 3323 Acked Read 4 Done Done Sent false
10224 3324 Acked Read 1 Done Done Sent false
10225 3325 Acked Write 7 Done Done Sent false
10226 3326 Acked Read 1 Done Done Sent false
10227 3327 Acked Read 6 Done Done Sent false
10228 3328 Acked Read 5 Done Done Sent false
10229 3329 Acked Write 1 Done Done Sent false
10230 3330 Acked Write 2 Done Done Sent false
10231 3331 Acked Read 8 Done Done Sent false
10232 3332 Acked Read 9 Done Done Sent false
10233 3333 Acked Flush 0 Done Done Sent false
10234 3334 Acked Read 10 Done Done Sent false
10235 3335 Acked Read 10 Done Done Sent false
10236 3336 Acked Write 5 Done Done Sent false
10237 3337 Acked Read 8 Done Done Sent false
10238 3338 Acked Write 1 Done Done Sent false
10239 3339 Acked Write 7 Done Done Sent false
10240 3340 Acked Flush 0 Done Done Sent false
10241 3341 Acked Read 4 Done Done Sent false
10242 3342 Acked Write 7 Done Done Sent false
10243 3343 Acked Read 5 Done Done Sent false
10244 3344 Acked Write 8 Done Done Sent false
10245 3345 Acked Write 7 Done Done Sent false
10246 3346 Acked Read 2 Done Done Sent false
10247 3347 Acked Flush 0 Done Done Sent false
10248 3348 Acked Write 8 Done Done Sent false
10249 3349 Acked Read 1 Done Done Sent false
10250 3350 Acked Write 3 Done Done Sent false
10251 3351 Acked Read 5 Done Done Sent false
10252 3352 Acked Write 3 Done Done Sent false
10253 3353 Acked Write 8 Done Done Sent false
10254 3354 Acked Read 9 Done Done Sent false
10255 3355 Acked Flush 0 Done Done Sent false
10256 3356 Acked Read 5 Done Done Sent false
10257 3357 Acked Write 6 Done Done Sent false
10258 3358 Acked Write 10 Done Done Sent false
10259 3359 Acked Write 7 Done Done Sent false
10260 3360 Acked Read 1 Done Done Sent false
10261 3361 Acked Flush 0 Done Done Sent false
10262 3362 Acked Write 3 Done Done Sent false
10263 3363 Acked Read 5 Done Done Sent false
10264 3364 Acked Read 5 Done Done Sent false
10265 3365 Acked Read 7 Done Done Sent false
10266 3366 Acked Write 1 Done Done Sent false
10267 3367 Acked Read 8 Done Done Sent false
10268 3368 Acked Read 3 Done Done Sent false
10269 3369 Acked Write 4 Done Done Sent false
10270 3370 Acked Read 9 Done Done Sent false
10271 3371 Acked Write 9 Done Done Sent false
10272 3372 Acked Read 2 Done Done Sent false
10273 3373 Acked Read 3 Done Done Sent false
10274 3374 Acked Read 7 Done Done Sent false
10275 3375 Acked Read 3 Done Done Sent false
10276 3376 Acked Write 5 Done Done Sent false
10277 3377 Acked Read 2 Done Done Sent false
10278 3378 Acked Write 3 Done Done Sent false
10279 3379 Acked Read 1 Done Done Sent false
10280 3380 Acked Read 10 Done Done Sent false
10281 3381 Acked Flush 0 Done Done Sent false
10282 3382 Acked Read 10 Done Done Sent false
10283 3383 Acked Write 7 Done Done Sent false
10284 3384 Acked Write 10 Done Done Sent false
10285 3385 Acked Read 6 Done Done Sent false
10286 3386 Acked Read 10 Done Done Sent false
10287 3387 Acked Flush 0 Done Done Sent false
10288 3388 Acked Read 6 Done Done Sent false
10289 3389 Acked Read 6 Done Done Sent false
10290 3390 Acked Read 10 Done Done Sent false
10291 3391 Acked Write 6 Done Done Sent false
10292 3392 Acked Read 1 Done Done Sent false
10293 3393 Acked Write 8 Done Done Sent false
10294 3394 Acked Write 6 Done Done Sent false
10295 3395 Acked Read 3 Done Done Sent false
10296 3396 Acked Read 4 Done Done Sent false
10297 3397 Acked Read 6 Done Done Sent false
10298 3398 Acked Write 1 Done Done Sent false
10299 3399 Acked Read 4 Done Done Sent false
10300 3400 Acked Read 7 Done Done Sent false
10301 3401 Acked Read 9 Done Done Sent false
10302 3402 Acked Read 7 Done Done Sent false
10303 3403 Acked Read 2 Done Done Sent false
10304 3404 Acked Read 3 Done Done Sent false
10305 3405 Acked Read 3 Done Done Sent false
10306 3406 Acked Write 9 Done Done Sent false
10307 3407 Acked Write 9 Done Done Sent false
10308 3408 Acked Write 2 Done Done Sent false
10309 3409 Acked Read 9 Done Done Sent false
10310 3410 Acked Read 3 Done Done Sent false
10311 3411 Acked Read 1 Done Done Sent false
10312 3412 Acked Read 4 Done Done Sent false
10313 3413 Acked Write 1 Done Done Sent false
10314 3414 Acked Flush 0 Done Done Sent false
10315 STATES DS:0 DS:1 DS:2 TOTAL
10316 Sent 0 0 301 301
10317 Done 301 301 0 602
10318 Skipped 0 0 0 0
10319 Error 0 0 0 0
10320 Last Flush: Some(JobId(3414)) Some(JobId(3414)) Some(JobId(3113))
10321 Downstairs last five completed: 3113 3112 3111 3110 3109
10322 Upstairs last five completed: 3414 3413 3412 3411 3410
10323 CLIENT: Up:0 ds:301 act:2
103242025-03-17T22:21:16.778ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
103252025-03-17T22:21:17.279ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
103262025-03-17T22:21:17.780ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
103272025-03-17T22:21:18.782ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
103282025-03-17T22:21:20.097ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
10329 ----------------------------------------------------------------
10330 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10331 States: Active Active Offline
10332 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10333 3114 Acked Read 10 Done Done Sent false
10334 3115 Acked Write 1 Done Done Sent false
10335 3116 Acked Write 5 Done Done Sent false
10336 3117 Acked Read 2 Done Done Sent false
10337 3118 Acked Read 1 Done Done Sent false
10338 3119 Acked Read 1 Done Done Sent false
10339 3120 Acked Write 3 Done Done Sent false
10340 3121 Acked Write 2 Done Done Sent false
10341 3122 Acked Flush 0 Done Done Sent false
10342 3123 Acked Write 4 Done Done Sent false
10343 3124 Acked Read 7 Done Done Sent false
10344 3125 Acked Flush 0 Done Done Sent false
10345 3126 Acked Write 6 Done Done Sent false
10346 3127 Acked Flush 0 Done Done Sent false
10347 3128 Acked Write 5 Done Done Sent false
10348 3129 Acked Read 5 Done Done Sent false
10349 3130 Acked Write 10 Done Done Sent false
10350 3131 Acked Read 5 Done Done Sent false
10351 3132 Acked Write 1 Done Done Sent false
10352 3133 Acked Write 4 Done Done Sent false
10353 3134 Acked Read 10 Done Done Sent false
10354 3135 Acked Write 7 Done Done Sent false
10355 3136 Acked Flush 0 Done Done Sent false
10356 3137 Acked Read 5 Done Done Sent false
10357 3138 Acked Read 3 Done Done Sent false
10358 3139 Acked Read 9 Done Done Sent false
10359 3140 Acked Write 2 Done Done Sent false
10360 3141 Acked Read 7 Done Done Sent false
10361 3142 Acked Read 9 Done Done Sent false
10362 3143 Acked Read 5 Done Done Sent false
10363 3144 Acked Write 4 Done Done Sent false
10364 3145 Acked Write 5 Done Done Sent false
10365 3146 Acked Read 7 Done Done Sent false
10366 3147 Acked Write 7 Done Done Sent false
10367 3148 Acked Read 3 Done Done Sent false
10368 3149 Acked Read 1 Done Done Sent false
10369 3150 Acked Read 4 Done Done Sent false
10370 3151 Acked Write 6 Done Done Sent false
10371 3152 Acked Read 7 Done Done Sent false
10372 3153 Acked Write 10 Done Done Sent false
10373 3154 Acked Read 6 Done Done Sent false
10374 3155 Acked Read 6 Done Done Sent false
10375 3156 Acked Write 5 Done Done Sent false
10376 3157 Acked Read 7 Done Done Sent false
10377 3158 Acked Write 7 Done Done Sent false
10378 3159 Acked Read 1 Done Done Sent false
10379 3160 Acked Write 2 Done Done Sent false
10380 3161 Acked Read 7 Done Done Sent false
10381 3162 Acked Write 9 Done Done Sent false
10382 3163 Acked Read 6 Done Done Sent false
10383 3164 Acked Read 7 Done Done Sent false
10384 3165 Acked Write 6 Done Done Sent false
10385 3166 Acked Read 6 Done Done Sent false
10386 3167 Acked Write 6 Done Done Sent false
10387 3168 Acked Read 1 Done Done Sent false
10388 3169 Acked Read 2 Done Done Sent false
10389 3170 Acked Write 3 Done Done Sent false
10390 3171 Acked Write 8 Done Done Sent false
10391 3172 Acked Flush 0 Done Done Sent false
10392 3173 Acked Write 7 Done Done Sent false
10393 3174 Acked Read 6 Done Done Sent false
10394 3175 Acked Read 10 Done Done Sent false
10395 3176 Acked Read 2 Done Done Sent false
10396 3177 Acked Write 5 Done Done Sent false
10397 3178 Acked Write 9 Done Done Sent false
10398 3179 Acked Flush 0 Done Done Sent false
10399 3180 Acked Write 2 Done Done Sent false
10400 3181 Acked Write 10 Done Done Sent false
10401 3182 Acked Read 9 Done Done Sent false
10402 3183 Acked Read 8 Done Done Sent false
10403 3184 Acked Read 9 Done Done Sent false
10404 3185 Acked Read 10 Done Done Sent false
10405 3186 Acked Read 4 Done Done Sent false
10406 3187 Acked Read 8 Done Done Sent false
10407 3188 Acked Write 7 Done Done Sent false
10408 3189 Acked Write 8 Done Done Sent false
10409 3190 Acked Read 2 Done Done Sent false
10410 3191 Acked Read 9 Done Done Sent false
10411 3192 Acked Write 4 Done Done Sent false
10412 3193 Acked Read 4 Done Done Sent false
10413 3194 Acked Read 6 Done Done Sent false
10414 3195 Acked Read 10 Done Done Sent false
10415 3196 Acked Read 10 Done Done Sent false
10416 3197 Acked Flush 0 Done Done Sent false
10417 3198 Acked Write 10 Done Done Sent false
10418 3199 Acked Write 7 Done Done Sent false
10419 3200 Acked Read 6 Done Done Sent false
10420 3201 Acked Flush 0 Done Done Sent false
10421 3202 Acked Read 3 Done Done Sent false
10422 3203 Acked Write 7 Done Done Sent false
10423 3204 Acked Write 1 Done Done Sent false
10424 3205 Acked Read 4 Done Done Sent false
10425 3206 Acked Read 1 Done Done Sent false
10426 3207 Acked Write 7 Done Done Sent false
10427 3208 Acked Read 8 Done Done Sent false
10428 3209 Acked Read 1 Done Done Sent false
10429 3210 Acked Read 6 Done Done Sent false
10430 3211 Acked Flush 0 Done Done Sent false
10431 3212 Acked Write 4 Done Done Sent false
10432 3213 Acked Flush 0 Done Done Sent false
10433 3214 Acked Flush 0 Done Done Sent false
10434 3215 Acked Read 2 Done Done Sent false
10435 3216 Acked Write 8 Done Done Sent false
10436 3217 Acked Write 4 Done Done Sent false
10437 3218 Acked Read 10 Done Done Sent false
10438 3219 Acked Write 3 Done Done Sent false
10439 3220 Acked Read 8 Done Done Sent false
10440 3221 Acked Read 8 Done Done Sent false
10441 3222 Acked Read 4 Done Done Sent false
10442 3223 Acked Read 10 Done Done Sent false
10443 3224 Acked Read 5 Done Done Sent false
10444 3225 Acked Write 10 Done Done Sent false
10445 3226 Acked Write 2 Done Done Sent false
10446 3227 Acked Flush 0 Done Done Sent false
10447 3228 Acked Write 8 Done Done Sent false
10448 3229 Acked Read 7 Done Done Sent false
10449 3230 Acked Read 7 Done Done Sent false
10450 3231 Acked Read 10 Done Done Sent false
10451 3232 Acked Read 3 Done Done Sent false
10452 3233 Acked Read 4 Done Done Sent false
10453 3234 Acked Read 9 Done Done Sent false
10454 3235 Acked Flush 0 Done Done Sent false
10455 3236 Acked Write 8 Done Done Sent false
10456 3237 Acked Read 4 Done Done Sent false
10457 3238 Acked Write 10 Done Done Sent false
10458 3239 Acked Read 10 Done Done Sent false
10459 3240 Acked Write 9 Done Done Sent false
10460 3241 Acked Write 1 Done Done Sent false
10461 3242 Acked Read 9 Done Done Sent false
10462 3243 Acked Flush 0 Done Done Sent false
10463 3244 Acked Flush 0 Done Done Sent false
10464 3245 Acked Write 1 Done Done Sent false
10465 3246 Acked Write 10 Done Done Sent false
10466 3247 Acked Write 7 Done Done Sent false
10467 3248 Acked Read 7 Done Done Sent false
10468 3249 Acked Read 5 Done Done Sent false
10469 3250 Acked Read 1 Done Done Sent false
10470 3251 Acked Read 9 Done Done Sent false
10471 3252 Acked Read 6 Done Done Sent false
10472 3253 Acked Read 6 Done Done Sent false
10473 3254 Acked Read 2 Done Done Sent false
10474 3255 Acked Read 8 Done Done Sent false
10475 3256 Acked Read 6 Done Done Sent false
10476 3257 Acked Read 9 Done Done Sent false
10477 3258 Acked Read 7 Done Done Sent false
10478 3259 Acked Read 10 Done Done Sent false
10479 3260 Acked Read 8 Done Done Sent false
10480 3261 Acked Write 1 Done Done Sent false
10481 3262 Acked Write 5 Done Done Sent false
10482 3263 Acked Write 2 Done Done Sent false
10483 3264 Acked Write 10 Done Done Sent false
10484 3265 Acked Write 6 Done Done Sent false
10485 3266 Acked Write 5 Done Done Sent false
10486 3267 Acked Write 3 Done Done Sent false
10487 3268 Acked Read 3 Done Done Sent false
10488 3269 Acked Read 4 Done Done Sent false
10489 3270 Acked Read 3 Done Done Sent false
10490 3271 Acked Read 7 Done Done Sent false
10491 3272 Acked Read 8 Done Done Sent false
10492 3273 Acked Write 9 Done Done Sent false
10493 3274 Acked Read 5 Done Done Sent false
10494 3275 Acked Write 5 Done Done Sent false
10495 3276 Acked Read 10 Done Done Sent false
10496 3277 Acked Write 1 Done Done Sent false
10497 3278 Acked Write 2 Done Done Sent false
10498 3279 Acked Write 2 Done Done Sent false
10499 3280 Acked Write 3 Done Done Sent false
10500 3281 Acked Read 10 Done Done Sent false
10501 3282 Acked Read 9 Done Done Sent false
10502 3283 Acked Read 1 Done Done Sent false
10503 3284 Acked Write 2 Done Done Sent false
10504 3285 Acked Write 3 Done Done Sent false
10505 3286 Acked Read 8 Done Done Sent false
10506 3287 Acked Read 10 Done Done Sent false
10507 3288 Acked Write 10 Done Done Sent false
10508 3289 Acked Read 2 Done Done Sent false
10509 3290 Acked Write 6 Done Done Sent false
10510 3291 Acked Write 2 Done Done Sent false
10511 3292 Acked Read 6 Done Done Sent false
10512 3293 Acked Read 6 Done Done Sent false
10513 3294 Acked Write 9 Done Done Sent false
10514 3295 Acked Write 3 Done Done Sent false
10515 3296 Acked Read 1 Done Done Sent false
10516 3297 Acked Write 9 Done Done Sent false
10517 3298 Acked Read 5 Done Done Sent false
10518 3299 Acked Read 8 Done Done Sent false
10519 3300 Acked Read 3 Done Done Sent false
10520 3301 Acked Write 3 Done Done Sent false
10521 3302 Acked Write 6 Done Done Sent false
10522 3303 Acked Read 3 Done Done Sent false
10523 3304 Acked Write 9 Done Done Sent false
10524 3305 Acked Read 6 Done Done Sent false
10525 3306 Acked Read 6 Done Done Sent false
10526 3307 Acked Read 3 Done Done Sent false
10527 3308 Acked Read 9 Done Done Sent false
10528 3309 Acked Flush 0 Done Done Sent false
10529 3310 Acked Read 8 Done Done Sent false
10530 3311 Acked Read 3 Done Done Sent false
10531 3312 Acked Write 9 Done Done Sent false
10532 3313 Acked Read 5 Done Done Sent false
10533 3314 Acked Write 6 Done Done Sent false
10534 3315 Acked Flush 0 Done Done Sent false
10535 3316 Acked Write 7 Done Done Sent false
10536 3317 Acked Read 9 Done Done Sent false
10537 3318 Acked Flush 0 Done Done Sent false
10538 3319 Acked Write 7 Done Done Sent false
10539 3320 Acked Write 4 Done Done Sent false
10540 3321 Acked Read 3 Done Done Sent false
10541 3322 Acked Write 2 Done Done Sent false
10542 3323 Acked Read 4 Done Done Sent false
10543 3324 Acked Read 1 Done Done Sent false
10544 3325 Acked Write 7 Done Done Sent false
10545 3326 Acked Read 1 Done Done Sent false
10546 3327 Acked Read 6 Done Done Sent false
10547 3328 Acked Read 5 Done Done Sent false
10548 3329 Acked Write 1 Done Done Sent false
10549 3330 Acked Write 2 Done Done Sent false
10550 3331 Acked Read 8 Done Done Sent false
10551 3332 Acked Read 9 Done Done Sent false
10552 3333 Acked Flush 0 Done Done Sent false
10553 3334 Acked Read 10 Done Done Sent false
10554 3335 Acked Read 10 Done Done Sent false
10555 3336 Acked Write 5 Done Done Sent false
10556 3337 Acked Read 8 Done Done Sent false
10557 3338 Acked Write 1 Done Done Sent false
10558 3339 Acked Write 7 Done Done Sent false
10559 3340 Acked Flush 0 Done Done Sent false
10560 3341 Acked Read 4 Done Done Sent false
10561 3342 Acked Write 7 Done Done Sent false
10562 3343 Acked Read 5 Done Done Sent false
10563 3344 Acked Write 8 Done Done Sent false
10564 3345 Acked Write 7 Done Done Sent false
10565 3346 Acked Read 2 Done Done Sent false
10566 3347 Acked Flush 0 Done Done Sent false
10567 3348 Acked Write 8 Done Done Sent false
10568 3349 Acked Read 1 Done Done Sent false
10569 3350 Acked Write 3 Done Done Sent false
10570 3351 Acked Read 5 Done Done Sent false
10571 3352 Acked Write 3 Done Done Sent false
10572 3353 Acked Write 8 Done Done Sent false
10573 3354 Acked Read 9 Done Done Sent false
10574 3355 Acked Flush 0 Done Done Sent false
10575 3356 Acked Read 5 Done Done Sent false
10576 3357 Acked Write 6 Done Done Sent false
10577 3358 Acked Write 10 Done Done Sent false
10578 3359 Acked Write 7 Done Done Sent false
10579 3360 Acked Read 1 Done Done Sent false
10580 3361 Acked Flush 0 Done Done Sent false
10581 3362 Acked Write 3 Done Done Sent false
10582 3363 Acked Read 5 Done Done Sent false
10583 3364 Acked Read 5 Done Done Sent false
10584 3365 Acked Read 7 Done Done Sent false
10585 3366 Acked Write 1 Done Done Sent false
10586 3367 Acked Read 8 Done Done Sent false
10587 3368 Acked Read 3 Done Done Sent false
10588 3369 Acked Write 4 Done Done Sent false
10589 3370 Acked Read 9 Done Done Sent false
10590 3371 Acked Write 9 Done Done Sent false
10591 3372 Acked Read 2 Done Done Sent false
10592 3373 Acked Read 3 Done Done Sent false
10593 3374 Acked Read 7 Done Done Sent false
10594 3375 Acked Read 3 Done Done Sent false
10595 3376 Acked Write 5 Done Done Sent false
10596 3377 Acked Read 2 Done Done Sent false
10597 3378 Acked Write 3 Done Done Sent false
10598 3379 Acked Read 1 Done Done Sent false
10599 3380 Acked Read 10 Done Done Sent false
10600 3381 Acked Flush 0 Done Done Sent false
10601 3382 Acked Read 10 Done Done Sent false
10602 3383 Acked Write 7 Done Done Sent false
10603 3384 Acked Write 10 Done Done Sent false
10604 3385 Acked Read 6 Done Done Sent false
10605 3386 Acked Read 10 Done Done Sent false
10606 3387 Acked Flush 0 Done Done Sent false
10607 3388 Acked Read 6 Done Done Sent false
10608 3389 Acked Read 6 Done Done Sent false
10609 3390 Acked Read 10 Done Done Sent false
10610 3391 Acked Write 6 Done Done Sent false
10611 3392 Acked Read 1 Done Done Sent false
10612 3393 Acked Write 8 Done Done Sent false
10613 3394 Acked Write 6 Done Done Sent false
10614 3395 Acked Read 3 Done Done Sent false
10615 3396 Acked Read 4 Done Done Sent false
10616 3397 Acked Read 6 Done Done Sent false
10617 3398 Acked Write 1 Done Done Sent false
10618 3399 Acked Read 4 Done Done Sent false
10619 3400 Acked Read 7 Done Done Sent false
10620 3401 Acked Read 9 Done Done Sent false
10621 3402 Acked Read 7 Done Done Sent false
10622 3403 Acked Read 2 Done Done Sent false
10623 3404 Acked Read 3 Done Done Sent false
10624 3405 Acked Read 3 Done Done Sent false
10625 3406 Acked Write 9 Done Done Sent false
10626 3407 Acked Write 9 Done Done Sent false
10627 3408 Acked Write 2 Done Done Sent false
10628 3409 Acked Read 9 Done Done Sent false
10629 3410 Acked Read 3 Done Done Sent false
10630 3411 Acked Read 1 Done Done Sent false
10631 3412 Acked Read 4 Done Done Sent false
10632 3413 Acked Write 1 Done Done Sent false
10633 3414 Acked Flush 0 Done Done Sent false
10634 STATES DS:0 DS:1 DS:2 TOTAL
10635 Sent 0 0 301 301
10636 Done 301 301 0 602
10637 Skipped 0 0 0 0
10638 Error 0 0 0 0
10639 Last Flush: Some(JobId(3414)) Some(JobId(3414)) Some(JobId(3113))
10640 Downstairs last five completed: 3113 3112 3111 3110 3109
10641 Upstairs last five completed: 3414 3413 3412 3411 3410
10642 CLIENT: Up:0 ds:301 act:2
106432025-03-17T22:21:20.785ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
106442025-03-17T22:21:21.787ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
106452025-03-17T22:21:21.852ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
106462025-03-17T22:21:21.852ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
106472025-03-17T22:21:21.852ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
106482025-03-17T22:21:21.852ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
106492025-03-17T22:21:21.852ZINFOcrucible: client task is exiting = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
106502025-03-17T22:21:22.788ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
106512025-03-17T22:21:23.790ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
10652 [8830] got start action
10653 [8830] I am going to restart
10654 Starting downstairs at port 8830
10655 Make output file at "/tmp/dsc/downstairs-8830.txt"
10656 [8830][2] reports Starting
10657 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1219)
10658 [8830][2] reports Running
10659 ----------------------------------------------------------------
10660 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10661 States: Active Active Offline
10662 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10663 3114 Acked Read 10 Done Done Sent false
10664 3115 Acked Write 1 Done Done Sent false
10665 3116 Acked Write 5 Done Done Sent false
10666 3117 Acked Read 2 Done Done Sent false
10667 3118 Acked Read 1 Done Done Sent false
10668 3119 Acked Read 1 Done Done Sent false
10669 3120 Acked Write 3 Done Done Sent false
10670 3121 Acked Write 2 Done Done Sent false
10671 3122 Acked Flush 0 Done Done Sent false
10672 3123 Acked Write 4 Done Done Sent false
10673 3124 Acked Read 7 Done Done Sent false
10674 3125 Acked Flush 0 Done Done Sent false
10675 3126 Acked Write 6 Done Done Sent false
10676 3127 Acked Flush 0 Done Done Sent false
10677 3128 Acked Write 5 Done Done Sent false
10678 3129 Acked Read 5 Done Done Sent false
10679 3130 Acked Write 10 Done Done Sent false
10680 3131 Acked Read 5 Done Done Sent false
10681 3132 Acked Write 1 Done Done Sent false
10682 3133 Acked Write 4 Done Done Sent false
10683 3134 Acked Read 10 Done Done Sent false
10684 3135 Acked Write 7 Done Done Sent false
10685 3136 Acked Flush 0 Done Done Sent false
10686 3137 Acked Read 5 Done Done Sent false
10687 3138 Acked Read 3 Done Done Sent false
10688 3139 Acked Read 9 Done Done Sent false
10689 3140 Acked Write 2 Done Done Sent false
10690 3141 Acked Read 7 Done Done Sent false
10691 3142 Acked Read 9 Done Done Sent false
10692 3143 Acked Read 5 Done Done Sent false
10693 3144 Acked Write 4 Done Done Sent false
10694 3145 Acked Write 5 Done Done Sent false
10695 3146 Acked Read 7 Done Done Sent false
10696 3147 Acked Write 7 Done Done Sent false
10697 3148 Acked Read 3 Done Done Sent false
10698 3149 Acked Read 1 Done Done Sent false
10699 3150 Acked Read 4 Done Done Sent false
10700 3151 Acked Write 6 Done Done Sent false
10701 3152 Acked Read 7 Done Done Sent false
10702 3153 Acked Write 10 Done Done Sent false
10703 3154 Acked Read 6 Done Done Sent false
10704 3155 Acked Read 6 Done Done Sent false
10705 3156 Acked Write 5 Done Done Sent false
10706 3157 Acked Read 7 Done Done Sent false
10707 3158 Acked Write 7 Done Done Sent false
10708 3159 Acked Read 1 Done Done Sent false
10709 3160 Acked Write 2 Done Done Sent false
10710 3161 Acked Read 7 Done Done Sent false
10711 3162 Acked Write 9 Done Done Sent false
10712 3163 Acked Read 6 Done Done Sent false
10713 3164 Acked Read 7 Done Done Sent false
10714 3165 Acked Write 6 Done Done Sent false
10715 3166 Acked Read 6 Done Done Sent false
10716 3167 Acked Write 6 Done Done Sent false
10717 3168 Acked Read 1 Done Done Sent false
10718 3169 Acked Read 2 Done Done Sent false
10719 3170 Acked Write 3 Done Done Sent false
10720 3171 Acked Write 8 Done Done Sent false
10721 3172 Acked Flush 0 Done Done Sent false
10722 3173 Acked Write 7 Done Done Sent false
10723 3174 Acked Read 6 Done Done Sent false
10724 3175 Acked Read 10 Done Done Sent false
10725 3176 Acked Read 2 Done Done Sent false
10726 3177 Acked Write 5 Done Done Sent false
10727 3178 Acked Write 9 Done Done Sent false
10728 3179 Acked Flush 0 Done Done Sent false
10729 3180 Acked Write 2 Done Done Sent false
10730 3181 Acked Write 10 Done Done Sent false
10731 3182 Acked Read 9 Done Done Sent false
10732 3183 Acked Read 8 Done Done Sent false
10733 3184 Acked Read 9 Done Done Sent false
10734 3185 Acked Read 10 Done Done Sent false
10735 3186 Acked Read 4 Done Done Sent false
10736 3187 Acked Read 8 Done Done Sent false
10737 3188 Acked Write 7 Done Done Sent false
10738 3189 Acked Write 8 Done Done Sent false
10739 3190 Acked Read 2 Done Done Sent false
10740 3191 Acked Read 9 Done Done Sent false
10741 3192 Acked Write 4 Done Done Sent false
10742 3193 Acked Read 4 Done Done Sent false
10743 3194 Acked Read 6 Done Done Sent false
10744 3195 Acked Read 10 Done Done Sent false
10745 3196 Acked Read 10 Done Done Sent false
10746 3197 Acked Flush 0 Done Done Sent false
10747 3198 Acked Write 10 Done Done Sent false
10748 3199 Acked Write 7 Done Done Sent false
10749 3200 Acked Read 6 Done Done Sent false
10750 3201 Acked Flush 0 Done Done Sent false
10751 3202 Acked Read 3 Done Done Sent false
10752 3203 Acked Write 7 Done Done Sent false
10753 3204 Acked Write 1 Done Done Sent false
10754 3205 Acked Read 4 Done Done Sent false
10755 3206 Acked Read 1 Done Done Sent false
10756 3207 Acked Write 7 Done Done Sent false
10757 3208 Acked Read 8 Done Done Sent false
10758 3209 Acked Read 1 Done Done Sent false
10759 3210 Acked Read 6 Done Done Sent false
10760 3211 Acked Flush 0 Done Done Sent false
10761 3212 Acked Write 4 Done Done Sent false
10762 3213 Acked Flush 0 Done Done Sent false
10763 3214 Acked Flush 0 Done Done Sent false
10764 3215 Acked Read 2 Done Done Sent false
10765 3216 Acked Write 8 Done Done Sent false
10766 3217 Acked Write 4 Done Done Sent false
10767 3218 Acked Read 10 Done Done Sent false
10768 3219 Acked Write 3 Done Done Sent false
10769 3220 Acked Read 8 Done Done Sent false
10770 3221 Acked Read 8 Done Done Sent false
10771 3222 Acked Read 4 Done Done Sent false
10772 3223 Acked Read 10 Done Done Sent false
10773 3224 Acked Read 5 Done Done Sent false
10774 3225 Acked Write 10 Done Done Sent false
10775 3226 Acked Write 2 Done Done Sent false
10776 3227 Acked Flush 0 Done Done Sent false
10777 3228 Acked Write 8 Done Done Sent false
10778 3229 Acked Read 7 Done Done Sent false
10779 3230 Acked Read 7 Done Done Sent false
10780 3231 Acked Read 10 Done Done Sent false
10781 3232 Acked Read 3 Done Done Sent false
10782 3233 Acked Read 4 Done Done Sent false
10783 3234 Acked Read 9 Done Done Sent false
10784 3235 Acked Flush 0 Done Done Sent false
10785 3236 Acked Write 8 Done Done Sent false
10786 3237 Acked Read 4 Done Done Sent false
10787 3238 Acked Write 10 Done Done Sent false
10788 3239 Acked Read 10 Done Done Sent false
10789 3240 Acked Write 9 Done Done Sent false
10790 3241 Acked Write 1 Done Done Sent false
10791 3242 Acked Read 9 Done Done Sent false
10792 3243 Acked Flush 0 Done Done Sent false
10793 3244 Acked Flush 0 Done Done Sent false
10794 3245 Acked Write 1 Done Done Sent false
10795 3246 Acked Write 10 Done Done Sent false
10796 3247 Acked Write 7 Done Done Sent false
10797 3248 Acked Read 7 Done Done Sent false
10798 3249 Acked Read 5 Done Done Sent false
10799 3250 Acked Read 1 Done Done Sent false
10800 3251 Acked Read 9 Done Done Sent false
10801 3252 Acked Read 6 Done Done Sent false
10802 3253 Acked Read 6 Done Done Sent false
10803 3254 Acked Read 2 Done Done Sent false
10804 3255 Acked Read 8 Done Done Sent false
10805 3256 Acked Read 6 Done Done Sent false
10806 3257 Acked Read 9 Done Done Sent false
10807 3258 Acked Read 7 Done Done Sent false
10808 3259 Acked Read 10 Done Done Sent false
10809 3260 Acked Read 8 Done Done Sent false
10810 3261 Acked Write 1 Done Done Sent false
10811 3262 Acked Write 5 Done Done Sent false
10812 3263 Acked Write 2 Done Done Sent false
10813 3264 Acked Write 10 Done Done Sent false
10814 3265 Acked Write 6 Done Done Sent false
10815 3266 Acked Write 5 Done Done Sent false
10816 3267 Acked Write 3 Done Done Sent false
10817 3268 Acked Read 3 Done Done Sent false
10818 3269 Acked Read 4 Done Done Sent false
10819 3270 Acked Read 3 Done Done Sent false
10820 3271 Acked Read 7 Done Done Sent false
10821 3272 Acked Read 8 Done Done Sent false
10822 3273 Acked Write 9 Done Done Sent false
10823 3274 Acked Read 5 Done Done Sent false
10824 3275 Acked Write 5 Done Done Sent false
10825 3276 Acked Read 10 Done Done Sent false
10826 3277 Acked Write 1 Done Done Sent false
10827 3278 Acked Write 2 Done Done Sent false
10828 3279 Acked Write 2 Done Done Sent false
10829 3280 Acked Write 3 Done Done Sent false
10830 3281 Acked Read 10 Done Done Sent false
10831 3282 Acked Read 9 Done Done Sent false
10832 3283 Acked Read 1 Done Done Sent false
10833 3284 Acked Write 2 Done Done Sent false
10834 3285 Acked Write 3 Done Done Sent false
10835 3286 Acked Read 8 Done Done Sent false
10836 3287 Acked Read 10 Done Done Sent false
10837 3288 Acked Write 10 Done Done Sent false
10838 3289 Acked Read 2 Done Done Sent false
10839 3290 Acked Write 6 Done Done Sent false
10840 3291 Acked Write 2 Done Done Sent false
10841 3292 Acked Read 6 Done Done Sent false
10842 3293 Acked Read 6 Done Done Sent false
10843 3294 Acked Write 9 Done Done Sent false
10844 3295 Acked Write 3 Done Done Sent false
10845 3296 Acked Read 1 Done Done Sent false
10846 3297 Acked Write 9 Done Done Sent false
10847 3298 Acked Read 5 Done Done Sent false
10848 3299 Acked Read 8 Done Done Sent false
10849 3300 Acked Read 3 Done Done Sent false
10850 3301 Acked Write 3 Done Done Sent false
10851 3302 Acked Write 6 Done Done Sent false
10852 3303 Acked Read 3 Done Done Sent false
10853 3304 Acked Write 9 Done Done Sent false
10854 3305 Acked Read 6 Done Done Sent false
10855 3306 Acked Read 6 Done Done Sent false
10856 3307 Acked Read 3 Done Done Sent false
10857 3308 Acked Read 9 Done Done Sent false
10858 3309 Acked Flush 0 Done Done Sent false
10859 3310 Acked Read 8 Done Done Sent false
10860 3311 Acked Read 3 Done Done Sent false
10861 3312 Acked Write 9 Done Done Sent false
10862 3313 Acked Read 5 Done Done Sent false
10863 3314 Acked Write 6 Done Done Sent false
10864 3315 Acked Flush 0 Done Done Sent false
10865 3316 Acked Write 7 Done Done Sent false
10866 3317 Acked Read 9 Done Done Sent false
10867 3318 Acked Flush 0 Done Done Sent false
10868 3319 Acked Write 7 Done Done Sent false
10869 3320 Acked Write 4 Done Done Sent false
10870 3321 Acked Read 3 Done Done Sent false
10871 3322 Acked Write 2 Done Done Sent false
10872 3323 Acked Read 4 Done Done Sent false
10873 3324 Acked Read 1 Done Done Sent false
10874 3325 Acked Write 7 Done Done Sent false
10875 3326 Acked Read 1 Done Done Sent false
10876 3327 Acked Read 6 Done Done Sent false
10877 3328 Acked Read 5 Done Done Sent false
10878 3329 Acked Write 1 Done Done Sent false
10879 3330 Acked Write 2 Done Done Sent false
10880 3331 Acked Read 8 Done Done Sent false
10881 3332 Acked Read 9 Done Done Sent false
10882 3333 Acked Flush 0 Done Done Sent false
10883 3334 Acked Read 10 Done Done Sent false
10884 3335 Acked Read 10 Done Done Sent false
10885 3336 Acked Write 5 Done Done Sent false
10886 3337 Acked Read 8 Done Done Sent false
10887 3338 Acked Write 1 Done Done Sent false
10888 3339 Acked Write 7 Done Done Sent false
10889 3340 Acked Flush 0 Done Done Sent false
10890 3341 Acked Read 4 Done Done Sent false
10891 3342 Acked Write 7 Done Done Sent false
10892 3343 Acked Read 5 Done Done Sent false
10893 3344 Acked Write 8 Done Done Sent false
10894 3345 Acked Write 7 Done Done Sent false
10895 3346 Acked Read 2 Done Done Sent false
10896 3347 Acked Flush 0 Done Done Sent false
10897 3348 Acked Write 8 Done Done Sent false
10898 3349 Acked Read 1 Done Done Sent false
10899 3350 Acked Write 3 Done Done Sent false
10900 3351 Acked Read 5 Done Done Sent false
10901 3352 Acked Write 3 Done Done Sent false
10902 3353 Acked Write 8 Done Done Sent false
10903 3354 Acked Read 9 Done Done Sent false
10904 3355 Acked Flush 0 Done Done Sent false
10905 3356 Acked Read 5 Done Done Sent false
10906 3357 Acked Write 6 Done Done Sent false
10907 3358 Acked Write 10 Done Done Sent false
10908 3359 Acked Write 7 Done Done Sent false
10909 3360 Acked Read 1 Done Done Sent false
10910 3361 Acked Flush 0 Done Done Sent false
10911 3362 Acked Write 3 Done Done Sent false
10912 3363 Acked Read 5 Done Done Sent false
10913 3364 Acked Read 5 Done Done Sent false
10914 3365 Acked Read 7 Done Done Sent false
10915 3366 Acked Write 1 Done Done Sent false
10916 3367 Acked Read 8 Done Done Sent false
10917 3368 Acked Read 3 Done Done Sent false
10918 3369 Acked Write 4 Done Done Sent false
10919 3370 Acked Read 9 Done Done Sent false
10920 3371 Acked Write 9 Done Done Sent false
10921 3372 Acked Read 2 Done Done Sent false
10922 3373 Acked Read 3 Done Done Sent false
10923 3374 Acked Read 7 Done Done Sent false
10924 3375 Acked Read 3 Done Done Sent false
10925 3376 Acked Write 5 Done Done Sent false
10926 3377 Acked Read 2 Done Done Sent false
10927 3378 Acked Write 3 Done Done Sent false
10928 3379 Acked Read 1 Done Done Sent false
10929 3380 Acked Read 10 Done Done Sent false
10930 3381 Acked Flush 0 Done Done Sent false
10931 3382 Acked Read 10 Done Done Sent false
10932 3383 Acked Write 7 Done Done Sent false
10933 3384 Acked Write 10 Done Done Sent false
10934 3385 Acked Read 6 Done Done Sent false
10935 3386 Acked Read 10 Done Done Sent false
10936 3387 Acked Flush 0 Done Done Sent false
10937 3388 Acked Read 6 Done Done Sent false
10938 3389 Acked Read 6 Done Done Sent false
10939 3390 Acked Read 10 Done Done Sent false
10940 3391 Acked Write 6 Done Done Sent false
10941 3392 Acked Read 1 Done Done Sent false
10942 3393 Acked Write 8 Done Done Sent false
10943 3394 Acked Write 6 Done Done Sent false
10944 3395 Acked Read 3 Done Done Sent false
10945 3396 Acked Read 4 Done Done Sent false
10946 3397 Acked Read 6 Done Done Sent false
10947 3398 Acked Write 1 Done Done Sent false
10948 3399 Acked Read 4 Done Done Sent false
10949 3400 Acked Read 7 Done Done Sent false
10950 3401 Acked Read 9 Done Done Sent false
10951 3402 Acked Read 7 Done Done Sent false
10952 3403 Acked Read 2 Done Done Sent false
10953 3404 Acked Read 3 Done Done Sent false
10954 3405 Acked Read 3 Done Done Sent false
10955 3406 Acked Write 9 Done Done Sent false
10956 3407 Acked Write 9 Done Done Sent false
10957 3408 Acked Write 2 Done Done Sent false
10958 3409 Acked Read 9 Done Done Sent false
10959 3410 Acked Read 3 Done Done Sent false
10960 3411 Acked Read 1 Done Done Sent false
10961 3412 Acked Read 4 Done Done Sent false
10962 3413 Acked Write 1 Done Done Sent false
10963 3414 Acked Flush 0 Done Done Sent false
10964 STATES DS:0 DS:1 DS:2 TOTAL
10965 Sent 0 0 301 301
10966 Done 301 301 0 602
10967 Skipped 0 0 0 0
10968 Error 0 0 0 0
10969 Last Flush: Some(JobId(3414)) Some(JobId(3414)) Some(JobId(3113))
10970 Downstairs last five completed: 3113 3112 3111 3110 3109
10971 Upstairs last five completed: 3414 3413 3412 3411 3410
10972 CLIENT: Up:0 ds:301 act:2
109732025-03-17T22:21:25.098ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
109742025-03-17T22:21:25.793ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
109752025-03-17T22:21:26.795ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
109762025-03-17T22:21:27.797ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
10977 ----------------------------------------------------------------
10978 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10979 States: Active Active Offline
10980 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10981 3114 Acked Read 10 Done Done Sent false
10982 3115 Acked Write 1 Done Done Sent false
10983 3116 Acked Write 5 Done Done Sent false
10984 3117 Acked Read 2 Done Done Sent false
10985 3118 Acked Read 1 Done Done Sent false
10986 3119 Acked Read 1 Done Done Sent false
10987 3120 Acked Write 3 Done Done Sent false
10988 3121 Acked Write 2 Done Done Sent false
10989 3122 Acked Flush 0 Done Done Sent false
10990 3123 Acked Write 4 Done Done Sent false
10991 3124 Acked Read 7 Done Done Sent false
10992 3125 Acked Flush 0 Done Done Sent false
10993 3126 Acked Write 6 Done Done Sent false
10994 3127 Acked Flush 0 Done Done Sent false
10995 3128 Acked Write 5 Done Done Sent false
10996 3129 Acked Read 5 Done Done Sent false
10997 3130 Acked Write 10 Done Done Sent false
10998 3131 Acked Read 5 Done Done Sent false
10999 3132 Acked Write 1 Done Done Sent false
11000 3133 Acked Write 4 Done Done Sent false
11001 3134 Acked Read 10 Done Done Sent false
11002 3135 Acked Write 7 Done Done Sent false
11003 3136 Acked Flush 0 Done Done Sent false
11004 3137 Acked Read 5 Done Done Sent false
11005 3138 Acked Read 3 Done Done Sent false
11006 3139 Acked Read 9 Done Done Sent false
11007 3140 Acked Write 2 Done Done Sent false
11008 3141 Acked Read 7 Done Done Sent false
11009 3142 Acked Read 9 Done Done Sent false
11010 3143 Acked Read 5 Done Done Sent false
11011 3144 Acked Write 4 Done Done Sent false
11012 3145 Acked Write 5 Done Done Sent false
11013 3146 Acked Read 7 Done Done Sent false
11014 3147 Acked Write 7 Done Done Sent false
11015 3148 Acked Read 3 Done Done Sent false
11016 3149 Acked Read 1 Done Done Sent false
11017 3150 Acked Read 4 Done Done Sent false
11018 3151 Acked Write 6 Done Done Sent false
11019 3152 Acked Read 7 Done Done Sent false
11020 3153 Acked Write 10 Done Done Sent false
11021 3154 Acked Read 6 Done Done Sent false
11022 3155 Acked Read 6 Done Done Sent false
11023 3156 Acked Write 5 Done Done Sent false
11024 3157 Acked Read 7 Done Done Sent false
11025 3158 Acked Write 7 Done Done Sent false
11026 3159 Acked Read 1 Done Done Sent false
11027 3160 Acked Write 2 Done Done Sent false
11028 3161 Acked Read 7 Done Done Sent false
11029 3162 Acked Write 9 Done Done Sent false
11030 3163 Acked Read 6 Done Done Sent false
11031 3164 Acked Read 7 Done Done Sent false
11032 3165 Acked Write 6 Done Done Sent false
11033 3166 Acked Read 6 Done Done Sent false
11034 3167 Acked Write 6 Done Done Sent false
11035 3168 Acked Read 1 Done Done Sent false
11036 3169 Acked Read 2 Done Done Sent false
11037 3170 Acked Write 3 Done Done Sent false
11038 3171 Acked Write 8 Done Done Sent false
11039 3172 Acked Flush 0 Done Done Sent false
11040 3173 Acked Write 7 Done Done Sent false
11041 3174 Acked Read 6 Done Done Sent false
11042 3175 Acked Read 10 Done Done Sent false
11043 3176 Acked Read 2 Done Done Sent false
11044 3177 Acked Write 5 Done Done Sent false
11045 3178 Acked Write 9 Done Done Sent false
11046 3179 Acked Flush 0 Done Done Sent false
11047 3180 Acked Write 2 Done Done Sent false
11048 3181 Acked Write 10 Done Done Sent false
11049 3182 Acked Read 9 Done Done Sent false
11050 3183 Acked Read 8 Done Done Sent false
11051 3184 Acked Read 9 Done Done Sent false
11052 3185 Acked Read 10 Done Done Sent false
11053 3186 Acked Read 4 Done Done Sent false
11054 3187 Acked Read 8 Done Done Sent false
11055 3188 Acked Write 7 Done Done Sent false
11056 3189 Acked Write 8 Done Done Sent false
11057 3190 Acked Read 2 Done Done Sent false
11058 3191 Acked Read 9 Done Done Sent false
11059 3192 Acked Write 4 Done Done Sent false
11060 3193 Acked Read 4 Done Done Sent false
11061 3194 Acked Read 6 Done Done Sent false
11062 3195 Acked Read 10 Done Done Sent false
11063 3196 Acked Read 10 Done Done Sent false
11064 3197 Acked Flush 0 Done Done Sent false
11065 3198 Acked Write 10 Done Done Sent false
11066 3199 Acked Write 7 Done Done Sent false
11067 3200 Acked Read 6 Done Done Sent false
11068 3201 Acked Flush 0 Done Done Sent false
11069 3202 Acked Read 3 Done Done Sent false
11070 3203 Acked Write 7 Done Done Sent false
11071 3204 Acked Write 1 Done Done Sent false
11072 3205 Acked Read 4 Done Done Sent false
11073 3206 Acked Read 1 Done Done Sent false
11074 3207 Acked Write 7 Done Done Sent false
11075 3208 Acked Read 8 Done Done Sent false
11076 3209 Acked Read 1 Done Done Sent false
11077 3210 Acked Read 6 Done Done Sent false
11078 3211 Acked Flush 0 Done Done Sent false
11079 3212 Acked Write 4 Done Done Sent false
11080 3213 Acked Flush 0 Done Done Sent false
11081 3214 Acked Flush 0 Done Done Sent false
11082 3215 Acked Read 2 Done Done Sent false
11083 3216 Acked Write 8 Done Done Sent false
11084 3217 Acked Write 4 Done Done Sent false
11085 3218 Acked Read 10 Done Done Sent false
11086 3219 Acked Write 3 Done Done Sent false
11087 3220 Acked Read 8 Done Done Sent false
11088 3221 Acked Read 8 Done Done Sent false
11089 3222 Acked Read 4 Done Done Sent false
11090 3223 Acked Read 10 Done Done Sent false
11091 3224 Acked Read 5 Done Done Sent false
11092 3225 Acked Write 10 Done Done Sent false
11093 3226 Acked Write 2 Done Done Sent false
11094 3227 Acked Flush 0 Done Done Sent false
11095 3228 Acked Write 8 Done Done Sent false
11096 3229 Acked Read 7 Done Done Sent false
11097 3230 Acked Read 7 Done Done Sent false
11098 3231 Acked Read 10 Done Done Sent false
11099 3232 Acked Read 3 Done Done Sent false
11100 3233 Acked Read 4 Done Done Sent false
11101 3234 Acked Read 9 Done Done Sent false
11102 3235 Acked Flush 0 Done Done Sent false
11103 3236 Acked Write 8 Done Done Sent false
11104 3237 Acked Read 4 Done Done Sent false
11105 3238 Acked Write 10 Done Done Sent false
11106 3239 Acked Read 10 Done Done Sent false
11107 3240 Acked Write 9 Done Done Sent false
11108 3241 Acked Write 1 Done Done Sent false
11109 3242 Acked Read 9 Done Done Sent false
11110 3243 Acked Flush 0 Done Done Sent false
11111 3244 Acked Flush 0 Done Done Sent false
11112 3245 Acked Write 1 Done Done Sent false
11113 3246 Acked Write 10 Done Done Sent false
11114 3247 Acked Write 7 Done Done Sent false
11115 3248 Acked Read 7 Done Done Sent false
11116 3249 Acked Read 5 Done Done Sent false
11117 3250 Acked Read 1 Done Done Sent false
11118 3251 Acked Read 9 Done Done Sent false
11119 3252 Acked Read 6 Done Done Sent false
11120 3253 Acked Read 6 Done Done Sent false
11121 3254 Acked Read 2 Done Done Sent false
11122 3255 Acked Read 8 Done Done Sent false
11123 3256 Acked Read 6 Done Done Sent false
11124 3257 Acked Read 9 Done Done Sent false
11125 3258 Acked Read 7 Done Done Sent false
11126 3259 Acked Read 10 Done Done Sent false
11127 3260 Acked Read 8 Done Done Sent false
11128 3261 Acked Write 1 Done Done Sent false
11129 3262 Acked Write 5 Done Done Sent false
11130 3263 Acked Write 2 Done Done Sent false
11131 3264 Acked Write 10 Done Done Sent false
11132 3265 Acked Write 6 Done Done Sent false
11133 3266 Acked Write 5 Done Done Sent false
11134 3267 Acked Write 3 Done Done Sent false
11135 3268 Acked Read 3 Done Done Sent false
11136 3269 Acked Read 4 Done Done Sent false
11137 3270 Acked Read 3 Done Done Sent false
11138 3271 Acked Read 7 Done Done Sent false
11139 3272 Acked Read 8 Done Done Sent false
11140 3273 Acked Write 9 Done Done Sent false
11141 3274 Acked Read 5 Done Done Sent false
11142 3275 Acked Write 5 Done Done Sent false
11143 3276 Acked Read 10 Done Done Sent false
11144 3277 Acked Write 1 Done Done Sent false
11145 3278 Acked Write 2 Done Done Sent false
11146 3279 Acked Write 2 Done Done Sent false
11147 3280 Acked Write 3 Done Done Sent false
11148 3281 Acked Read 10 Done Done Sent false
11149 3282 Acked Read 9 Done Done Sent false
11150 3283 Acked Read 1 Done Done Sent false
11151 3284 Acked Write 2 Done Done Sent false
11152 3285 Acked Write 3 Done Done Sent false
11153 3286 Acked Read 8 Done Done Sent false
11154 3287 Acked Read 10 Done Done Sent false
11155 3288 Acked Write 10 Done Done Sent false
11156 3289 Acked Read 2 Done Done Sent false
11157 3290 Acked Write 6 Done Done Sent false
11158 3291 Acked Write 2 Done Done Sent false
11159 3292 Acked Read 6 Done Done Sent false
11160 3293 Acked Read 6 Done Done Sent false
11161 3294 Acked Write 9 Done Done Sent false
11162 3295 Acked Write 3 Done Done Sent false
11163 3296 Acked Read 1 Done Done Sent false
11164 3297 Acked Write 9 Done Done Sent false
11165 3298 Acked Read 5 Done Done Sent false
11166 3299 Acked Read 8 Done Done Sent false
11167 3300 Acked Read 3 Done Done Sent false
11168 3301 Acked Write 3 Done Done Sent false
11169 3302 Acked Write 6 Done Done Sent false
11170 3303 Acked Read 3 Done Done Sent false
11171 3304 Acked Write 9 Done Done Sent false
11172 3305 Acked Read 6 Done Done Sent false
11173 3306 Acked Read 6 Done Done Sent false
11174 3307 Acked Read 3 Done Done Sent false
11175 3308 Acked Read 9 Done Done Sent false
11176 3309 Acked Flush 0 Done Done Sent false
11177 3310 Acked Read 8 Done Done Sent false
11178 3311 Acked Read 3 Done Done Sent false
11179 3312 Acked Write 9 Done Done Sent false
11180 3313 Acked Read 5 Done Done Sent false
11181 3314 Acked Write 6 Done Done Sent false
11182 3315 Acked Flush 0 Done Done Sent false
11183 3316 Acked Write 7 Done Done Sent false
11184 3317 Acked Read 9 Done Done Sent false
11185 3318 Acked Flush 0 Done Done Sent false
11186 3319 Acked Write 7 Done Done Sent false
11187 3320 Acked Write 4 Done Done Sent false
11188 3321 Acked Read 3 Done Done Sent false
11189 3322 Acked Write 2 Done Done Sent false
11190 3323 Acked Read 4 Done Done Sent false
11191 3324 Acked Read 1 Done Done Sent false
11192 3325 Acked Write 7 Done Done Sent false
11193 3326 Acked Read 1 Done Done Sent false
11194 3327 Acked Read 6 Done Done Sent false
11195 3328 Acked Read 5 Done Done Sent false
11196 3329 Acked Write 1 Done Done Sent false
11197 3330 Acked Write 2 Done Done Sent false
11198 3331 Acked Read 8 Done Done Sent false
11199 3332 Acked Read 9 Done Done Sent false
11200 3333 Acked Flush 0 Done Done Sent false
11201 3334 Acked Read 10 Done Done Sent false
11202 3335 Acked Read 10 Done Done Sent false
11203 3336 Acked Write 5 Done Done Sent false
11204 3337 Acked Read 8 Done Done Sent false
11205 3338 Acked Write 1 Done Done Sent false
11206 3339 Acked Write 7 Done Done Sent false
11207 3340 Acked Flush 0 Done Done Sent false
11208 3341 Acked Read 4 Done Done Sent false
11209 3342 Acked Write 7 Done Done Sent false
11210 3343 Acked Read 5 Done Done Sent false
11211 3344 Acked Write 8 Done Done Sent false
11212 3345 Acked Write 7 Done Done Sent false
11213 3346 Acked Read 2 Done Done Sent false
11214 3347 Acked Flush 0 Done Done Sent false
11215 3348 Acked Write 8 Done Done Sent false
11216 3349 Acked Read 1 Done Done Sent false
11217 3350 Acked Write 3 Done Done Sent false
11218 3351 Acked Read 5 Done Done Sent false
11219 3352 Acked Write 3 Done Done Sent false
11220 3353 Acked Write 8 Done Done Sent false
11221 3354 Acked Read 9 Done Done Sent false
11222 3355 Acked Flush 0 Done Done Sent false
11223 3356 Acked Read 5 Done Done Sent false
11224 3357 Acked Write 6 Done Done Sent false
11225 3358 Acked Write 10 Done Done Sent false
11226 3359 Acked Write 7 Done Done Sent false
11227 3360 Acked Read 1 Done Done Sent false
11228 3361 Acked Flush 0 Done Done Sent false
11229 3362 Acked Write 3 Done Done Sent false
11230 3363 Acked Read 5 Done Done Sent false
11231 3364 Acked Read 5 Done Done Sent false
11232 3365 Acked Read 7 Done Done Sent false
11233 3366 Acked Write 1 Done Done Sent false
11234 3367 Acked Read 8 Done Done Sent false
11235 3368 Acked Read 3 Done Done Sent false
11236 3369 Acked Write 4 Done Done Sent false
11237 3370 Acked Read 9 Done Done Sent false
11238 3371 Acked Write 9 Done Done Sent false
11239 3372 Acked Read 2 Done Done Sent false
11240 3373 Acked Read 3 Done Done Sent false
11241 3374 Acked Read 7 Done Done Sent false
11242 3375 Acked Read 3 Done Done Sent false
11243 3376 Acked Write 5 Done Done Sent false
11244 3377 Acked Read 2 Done Done Sent false
11245 3378 Acked Write 3 Done Done Sent false
11246 3379 Acked Read 1 Done Done Sent false
11247 3380 Acked Read 10 Done Done Sent false
11248 3381 Acked Flush 0 Done Done Sent false
11249 3382 Acked Read 10 Done Done Sent false
11250 3383 Acked Write 7 Done Done Sent false
11251 3384 Acked Write 10 Done Done Sent false
11252 3385 Acked Read 6 Done Done Sent false
11253 3386 Acked Read 10 Done Done Sent false
11254 3387 Acked Flush 0 Done Done Sent false
11255 3388 Acked Read 6 Done Done Sent false
11256 3389 Acked Read 6 Done Done Sent false
11257 3390 Acked Read 10 Done Done Sent false
11258 3391 Acked Write 6 Done Done Sent false
11259 3392 Acked Read 1 Done Done Sent false
11260 3393 Acked Write 8 Done Done Sent false
11261 3394 Acked Write 6 Done Done Sent false
11262 3395 Acked Read 3 Done Done Sent false
11263 3396 Acked Read 4 Done Done Sent false
11264 3397 Acked Read 6 Done Done Sent false
11265 3398 Acked Write 1 Done Done Sent false
11266 3399 Acked Read 4 Done Done Sent false
11267 3400 Acked Read 7 Done Done Sent false
11268 3401 Acked Read 9 Done Done Sent false
11269 3402 Acked Read 7 Done Done Sent false
11270 3403 Acked Read 2 Done Done Sent false
11271 3404 Acked Read 3 Done Done Sent false
11272 3405 Acked Read 3 Done Done Sent false
11273 3406 Acked Write 9 Done Done Sent false
11274 3407 Acked Write 9 Done Done Sent false
11275 3408 Acked Write 2 Done Done Sent false
11276 3409 Acked Read 9 Done Done Sent false
11277 3410 Acked Read 3 Done Done Sent false
11278 3411 Acked Read 1 Done Done Sent false
11279 3412 Acked Read 4 Done Done Sent false
11280 3413 Acked Write 1 Done Done Sent false
11281 3414 Acked Flush 0 Done Done Sent false
11282 STATES DS:0 DS:1 DS:2 TOTAL
11283 Sent 0 0 301 301
11284 Done 301 301 0 602
11285 Skipped 0 0 0 0
11286 Error 0 0 0 0
11287 Last Flush: Some(JobId(3414)) Some(JobId(3414)) Some(JobId(3113))
11288 Downstairs last five completed: 3113 3112 3111 3110 3109
11289 Upstairs last five completed: 3414 3413 3412 3411 3410
11290 CLIENT: Up:0 ds:301 act:2
112912025-03-17T22:21:28.799ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
112922025-03-17T22:21:30.099ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
112932025-03-17T22:21:30.803ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
112942025-03-17T22:21:31.804ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
112952025-03-17T22:21:31.853ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
112962025-03-17T22:21:31.853ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:65208) = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
112972025-03-17T22:21:31.854ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
112982025-03-17T22:21:31.855ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
112992025-03-17T22:21:31.855ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
113002025-03-17T22:21:31.855ZINFOcrucible: send last flush ID to this DS: Some(JobId(3113)) = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
113012025-03-17T22:21:31.855ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
113022025-03-17T22:21:31.855ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(3113)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
113032025-03-17T22:21:31.855ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(3113)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
11304 ----------------------------------------------------------------
11305 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
11306 Downstairs last five completed: 3414 3413 3412 3411 3410
11307 Upstairs last five completed: 3414 3413 3412 3411 3410
11308 CLIENT: Up:0 ds:0 act:3
11309 Replay: All jobs finished
11310 Main task has work to do, go find it
11311 got dsc Stop(0)
11312 stop 0
11313 Mar 17 22:21:32.118 INFO request completed, latency_us: 351, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 37c1de5b-4f4b-41f2-a478-aeca565ca392, remote_addr: 127.0.[8810] Got stop action so:false kr:false
11314 0.1:58317, local_addr: 127.0.0.1:9998
11315 Mar 17 22:21:32.118 INFO accepted connection, remote_addr: 127.0.0.1:36841, local_addr: 127.0.0.1:9998
113162025-03-17T22:21:32.119ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
11317 Mar 17 22:21:32.119 INFO request completed{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:21:32.119070376Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
11318 , latency_us: 158, response_code: 200, uri: /state/cid/0, method: GET, req_id: 37dc3329-fb11-4364-8e7a-d393bf1465ae, remote_addr: 127{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:21:32.119099143Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
11319 .0.0.1:58317, local_addr: 127.0.0.1:9998
113202025-03-17T22:21:32.119ZINFOcrucible: client task is exiting = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
11321 [8810] Exited with: ExitStatus(unix_wait_status(9))
11322 [8810][0] reports Exit
11323 Mar 17 22:21:36.121 INFO request completed, latency_us: 210, response_code: 200, uri: /state/cid/0, method: GET, req_id: 5147742e-a5b0-4848-a17b-6f401eb6b5e3, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
11324 001/300 Read block 817 len 8192
11325 002/300 Flush
11326 003/300 Read block 1040 len 28672
11327 004/300 Flush
11328 005/300 Write block 1518 len 16384 data: 2 2 2 3
11329 006/300 Read block 1137 len 16384
11330 007/300 Write block 1751 len 4096 data: 4
11331 008/300 Read block 1286 len 12288
11332 009/300 Read block 4614 len 16384
11333 010/300 Read block 3201 len 20480
11334 011/300 Flush
11335 012/300 Read block 149 len 28672
11336 013/300 Write block 3055 len 12288 data: 6 6 6
11337 014/300 Write block 3081 len 32768 data: 6 6 3 3 3 3 4 4
11338 015/300 Write block 3173 len 16384 data: 4 4 4 4
11339 016/300 Write block 3047 len 12288 data: 4 4 4
11340 017/300 Write block 4444 len 28672 data: 3 2 2 3 4 5 4
11341 018/300 Write block 2079 len 36864 data: 3 3 2 2 2 2 2 3 3
11342 019/300 Read block 1531 len 32768
11343 020/300 Flush
11344 021/300 Read block 287 len 8192
11345 022/300 Read block 3901 len 8192
11346 023/300 Read block 3889 len 12288
11347 024/300 Write block 2770 len 8192 data: 2 2
11348 025/300 Write block 1875 len 36864 data: 3 3 3 3 4 4 4 3 4
11349 026/300 Write block 1710 len 24576 data: 5 4 4 4 4 4
11350 027/300 Read block 1216 len 8192
11351 028/300 Write block 4405 len 28672 data: 2 2 2 2 2 2 2
11352 029/300 Read block 2323 len 16384
11353 030/300 Read block 118 len 8192
11354 031/300 Read block 3649 len 16384
11355 032/300 Read block 2271 len 8192
11356 033/300 Write block 4835 len 20480 data: 3 3 2 2 2
11357 034/300 Write block 3048 len 12288 data: 5 5 4
11358 035/300 Read block 4134 len 16384
11359 036/300 Write block 1608 len 12288 data: 3 3 3
11360 037/300 Write block 3788 len 20480 data: 3 3 3 3 2
11361 038/300 Write block 3425 len 24576 data: 2 2 2 2 2 2
11362 039/300 Read block 1260 len 12288
11363 040/300 Write block 150 len 40960 data: 5 4 5 6 7 6 4 4 4 4
11364 041/300 Write block 2466 len 24576 data: 2 2 2 4 5 5
11365 042/300 Write block 2900 len 28672 data: 3 3 3 2 2 2 2
11366 043/300 Read block 3384 len 36864
11367 044/300 Read block 2600 len 16384
11368 045/300 Read block 3071 len 20480
11369 046/300 Write block 2297 len 28672 data: 2 3 3 3 2 2 2
11370 047/300 Read block 3668 len 20480
11371 048/300 Read block 4679 len 16384
11372 049/300 Read block 1667 len 4096
11373 050/300 Write block 2288 len 24576 data: 4 4 4 4 3 3
11374 051/300 Read block 3101 len 8192
11375 052/300 Flush
11376 053/300 Read block 4604 len 36864
11377 054/300 Read block 3221 len 32768
11378 055/300 Read block 800 len 36864
11379 056/300 Read block 810 len 16384
11380 057/300 Write block 2498 len 4096 data: 2
11381 058/300 Read block 2433 len 4096
11382 059/300 Read block 2304 len 36864
11383 060/300 Read block 2874 len 24576
11384 061/300 Write block 2980 len 28672 data: 2 2 2 2 2 2 2
11385 062/300 Read block 666 len 32768
11386 063/300 Read block 1666 len 16384
11387 064/300 Read block 4769 len 28672
11388 065/300 Write block 245 len 12288 data: 2 2 2
11389 066/300 Read block 1999 len 24576
11390 067/300 Flush
11391 068/300 Write block 4887 len 16384 data: 2 2 2 2
11392 069/300 Write block 3600 len 12288 data: 2 2 2
11393 070/300 Read block 1000 len 4096
11394 071/300 Write block 1673 len 24576 data: 2 2 2 2 2 2
11395 072/300 Read block 784 len 4096
11396 073/300 Write block 3830 len 24576 data: 3 2 2 3 4 4
11397 074/300 Read block 2275 len 40960
11398 075/300 Write block 2658 len 8192 data: 3 3
11399 076/300 Write block 2586 len 16384 data: 5 3 3 3
11400 077/300 Write block 1072 len 8192 data: 3 3
11401 078/300 Read block 2141 len 24576
11402 079/300 Write block 1046 len 40960 data: 4 4 5 5 5 4 3 5 5 5
11403 080/300 Write block 2379 len 24576 data: 3 3 2 3 3 3
11404 081/300 Flush
11405 082/300 Flush
11406 083/300 Write block 3049 len 8192 data: 6 5
11407 084/300 Flush
11408 085/300 Flush
11409 086/300 Write block 2016 len 16384 data: 3 3 4 4
11410 087/300 Read block 4886 len 16384
11411 088/300 Read block 1539 len 20480
11412 089/300 Read block 4937 len 12288
11413 090/300 Read block 2089 len 32768
11414 091/300 Read block 3628 len 40960
11415 092/300 Read block 4689 len 4096
11416 093/300 Read block 3621 len 24576
11417 094/300 Read block 251 len 12288
11418 095/300 Write block 4650 len 12288 data: 2 3 4
11419 096/300 Write block 782 len 40960 data: 6 6 5 5 5 4 4 3 3 3
11420 097/300 Write block 1635 len 36864 data: 4 4 3 3 3 3 3 3 3
11421 098/300 Write block 2411 len 28672 data: 3 3 3 3 3 3 4
11422 099/300 Flush
11423 100/300 Flush
11424 101/300 Read block 875 len 36864
11425 102/300 Write block 3261 len 20480 data: 4 4 3 3 2
11426 103/300 Flush
11427 104/300 Write block 4350 len 36864 data: 2 2 2 2 2 2 2 2 2
11428 105/300 Write block 3500 len 24576 data: 4 4 4 4 4 4
11429 106/300 Write block 1026 len 36864 data: 2 2 2 2 2 2 2 2 2
11430 107/300 Write block 4056 len 16384 data: 2 2 3 3
11431 108/300 Read block 4963 len 8192
11432 109/300 Write block 1493 len 8192 data: 3 3
11433 110/300 Write block 4895 len 36864 data: 2 2 2 2 2 2 2 2 3
11434 111/300 Read block 1123 len 8192
11435 112/300 Read block 4402 len 4096
11436 113/300 Flush
11437 114/300 Write block 2946 len 32768 data: 2 2 2 2 2 2 2 2
11438 115/300 Read block 4867 len 28672
11439 116/300 Read block 472 len 32768
11440 117/300 Write block 1614 len 12288 data: 2 2 2
11441 118/300 Flush
11442 119/300 Read block 3449 len 12288
11443 120/300 Write block 4834 len 36864 data: 3 4 4 3 3 3 2 2 2
11444 121/300 Write block 3726 len 32768 data: 2 2 2 2 3 3 3 3
11445 122/300 Write block 1452 len 28672 data: 3 2 2 2 2 2 3
11446 123/300 Write block 933 len 16384 data: 3 3 3 2
11447 124/300 Read block 1698 len 20480
11448 125/300 Flush
11449 126/300 Write block 2437 len 20480 data: 3 3 3 3 4
11450 127/300 Read block 486 len 20480
11451 128/300 Flush
11452 129/300 Write block 1049 len 8192 data: 6 6
11453 130/300 Read block 2119 len 4096
11454 131/300 Read block 3465 len 16384
11455 132/300 Flush
11456 133/300 Write block 2557 len 24576 data: 3 3 2 2 2 2
11457 134/300 Write block 2868 len 20480 data: 2 2 3 3 2
11458 135/300 Flush
11459 136/300 Flush
11460 137/300 Read block 897 len 16384
11461 138/300 Read block 991 len 40960
11462 139/300 Write block 2746 len 16384 data: 4 3 3 3
11463 140/300 Write block 2682 len 8192 data: 3 3
11464 141/300 Read block 153 len 24576
11465 142/300 Read block 1281 len 28672
11466 143/300 Write block 2637 len 40960 data: 4 4 5 5 4 3 3 2 2 2
11467 144/300 Write block 450 len 28672 data: 5 5 4 4 3 3 4
11468 145/300 Write block 58 len 40960 data: 3 3 3 4 4 4 4 2 2 2
11469 146/300 Read block 2 len 32768
11470 147/300 Write block 3933 len 20480 data: 2 2 2 2 2
11471 148/300 Read block 3649 len 8192
11472 149/300 Read block 859 len 32768
11473 150/300 Flush
11474 151/300 Write block 1922 len 16384 data: 4 5 5 4
11475 152/300 Write block 3727 len 8192 data: 3 3
11476 153/300 Write block 1470 len 28672 data: 2 3 3 4 4 4 3
11477 154/300 Write block 4098 len 16384 data: 2 2 2 3
11478 155/300 Read block 2277 len 24576
11479 156/300 Write block 1524 len 4096 data: 4
11480 157/300 Read block 1414 len 12288
11481 158/300 Read block 4830 len 12288
11482 159/300 Write block 3153 len 40960 data: 4 3 3 4 3 3 2 2 2 2
11483 160/300 Write block 4445 len 28672 data: 3 3 4 5 6 5 4
11484 161/300 Write block 4676 len 36864 data: 2 3 3 3 3 3 3 3 2
11485 162/300 Write block 4211 len 36864 data: 4 4 4 4 3 3 3 4 3
11486 163/300 Read block 4012 len 20480
11487 164/300 Read block 3725 len 40960
11488 165/300 Write block 4756 len 12288 data: 4 4 2
11489 166/300 Read block 2997 len 24576
11490 167/300 Read block 4485 len 40960
11491 168/300 Read block 429 len 28672
11492 169/300 Read block 2677 len 4096
11493 170/300 Write block 3587 len 20480 data: 3 3 4 4 4
11494 171/300 Write block 1164 len 8192 data: 2 2
11495 172/300 Read block 2300 len 40960
11496 173/300 Write block 3473 len 20480 data: 3 3 3 2 2
11497 174/300 Read block 2176 len 28672
11498 175/300 Write block 4525 len 16384 data: 2 3 3 3
11499 176/300 Write block 738 len 36864 data: 3 3 4 4 4 4 5 3 3
11500 177/300 Flush
11501 178/300 Write block 2453 len 32768 data: 2 3 2 2 2 2 2 2
11502 179/300 Read block 599 len 16384
11503 180/300 Read block 1044 len 8192
11504 181/300 Read block 1464 len 12288
11505 182/300 Write block 2386 len 4096 data: 4
11506 183/300 Read block 3100 len 36864
11507 184/300 Read block 2375 len 28672
11508 185/300 Read block 1544 len 4096
11509 186/300 Read block 2094 len 40960
11510 187/300 Read block 1575 len 24576
11511 188/300 Read block 4726 len 16384
11512 189/300 Read block 3982 len 8192
11513 190/300 Write block 522 len 24576 data: 3 2 2 2 2 2
11514 191/300 Read block 2942 len 32768
11515 192/300 Write block 2108 len 28672 data: 2 2 2 2 2 2 2
11516 193/300 Write block 2343 len 16384 data: 4 4 4 3
11517 194/300 Write block 4677 len 32768 data: 4 4 4 4 4 4 4 3
11518 195/300 Write block 591 len 24576 data: 3 4 4 4 4 3
11519 196/300 Write block 4216 len 28672 data: 4 4 5 4 5 5 5
11520 197/300 Read block 34 len 4096
11521 198/300 Write block 2411 len 16384 data: 4 4 4 4
11522 199/300 Write block 922 len 16384 data: 3 4 4 2
11523 200/300 Read block 2743 len 28672
11524 201/300 Write block 3325 len 32768 data: 3 4 5 3 2 2 2 2
11525 202/300 Write block 3132 len 36864 data: 4 3 3 4 6 6 6 5 5
11526 203/300 Read block 470 len 40960
11527 204/300 Flush
11528 205/300 Write block 3206 len 36864 data: 4 2 2 2 2 3 3 3 4
11529 206/300 Write block 3681 len 40960 data: 2 2 3 3 3 3 3 2 2 2
11530 207/300 Flush
11531 208/300 Read block 695 len 12288
11532 209/300 Read block 4025 len 16384
11533 210/300 Read block 3394 len 24576
11534 211/300 Read block 3483 len 32768
11535 212/300 Read block 3189 len 36864
11536 213/300 Write block 2685 len 4096 data: 3
11537 214/300 Read block 1660 len 12288
11538 215/300 Write block 2364 len 28672 data: 2 2 3 3 3 2 3
11539 216/300 Read block 4661 len 4096
11540 217/300 Read block 3096 len 16384
11541 218/300 Read block 4354 len 16384
11542 219/300 Flush
11543 220/300 Read block 1106 len 20480
11544 221/300 Read block 1258 len 28672
11545 222/300 Read block 3842 len 40960
11546 223/300 Read block 4498 len 32768
11547 224/300 Read block 4434 len 32768
11548 225/300 Flush
11549 226/300 Read block 478 len 12288
11550 227/300 Write block 1776 len 40960 data: 3 3 3 2 2 2 3 3 3 2
11551 228/300 Flush
11552 229/300 Read block 4095 len 32768
11553 230/300 Write block 2051 len 28672 data: 2 2 3 2 2 3 4
11554 231/300 Write block 398 len 12288 data: 3 3 3
11555 232/300 Read block 1453 len 16384
11556 233/300 Write block 3466 len 32768 data: 2 3 4 4 4 3 3 4
11557 234/300 Write block 1709 len 12288 data: 3 6 5
11558 235/300 Read block 3480 len 24576
11559 236/300 Read block 1637 len 24576
11560 237/300 Read block 1824 len 8192
11561 238/300 Write block 612 len 40960 data: 2 2 2 2 2 2 2 2 2 2
11562 239/300 Read block 3177 len 24576
11563 240/300 Write block 2336 len 24576 data: 2 2 5 5 5 5
11564 241/300 Read block 848 len 8192
11565 242/300 Read block 2878 len 28672
11566 243/300 Read block 4822 len 8192
11567 244/300 Write block 128 len 32768 data: 3 3 3 4 4 3 3 2
11568 245/300 Read block 2909 len 36864
11569 246/300 Read block 576 len 16384
11570 247/300 Write block 3227 len 32768 data: 2 2 2 3 3 3 2 2
11571 248/300 Flush
11572 249/300 Write block 4332 len 40960 data: 3 3 3 3 3 2 2 2 2 3
11573 250/300 Read block 297 len 40960
11574 251/300 Write block 1533 len 24576 data: 3 3 3 3 2 2
11575 252/300 Read block 1707 len 24576
11576 253/300 Write block 1839 len 12288 data: 5 5 4
11577 254/300 Read block 3918 len 40960
11578 255/300 Write block 1233 len 20480 data: 3 3 2 2 2
11579 256/300 Write block 3942 len 12288 data: 6 5 4
11580 257/300 Write block 413 len 20480 data: 6 6 6 6 6
11581 258/300 Read block 814 len 24576
11582 259/300 Read block 1353 len 20480
11583 260/300 Write block 1441 len 8192 data: 2 2
11584 261/300 Read block 4032 len 4096
11585 262/300 Read block 3683 len 40960
11586 263/300 Read block 1652 len 28672
11587 264/300 Read block 3379 len 40960
11588 265/300 Write block 755 len 32768 data: 4 4 3 3 3 3 2 2
11589 266/300 Write block 2819 len 16384 data: 2 2 2 2
11590 267/300 Read block 396 len 4096
11591 268/300 Read block 1263 len 16384
11592 269/300 Write block 3244 len 32768 data: 2 2 2 3 5 5 5 5
11593 270/300 Write block 3525 len 40960 data: 4 3 3 4 5 5 6 4 4 4
11594 271/300 Read block 3469 len 12288
11595 272/300 Read block 3303 len 36864
11596 273/300 Write block 4468 len 24576 data: 2 2 2 2 2 3
11597 274/300 Write block 1193 len 16384 data: 2 2 2 2
11598 275/300 Read block 4368 len 32768
11599 276/300 Read block 3743 len 24576
11600 277/300 Read block 2835 len 12288
11601 278/300 Write block 1212 len 16384 data: 3 3 2 2
11602 279/300 Write block 2766 len 40960 data: 3 2 2 2 3 3 2 2 2 2
11603 280/300 Read block 2754 len 24576
11604 281/300 Read block 2662 len 4096
11605 282/300 Read block 1103 len 4096
11606 283/300 Read block 780 len 16384
11607 284/300 Read block 1090 len 32768
11608 285/300 Write block 1791 len 24576 data: 2 2 2 2 2 2
11609 286/300 Write block 2020 len 8192 data: 3 3
11610 287/300 Write block 295 len 16384 data: 4 4 4 3
11611 288/300 Read block 752 len 40960
11612 289/300 Read block 988 len 32768
11613 290/300 Write block 121 len 40960 data: 4 4 4 3 3 3 3 4 4 4
11614 291/300 Read block 3755 len 8192
11615 292/300 Read block 2039 len 32768
11616 293/300 Read block 3925 len 12288
11617 294/300 Read block 4346 len 12288
11618 295/300 Read block 3589 len 12288
11619 296/300 Read block 2325 len 8192
11620 297/300 Write block 362 len 36864 data: 2 2 3 4 6 5 5 5 5
11621 298/300 Read block 3799 len 16384
11622 299/300 Flush
11623 300/300 Write block 740 len 36864 data: 5 5 5 5 6 4 4 3 3
11624 301/300 Write block 106 len 32768 data: 3 2 2 2 2 2 2 2
11625 Main task has work to do, go find it
11626 got dsc Start(0)
11627 start 0
11628 Mar 17 22:21:36.363 INFO request completed, latency_us: 265, response_code: 204, uri: /start/cid/0, method: POST, req_id: b60f6450-4dab-48a4-8c19-46063ff97f3d, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
11629 Replay: started 0, returned:Ok(())
11630 ----------------------------------------------------------------
11631 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11632 States: Offline Active Active
11633 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11634 3415 Acked Read 2 Sent Done Done false
11635 3416 Acked Flush 0 Sent Done Done false
11636 3417 Acked Read 7 Sent Done Done false
11637 3418 Acked Flush 0 Sent Done Done false
11638 3419 Acked Write 4 Sent Done Done false
11639 3420 Acked Read 4 Sent Done Done false
11640 3421 Acked Write 1 Sent Done Done false
11641 3422 Acked Read 3 Sent Done Done false
11642 3423 Acked Read 4 Sent Done Done false
11643 3424 Acked Read 5 Sent Done Done false
11644 3425 Acked Flush 0 Sent Done Done false
11645 3426 Acked Read 7 Sent Done Done false
11646 3427 Acked Write 3 Sent Done Done false
11647 3428 Acked Write 8 Sent Done Done false
11648 3429 Acked Write 4 Sent Done Done false
11649 3430 Acked Write 3 Sent Done Done false
11650 3431 Acked Write 7 Sent Done Done false
11651 3432 Acked Write 9 Sent Done Done false
11652 3433 Acked Read 8 Sent Done Done false
11653 3434 Acked Flush 0 Sent Done Done false
11654 3435 Acked Read 2 Sent Done Done false
11655 3436 Acked Read 2 Sent Done Done false
11656 3437 Acked Read 3 Sent Done Done false
11657 3438 Acked Write 2 Sent Done Done false
11658 3439 Acked Write 9 Sent Done Done false
11659 3440 Acked Write 6 Sent Done Done false
11660 3441 Acked Read 2 Sent Done Done false
11661 3442 Acked Write 7 Sent Done Done false
11662 3443 Acked Read 4 Sent Done Done false
11663 3444 Acked Read 2 Sent Done Done false
11664 3445 Acked Read 4 Sent Done Done false
11665 3446 Acked Read 2 Sent Done Done false
11666 3447 Acked Write 5 Sent Done Done false
11667 3448 Acked Write 3 Sent Done Done false
11668 3449 Acked Read 4 Sent Done Done false
11669 3450 Acked Write 3 Sent Done Done false
11670 3451 Acked Write 5 Sent Done Done false
11671 3452 Acked Write 6 Sent Done Done false
11672 3453 Acked Read 3 Sent Done Done false
11673 3454 Acked Write 10 Sent Done Done false
11674 3455 Acked Write 6 Sent Done Done false
11675 3456 Acked Write 7 Sent Done Done false
11676 3457 Acked Read 9 Sent Done Done false
11677 3458 Acked Read 4 Sent Done Done false
11678 3459 Acked Read 5 Sent Done Done false
11679 3460 Acked Write 7 Sent Done Done false
11680 3461 Acked Read 5 Sent Done Done false
11681 3462 Acked Read 4 Sent Done Done false
11682 3463 Acked Read 1 Sent Done Done false
11683 3464 Acked Write 6 Sent Done Done false
11684 3465 Acked Read 2 Sent Done Done false
11685 3466 Acked Flush 0 Sent Done Done false
11686 3467 Acked Read 9 Sent Done Done false
11687 3468 Acked Read 8 Sent Done Done false
11688 3469 Acked Read 9 Sent Done Done false
11689 3470 Acked Read 4 Sent Done Done false
11690 3471 Acked Write 1 Sent Done Done false
11691 3472 Acked Read 1 Sent Done Done false
11692 3473 Acked Read 9 Sent Done Done false
11693 3474 Acked Read 6 Sent Done Done false
11694 3475 Acked Write 7 Sent Done Done false
11695 3476 Acked Read 8 Sent Done Done false
11696 3477 Acked Read 4 Sent Done Done false
11697 3478 Acked Read 7 Sent Done Done false
11698 3479 Acked Write 3 Sent Done Done false
11699 3480 Acked Read 6 Sent Done Done false
11700 3481 Acked Flush 0 Sent Done Done false
11701 3482 Acked Write 4 Sent Done Done false
11702 3483 Acked Write 3 Sent Done Done false
11703 3484 Acked Read 1 Sent Done Done false
11704 3485 Acked Write 6 Sent Done Done false
11705 3486 Acked Read 1 Sent Done Done false
11706 3487 Acked Write 6 Sent Done Done false
11707 3488 Acked Read 10 Sent Done Done false
11708 3489 Acked Write 2 Sent Done Done false
11709 3490 Acked Write 4 Sent Done Done false
11710 3491 Acked Write 2 Sent Done Done false
11711 3492 Acked Read 6 Sent Done Done false
11712 3493 Acked Write 10 Sent Done Done false
11713 3494 Acked Write 6 Sent Done Done false
11714 3495 Acked Flush 0 Sent Done Done false
11715 3496 Acked Flush 0 Sent Done Done false
11716 3497 Acked Write 2 Sent Done Done false
11717 3498 Acked Flush 0 Sent Done Done false
11718 3499 Acked Flush 0 Sent Done Done false
11719 3500 Acked Write 4 Sent Done Done false
11720 3501 Acked Read 4 Sent Done Done false
11721 3502 Acked Read 5 Sent Done Done false
11722 3503 Acked Read 3 Sent Done Done false
11723 3504 Acked Read 8 Sent Done Done false
11724 3505 Acked Read 10 Sent Done Done false
11725 3506 Acked Read 1 Sent Done Done false
11726 3507 Acked Read 6 Sent Done Done false
11727 3508 Acked Read 3 Sent Done Done false
11728 3509 Acked Write 3 Sent Done Done false
11729 3510 Acked Write 10 Sent Done Done false
11730 3511 Acked Write 9 Sent Done Done false
11731 3512 Acked Write 7 Sent Done Done false
11732 3513 Acked Flush 0 Sent Done Done false
11733 3514 Acked Flush 0 Sent Done Done false
11734 3515 Acked Read 9 Sent Done Done false
11735 3516 Acked Write 5 Sent Done Done false
11736 3517 Acked Flush 0 Sent Done Done false
11737 3518 Acked Write 9 Sent Done Done false
11738 3519 Acked Write 6 Sent Done Done false
11739 3520 Acked Write 9 Sent Done Done false
11740 3521 Acked Write 4 Sent Done Done false
11741 3522 Acked Read 2 Sent Done Done false
11742 3523 Acked Write 2 Sent Done Done false
11743 3524 Acked Write 9 Sent Done Done false
11744 3525 Acked Read 2 Sent Done Done false
11745 3526 Acked Read 1 Sent Done Done false
11746 3527 Acked Flush 0 Sent Done Done false
11747 3528 Acked Write 8 Sent Done Done false
11748 3529 Acked Read 7 Sent Done Done false
11749 3530 Acked Read 8 Sent Done Done false
11750 3531 Acked Write 3 Sent Done Done false
11751 3532 Acked Flush 0 Sent Done Done false
11752 3533 Acked Read 3 Sent Done Done false
11753 3534 Acked Write 9 Sent Done Done false
11754 3535 Acked Write 8 Sent Done Done false
11755 3536 Acked Write 7 Sent Done Done false
11756 3537 Acked Write 4 Sent Done Done false
11757 3538 Acked Read 5 Sent Done Done false
11758 3539 Acked Flush 0 Sent Done Done false
11759 3540 Acked Write 5 Sent Done Done false
11760 3541 Acked Read 5 Sent Done Done false
11761 3542 Acked Flush 0 Sent Done Done false
11762 3543 Acked Write 2 Sent Done Done false
11763 3544 Acked Read 1 Sent Done Done false
11764 3545 Acked Read 4 Sent Done Done false
11765 3546 Acked Flush 0 Sent Done Done false
11766 3547 Acked Write 6 Sent Done Done false
11767 3548 Acked Write 5 Sent Done Done false
11768 3549 Acked Flush 0 Sent Done Done false
11769 3550 Acked Flush 0 Sent Done Done false
11770 3551 Acked Read 4 Sent Done Done false
11771 3552 Acked Read 10 Sent Done Done false
11772 3553 Acked Write 4 Sent Done Done false
11773 3554 Acked Write 2 Sent Done Done false
11774 3555 Acked Read 6 Sent Done Done false
11775 3556 Acked Read 7 Sent Done Done false
11776 3557 Acked Write 10 Sent Done Done false
11777 3558 Acked Write 7 Sent Done Done false
11778 3559 Acked Write 10 Sent Done Done false
11779 3560 Acked Read 8 Sent Done Done false
11780 3561 Acked Write 5 Sent Done Done false
11781 3562 Acked Read 2 Sent Done Done false
11782 3563 Acked Read 8 Sent Done Done false
11783 3564 Acked Flush 0 Sent Done Done false
11784 3565 Acked Write 4 Sent Done Done false
11785 3566 Acked Write 2 Sent Done Done false
11786 3567 Acked Write 7 Sent Done Done false
11787 3568 Acked Write 4 Sent Done Done false
11788 3569 Acked Read 6 Sent Done Done false
11789 3570 Acked Write 1 Sent Done Done false
11790 3571 Acked Read 3 Sent Done Done false
11791 3572 Acked Read 3 Sent Done Done false
11792 3573 Acked Write 10 Sent Done Done false
11793 3574 Acked Write 7 Sent Done Done false
11794 3575 Acked Write 9 Sent Done Done false
11795 3576 Acked Write 9 Sent Done Done false
11796 3577 Acked Read 5 Sent Done Done false
11797 3578 Acked Read 10 Sent Done Done false
11798 3579 Acked Write 3 Sent Done Done false
11799 3580 Acked Read 6 Sent Done Done false
11800 3581 Acked Read 10 Sent Done Done false
11801 3582 Acked Read 7 Sent Done Done false
11802 3583 Acked Read 1 Sent Done Done false
11803 3584 Acked Write 5 Sent Done Done false
11804 3585 Acked Write 2 Sent Done Done false
11805 3586 Acked Read 10 Sent Done Done false
11806 3587 Acked Write 5 Sent Done Done false
11807 3588 Acked Read 7 Sent Done Done false
11808 3589 Acked Write 4 Sent Done Done false
11809 3590 Acked Write 9 Sent Done Done false
11810 3591 Acked Flush 0 Sent Done Done false
11811 3592 Acked Write 8 Sent Done Done false
11812 3593 Acked Read 4 Sent Done Done false
11813 3594 Acked Read 2 Sent Done Done false
11814 3595 Acked Read 3 Sent Done Done false
11815 3596 Acked Write 1 Sent Done Done false
11816 3597 Acked Read 9 Sent Done Done false
11817 3598 Acked Read 7 Sent Done Done false
11818 3599 Acked Read 1 Sent Done Done false
11819 3600 Acked Read 10 Sent Done Done false
11820 3601 Acked Read 6 Sent Done Done false
11821 3602 Acked Read 4 Sent Done Done false
11822 3603 Acked Read 2 Sent Done Done false
11823 3604 Acked Write 6 Sent Done Done false
11824 3605 Acked Read 8 Sent Done Done false
11825 3606 Acked Write 7 Sent Done Done false
11826 3607 Acked Write 4 Sent Done Done false
11827 3608 Acked Write 8 Sent Done Done false
11828 3609 Acked Write 6 Sent Done Done false
11829 3610 Acked Write 7 Sent Done Done false
11830 3611 Acked Read 1 Sent Done Done false
11831 3612 Acked Write 4 Sent Done Done false
11832 3613 Acked Write 4 Sent Done Done false
11833 3614 Acked Read 7 Sent Done Done false
11834 3615 Acked Write 8 Sent Done Done false
11835 3616 Acked Write 9 Sent Done Done false
11836 3617 Acked Read 10 Sent Done Done false
11837 3618 Acked Flush 0 Sent Done Done false
11838 3619 Acked Write 9 Sent Done Done false
11839 3620 Acked Write 10 Sent Done Done false
11840 3621 Acked Flush 0 Sent Done Done false
11841 3622 Acked Read 3 Sent Done Done false
11842 3623 Acked Read 4 Sent Done Done false
11843 3624 Acked Read 6 Sent Done Done false
11844 3625 Acked Read 8 Sent Done Done false
11845 3626 Acked Read 9 Sent Done Done false
11846 3627 Acked Write 1 Sent Done Done false
11847 3628 Acked Read 3 Sent Done Done false
11848 3629 Acked Write 7 Sent Done Done false
11849 3630 Acked Read 1 Sent Done Done false
11850 3631 Acked Read 4 Sent Done Done false
11851 3632 Acked Read 4 Sent Done Done false
11852 3633 Acked Flush 0 Sent Done Done false
11853 3634 Acked Read 5 Sent Done Done false
11854 3635 Acked Read 7 Sent Done Done false
11855 3636 Acked Read 10 Sent Done Done false
11856 3637 Acked Read 8 Sent Done Done false
11857 3638 Acked Read 8 Sent Done Done false
11858 3639 Acked Flush 0 Sent Done Done false
11859 3640 Acked Read 3 Sent Done Done false
11860 3641 Acked Write 10 Sent Done Done false
11861 3642 Acked Flush 0 Sent Done Done false
11862 3643 Acked Read 8 Sent Done Done false
11863 3644 Acked Write 7 Sent Done Done false
11864 3645 Acked Write 3 Sent Done Done false
11865 3646 Acked Read 4 Sent Done Done false
11866 3647 Acked Write 8 Sent Done Done false
11867 3648 Acked Write 3 Sent Done Done false
11868 3649 Acked Read 6 Sent Done Done false
11869 3650 Acked Read 6 Sent Done Done false
11870 3651 Acked Read 2 Sent Done Done false
11871 3652 Acked Write 10 Sent Done Done false
11872 3653 Acked Read 6 Sent Done Done false
11873 3654 Acked Write 6 Sent Done Done false
11874 3655 Acked Read 2 Sent Done Done false
11875 3656 Acked Read 7 Sent Done Done false
11876 3657 Acked Read 2 Sent Done Done false
11877 3658 Acked Write 8 Sent Done Done false
11878 3659 Acked Read 9 Sent Done Done false
11879 3660 Acked Read 4 Sent Done Done false
11880 3661 Acked Write 8 Sent Done Done false
11881 3662 Acked Flush 0 Sent Done Done false
11882 3663 Acked Write 10 Sent Done Done false
11883 3664 Acked Read 10 Sent Done Done false
11884 3665 Acked Write 6 Sent Done Done false
11885 3666 Acked Read 6 Sent Done Done false
11886 3667 Acked Write 3 Sent Done Done false
11887 3668 Acked Read 10 Sent Done Done false
11888 3669 Acked Write 5 Sent Done Done false
11889 3670 Acked Write 3 Sent Done Done false
11890 3671 Acked Write 5 Sent Done Done false
11891 3672 Acked Read 6 Sent Done Done false
11892 3673 Acked Read 5 Sent Done Done false
11893 3674 Acked Write 2 Sent Done Done false
11894 3675 Acked Read 1 Sent Done Done false
11895 3676 Acked Read 10 Sent Done Done false
11896 3677 Acked Read 7 Sent Done Done false
11897 3678 Acked Read 10 Sent Done Done false
11898 3679 Acked Write 8 Sent Done Done false
11899 3680 Acked Write 4 Sent Done Done false
11900 3681 Acked Read 1 Sent Done Done false
11901 3682 Acked Read 4 Sent Done Done false
11902 3683 Acked Write 8 Sent Done Done false
11903 3684 Acked Write 10 Sent Done Done false
11904 3685 Acked Read 3 Sent Done Done false
11905 3686 Acked Read 9 Sent Done Done false
11906 3687 Acked Write 6 Sent Done Done false
11907 3688 Acked Write 4 Sent Done Done false
11908 3689 Acked Read 8 Sent Done Done false
11909 3690 Acked Read 6 Sent Done Done false
11910 3691 Acked Read 3 Sent Done Done false
11911 3692 Acked Write 4 Sent Done Done false
11912 3693 Acked Write 10 Sent Done Done false
11913 3694 Acked Read 6 Sent Done Done false
11914 3695 Acked Read 1 Sent Done Done false
11915 3696 Acked Read 1 Sent Done Done false
11916 3697 Acked Read 4 Sent Done Done false
11917 3698 Acked Read 8 Sent Done Done false
11918 3699 Acked Write 6 Sent Done Done false
11919 3700 Acked Write 2 Sent Done Done false
11920 3701 Acked Write 4 Sent Done Done false
11921 3702 Acked Read 10 Sent Done Done false
11922 3703 Acked Read 8 Sent Done Done false
11923 3704 Acked Write 10 Sent Done Done false
11924 3705 Acked Read 2 Sent Done Done false
11925 3706 Acked Read 8 Sent Done Done false
11926 3707 Acked Read 3 Sent Done Done false
11927 3708 Acked Read 3 Sent Done Done false
11928 3709 Acked Read 3 Sent Done Done false
11929 3710 Acked Read 2 Sent Done Done false
11930 3711 Acked Write 9 Sent Done Done false
11931 3712 Acked Read 4 Sent Done Done false
11932 3713 Acked Flush 0 Sent Done Done false
11933 3714 Acked Write 9 Sent Done Done false
11934 3715 Acked Write 8 Sent Sent Sent false
11935 STATES DS:0 DS:1 DS:2 TOTAL
11936 Sent 301 1 1 303
11937 Done 0 300 300 600
11938 Skipped 0 0 0 0
11939 Error 0 0 0 0
11940 Last Flush: Some(JobId(3414)) Some(JobId(3713)) Some(JobId(3713))
11941 Downstairs last five completed: 3414 3413 3412 3411 3410
11942 Upstairs last five completed: 3715 3714 3713 3712 3711
11943 CLIENT: Up:0 ds:301 act:2
119442025-03-17T22:21:37.813ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
119452025-03-17T22:21:38.815ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
119462025-03-17T22:21:40.102ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
11947 [8810] got start action
11948 [8810] I am going to restart
11949 Starting downstairs at port 8810
11950 Make output file at "/tmp/dsc/downstairs-8810.txt"
11951 [8810][0] reports Starting
11952 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1220)
11953 [8810][0] reports Running
11954 ----------------------------------------------------------------
11955 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11956 States: Offline Active Active
11957 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11958 3415 Acked Read 2 Sent Done Done false
11959 3416 Acked Flush 0 Sent Done Done false
11960 3417 Acked Read 7 Sent Done Done false
11961 3418 Acked Flush 0 Sent Done Done false
11962 3419 Acked Write 4 Sent Done Done false
11963 3420 Acked Read 4 Sent Done Done false
11964 3421 Acked Write 1 Sent Done Done false
11965 3422 Acked Read 3 Sent Done Done false
11966 3423 Acked Read 4 Sent Done Done false
11967 3424 Acked Read 5 Sent Done Done false
11968 3425 Acked Flush 0 Sent Done Done false
11969 3426 Acked Read 7 Sent Done Done false
11970 3427 Acked Write 3 Sent Done Done false
11971 3428 Acked Write 8 Sent Done Done false
11972 3429 Acked Write 4 Sent Done Done false
11973 3430 Acked Write 3 Sent Done Done false
11974 3431 Acked Write 7 Sent Done Done false
11975 3432 Acked Write 9 Sent Done Done false
11976 3433 Acked Read 8 Sent Done Done false
11977 3434 Acked Flush 0 Sent Done Done false
11978 3435 Acked Read 2 Sent Done Done false
11979 3436 Acked Read 2 Sent Done Done false
11980 3437 Acked Read 3 Sent Done Done false
11981 3438 Acked Write 2 Sent Done Done false
11982 3439 Acked Write 9 Sent Done Done false
11983 3440 Acked Write 6 Sent Done Done false
11984 3441 Acked Read 2 Sent Done Done false
11985 3442 Acked Write 7 Sent Done Done false
11986 3443 Acked Read 4 Sent Done Done false
11987 3444 Acked Read 2 Sent Done Done false
11988 3445 Acked Read 4 Sent Done Done false
11989 3446 Acked Read 2 Sent Done Done false
11990 3447 Acked Write 5 Sent Done Done false
11991 3448 Acked Write 3 Sent Done Done false
11992 3449 Acked Read 4 Sent Done Done false
11993 3450 Acked Write 3 Sent Done Done false
11994 3451 Acked Write 5 Sent Done Done false
11995 3452 Acked Write 6 Sent Done Done false
11996 3453 Acked Read 3 Sent Done Done false
11997 3454 Acked Write 10 Sent Done Done false
11998 3455 Acked Write 6 Sent Done Done false
11999 3456 Acked Write 7 Sent Done Done false
12000 3457 Acked Read 9 Sent Done Done false
12001 3458 Acked Read 4 Sent Done Done false
12002 3459 Acked Read 5 Sent Done Done false
12003 3460 Acked Write 7 Sent Done Done false
12004 3461 Acked Read 5 Sent Done Done false
12005 3462 Acked Read 4 Sent Done Done false
12006 3463 Acked Read 1 Sent Done Done false
12007 3464 Acked Write 6 Sent Done Done false
12008 3465 Acked Read 2 Sent Done Done false
12009 3466 Acked Flush 0 Sent Done Done false
12010 3467 Acked Read 9 Sent Done Done false
12011 3468 Acked Read 8 Sent Done Done false
12012 3469 Acked Read 9 Sent Done Done false
12013 3470 Acked Read 4 Sent Done Done false
12014 3471 Acked Write 1 Sent Done Done false
12015 3472 Acked Read 1 Sent Done Done false
12016 3473 Acked Read 9 Sent Done Done false
12017 3474 Acked Read 6 Sent Done Done false
12018 3475 Acked Write 7 Sent Done Done false
12019 3476 Acked Read 8 Sent Done Done false
12020 3477 Acked Read 4 Sent Done Done false
12021 3478 Acked Read 7 Sent Done Done false
12022 3479 Acked Write 3 Sent Done Done false
12023 3480 Acked Read 6 Sent Done Done false
12024 3481 Acked Flush 0 Sent Done Done false
12025 3482 Acked Write 4 Sent Done Done false
12026 3483 Acked Write 3 Sent Done Done false
12027 3484 Acked Read 1 Sent Done Done false
12028 3485 Acked Write 6 Sent Done Done false
12029 3486 Acked Read 1 Sent Done Done false
12030 3487 Acked Write 6 Sent Done Done false
12031 3488 Acked Read 10 Sent Done Done false
12032 3489 Acked Write 2 Sent Done Done false
12033 3490 Acked Write 4 Sent Done Done false
12034 3491 Acked Write 2 Sent Done Done false
12035 3492 Acked Read 6 Sent Done Done false
12036 3493 Acked Write 10 Sent Done Done false
12037 3494 Acked Write 6 Sent Done Done false
12038 3495 Acked Flush 0 Sent Done Done false
12039 3496 Acked Flush 0 Sent Done Done false
12040 3497 Acked Write 2 Sent Done Done false
12041 3498 Acked Flush 0 Sent Done Done false
12042 3499 Acked Flush 0 Sent Done Done false
12043 3500 Acked Write 4 Sent Done Done false
12044 3501 Acked Read 4 Sent Done Done false
12045 3502 Acked Read 5 Sent Done Done false
12046 3503 Acked Read 3 Sent Done Done false
12047 3504 Acked Read 8 Sent Done Done false
12048 3505 Acked Read 10 Sent Done Done false
12049 3506 Acked Read 1 Sent Done Done false
12050 3507 Acked Read 6 Sent Done Done false
12051 3508 Acked Read 3 Sent Done Done false
12052 3509 Acked Write 3 Sent Done Done false
12053 3510 Acked Write 10 Sent Done Done false
12054 3511 Acked Write 9 Sent Done Done false
12055 3512 Acked Write 7 Sent Done Done false
12056 3513 Acked Flush 0 Sent Done Done false
12057 3514 Acked Flush 0 Sent Done Done false
12058 3515 Acked Read 9 Sent Done Done false
12059 3516 Acked Write 5 Sent Done Done false
12060 3517 Acked Flush 0 Sent Done Done false
12061 3518 Acked Write 9 Sent Done Done false
12062 3519 Acked Write 6 Sent Done Done false
12063 3520 Acked Write 9 Sent Done Done false
12064 3521 Acked Write 4 Sent Done Done false
12065 3522 Acked Read 2 Sent Done Done false
12066 3523 Acked Write 2 Sent Done Done false
12067 3524 Acked Write 9 Sent Done Done false
12068 3525 Acked Read 2 Sent Done Done false
12069 3526 Acked Read 1 Sent Done Done false
12070 3527 Acked Flush 0 Sent Done Done false
12071 3528 Acked Write 8 Sent Done Done false
12072 3529 Acked Read 7 Sent Done Done false
12073 3530 Acked Read 8 Sent Done Done false
12074 3531 Acked Write 3 Sent Done Done false
12075 3532 Acked Flush 0 Sent Done Done false
12076 3533 Acked Read 3 Sent Done Done false
12077 3534 Acked Write 9 Sent Done Done false
12078 3535 Acked Write 8 Sent Done Done false
12079 3536 Acked Write 7 Sent Done Done false
12080 3537 Acked Write 4 Sent Done Done false
12081 3538 Acked Read 5 Sent Done Done false
12082 3539 Acked Flush 0 Sent Done Done false
12083 3540 Acked Write 5 Sent Done Done false
12084 3541 Acked Read 5 Sent Done Done false
12085 3542 Acked Flush 0 Sent Done Done false
12086 3543 Acked Write 2 Sent Done Done false
12087 3544 Acked Read 1 Sent Done Done false
12088 3545 Acked Read 4 Sent Done Done false
12089 3546 Acked Flush 0 Sent Done Done false
12090 3547 Acked Write 6 Sent Done Done false
12091 3548 Acked Write 5 Sent Done Done false
12092 3549 Acked Flush 0 Sent Done Done false
12093 3550 Acked Flush 0 Sent Done Done false
12094 3551 Acked Read 4 Sent Done Done false
12095 3552 Acked Read 10 Sent Done Done false
12096 3553 Acked Write 4 Sent Done Done false
12097 3554 Acked Write 2 Sent Done Done false
12098 3555 Acked Read 6 Sent Done Done false
12099 3556 Acked Read 7 Sent Done Done false
12100 3557 Acked Write 10 Sent Done Done false
12101 3558 Acked Write 7 Sent Done Done false
12102 3559 Acked Write 10 Sent Done Done false
12103 3560 Acked Read 8 Sent Done Done false
12104 3561 Acked Write 5 Sent Done Done false
12105 3562 Acked Read 2 Sent Done Done false
12106 3563 Acked Read 8 Sent Done Done false
12107 3564 Acked Flush 0 Sent Done Done false
12108 3565 Acked Write 4 Sent Done Done false
12109 3566 Acked Write 2 Sent Done Done false
12110 3567 Acked Write 7 Sent Done Done false
12111 3568 Acked Write 4 Sent Done Done false
12112 3569 Acked Read 6 Sent Done Done false
12113 3570 Acked Write 1 Sent Done Done false
12114 3571 Acked Read 3 Sent Done Done false
12115 3572 Acked Read 3 Sent Done Done false
12116 3573 Acked Write 10 Sent Done Done false
12117 3574 Acked Write 7 Sent Done Done false
12118 3575 Acked Write 9 Sent Done Done false
12119 3576 Acked Write 9 Sent Done Done false
12120 3577 Acked Read 5 Sent Done Done false
12121 3578 Acked Read 10 Sent Done Done false
12122 3579 Acked Write 3 Sent Done Done false
12123 3580 Acked Read 6 Sent Done Done false
12124 3581 Acked Read 10 Sent Done Done false
12125 3582 Acked Read 7 Sent Done Done false
12126 3583 Acked Read 1 Sent Done Done false
12127 3584 Acked Write 5 Sent Done Done false
12128 3585 Acked Write 2 Sent Done Done false
12129 3586 Acked Read 10 Sent Done Done false
12130 3587 Acked Write 5 Sent Done Done false
12131 3588 Acked Read 7 Sent Done Done false
12132 3589 Acked Write 4 Sent Done Done false
12133 3590 Acked Write 9 Sent Done Done false
12134 3591 Acked Flush 0 Sent Done Done false
12135 3592 Acked Write 8 Sent Done Done false
12136 3593 Acked Read 4 Sent Done Done false
12137 3594 Acked Read 2 Sent Done Done false
12138 3595 Acked Read 3 Sent Done Done false
12139 3596 Acked Write 1 Sent Done Done false
12140 3597 Acked Read 9 Sent Done Done false
12141 3598 Acked Read 7 Sent Done Done false
12142 3599 Acked Read 1 Sent Done Done false
12143 3600 Acked Read 10 Sent Done Done false
12144 3601 Acked Read 6 Sent Done Done false
12145 3602 Acked Read 4 Sent Done Done false
12146 3603 Acked Read 2 Sent Done Done false
12147 3604 Acked Write 6 Sent Done Done false
12148 3605 Acked Read 8 Sent Done Done false
12149 3606 Acked Write 7 Sent Done Done false
12150 3607 Acked Write 4 Sent Done Done false
12151 3608 Acked Write 8 Sent Done Done false
12152 3609 Acked Write 6 Sent Done Done false
12153 3610 Acked Write 7 Sent Done Done false
12154 3611 Acked Read 1 Sent Done Done false
12155 3612 Acked Write 4 Sent Done Done false
12156 3613 Acked Write 4 Sent Done Done false
12157 3614 Acked Read 7 Sent Done Done false
12158 3615 Acked Write 8 Sent Done Done false
12159 3616 Acked Write 9 Sent Done Done false
12160 3617 Acked Read 10 Sent Done Done false
12161 3618 Acked Flush 0 Sent Done Done false
12162 3619 Acked Write 9 Sent Done Done false
12163 3620 Acked Write 10 Sent Done Done false
12164 3621 Acked Flush 0 Sent Done Done false
12165 3622 Acked Read 3 Sent Done Done false
12166 3623 Acked Read 4 Sent Done Done false
12167 3624 Acked Read 6 Sent Done Done false
12168 3625 Acked Read 8 Sent Done Done false
12169 3626 Acked Read 9 Sent Done Done false
12170 3627 Acked Write 1 Sent Done Done false
12171 3628 Acked Read 3 Sent Done Done false
12172 3629 Acked Write 7 Sent Done Done false
12173 3630 Acked Read 1 Sent Done Done false
12174 3631 Acked Read 4 Sent Done Done false
12175 3632 Acked Read 4 Sent Done Done false
12176 3633 Acked Flush 0 Sent Done Done false
12177 3634 Acked Read 5 Sent Done Done false
12178 3635 Acked Read 7 Sent Done Done false
12179 3636 Acked Read 10 Sent Done Done false
12180 3637 Acked Read 8 Sent Done Done false
12181 3638 Acked Read 8 Sent Done Done false
12182 3639 Acked Flush 0 Sent Done Done false
12183 3640 Acked Read 3 Sent Done Done false
12184 3641 Acked Write 10 Sent Done Done false
12185 3642 Acked Flush 0 Sent Done Done false
12186 3643 Acked Read 8 Sent Done Done false
12187 3644 Acked Write 7 Sent Done Done false
12188 3645 Acked Write 3 Sent Done Done false
12189 3646 Acked Read 4 Sent Done Done false
12190 3647 Acked Write 8 Sent Done Done false
12191 3648 Acked Write 3 Sent Done Done false
12192 3649 Acked Read 6 Sent Done Done false
12193 3650 Acked Read 6 Sent Done Done false
12194 3651 Acked Read 2 Sent Done Done false
12195 3652 Acked Write 10 Sent Done Done false
12196 3653 Acked Read 6 Sent Done Done false
12197 3654 Acked Write 6 Sent Done Done false
12198 3655 Acked Read 2 Sent Done Done false
12199 3656 Acked Read 7 Sent Done Done false
12200 3657 Acked Read 2 Sent Done Done false
12201 3658 Acked Write 8 Sent Done Done false
12202 3659 Acked Read 9 Sent Done Done false
12203 3660 Acked Read 4 Sent Done Done false
12204 3661 Acked Write 8 Sent Done Done false
12205 3662 Acked Flush 0 Sent Done Done false
12206 3663 Acked Write 10 Sent Done Done false
12207 3664 Acked Read 10 Sent Done Done false
12208 3665 Acked Write 6 Sent Done Done false
12209 3666 Acked Read 6 Sent Done Done false
12210 3667 Acked Write 3 Sent Done Done false
12211 3668 Acked Read 10 Sent Done Done false
12212 3669 Acked Write 5 Sent Done Done false
12213 3670 Acked Write 3 Sent Done Done false
12214 3671 Acked Write 5 Sent Done Done false
12215 3672 Acked Read 6 Sent Done Done false
12216 3673 Acked Read 5 Sent Done Done false
12217 3674 Acked Write 2 Sent Done Done false
12218 3675 Acked Read 1 Sent Done Done false
12219 3676 Acked Read 10 Sent Done Done false
12220 3677 Acked Read 7 Sent Done Done false
12221 3678 Acked Read 10 Sent Done Done false
12222 3679 Acked Write 8 Sent Done Done false
12223 3680 Acked Write 4 Sent Done Done false
12224 3681 Acked Read 1 Sent Done Done false
12225 3682 Acked Read 4 Sent Done Done false
12226 3683 Acked Write 8 Sent Done Done false
12227 3684 Acked Write 10 Sent Done Done false
12228 3685 Acked Read 3 Sent Done Done false
12229 3686 Acked Read 9 Sent Done Done false
12230 3687 Acked Write 6 Sent Done Done false
12231 3688 Acked Write 4 Sent Done Done false
12232 3689 Acked Read 8 Sent Done Done false
12233 3690 Acked Read 6 Sent Done Done false
12234 3691 Acked Read 3 Sent Done Done false
12235 3692 Acked Write 4 Sent Done Done false
12236 3693 Acked Write 10 Sent Done Done false
12237 3694 Acked Read 6 Sent Done Done false
12238 3695 Acked Read 1 Sent Done Done false
12239 3696 Acked Read 1 Sent Done Done false
12240 3697 Acked Read 4 Sent Done Done false
12241 3698 Acked Read 8 Sent Done Done false
12242 3699 Acked Write 6 Sent Done Done false
12243 3700 Acked Write 2 Sent Done Done false
12244 3701 Acked Write 4 Sent Done Done false
12245 3702 Acked Read 10 Sent Done Done false
12246 3703 Acked Read 8 Sent Done Done false
12247 3704 Acked Write 10 Sent Done Done false
12248 3705 Acked Read 2 Sent Done Done false
12249 3706 Acked Read 8 Sent Done Done false
12250 3707 Acked Read 3 Sent Done Done false
12251 3708 Acked Read 3 Sent Done Done false
12252 3709 Acked Read 3 Sent Done Done false
12253 3710 Acked Read 2 Sent Done Done false
12254 3711 Acked Write 9 Sent Done Done false
12255 3712 Acked Read 4 Sent Done Done false
12256 3713 Acked Flush 0 Sent Done Done false
12257 3714 Acked Write 9 Sent Done Done false
12258 3715 Acked Write 8 Sent Done Done false
12259 STATES DS:0 DS:1 DS:2 TOTAL
12260 Sent 301 0 0 301
12261 Done 0 301 301 602
12262 Skipped 0 0 0 0
12263 Error 0 0 0 0
12264 Last Flush: Some(JobId(3414)) Some(JobId(3713)) Some(JobId(3713))
12265 Downstairs last five completed: 3414 3413 3412 3411 3410
12266 Upstairs last five completed: 3715 3714 3713 3712 3711
12267 CLIENT: Up:0 ds:301 act:2
122682025-03-17T22:21:40.868ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
122692025-03-17T22:21:41.820ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
122702025-03-17T22:21:42.120ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
122712025-03-17T22:21:42.120ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:60247) = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
122722025-03-17T22:21:42.122ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
122732025-03-17T22:21:42.122ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
122742025-03-17T22:21:42.122ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
122752025-03-17T22:21:42.122ZINFOcrucible: send last flush ID to this DS: Some(JobId(3414)) = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
122762025-03-17T22:21:42.122ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
122772025-03-17T22:21:42.122ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(3414)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
122782025-03-17T22:21:42.122ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(3414)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
12279 ----------------------------------------------------------------
12280 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
12281 Downstairs last five completed: 3716 3715 3714 3713 3712
12282 Upstairs last five completed: 3716 3715 3714 3713 3712
12283 CLIENT: Up:0 ds:0 act:3
12284 Replay: All jobs finished
12285 Main task has work to do, go find it
12286 got dsc Stop(1)
12287 stop 1
12288 Mar 17 22:21:44.368 INFO request completed, latency_us: 363, response_code: 204, uri: /stop/cid/1, method: POST[8820] Got stop action so:false kr:false
12289 , req_id: a10accad-10ca-4937-a5a7-ddc7055586b9, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
122902025-03-17T22:21:44.369ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
12291 Mar 17 22:21:44.369 INFO {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:21:44.369159594Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
12292 request completed, latency_us: 138, response_code: 200{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:21:44.369224497Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
12293 , uri: /state/cid/1, method: GET, req_id: aa64fc77-5d90-4808-aadd-3e05eaae73fd, remote_addr: 127.0.0{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:21:44.369257043Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
12294 .1:58317, local_addr: 127.0.0.1:9998
12295 [8820] Exited with: ExitStatus(unix_wait_status(9))
12296 [8820][1] reports Exit
12297 Mar 17 22:21:48.372 INFO request completed, latency_us: 302, response_code: 200, uri: /state/cid/1, method: GET, req_id: 9b752e79-4f5e-4691-a78d-873b9944e022, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
12298 001/300 Read block 4235 len 24576
12299 002/300 Read block 3966 len 20480
12300 003/300 Read block 4503 len 40960
12301 004/300 Read block 3862 len 32768
12302 005/300 Read block 1284 len 4096
12303 006/300 Write block 3242 len 4096 data: 2
12304 007/300 Read block 3484 len 12288
12305 008/300 Write block 3513 len 32768 data: 2 2 3 4 4 4 4 4
12306 009/300 Read block 4799 len 32768
12307 010/300 Write block 1325 len 8192 data: 2 3
12308 011/300 Read block 3131 len 4096
12309 012/300 Write block 4196 len 28672 data: 5 5 5 4 3 2 2
12310 013/300 Read block 1397 len 24576
12311 014/300 Write block 1327 len 40960 data: 4 3 3 4 5 4 4 4 3 3
12312 015/300 Write block 1053 len 32768 data: 6 6 6 5 5 4 4 3
12313 016/300 Write block 998 len 24576 data: 2 2 3 3 3 3
12314 017/300 Read block 2378 len 12288
12315 018/300 Read block 1764 len 20480
12316 019/300 Read block 4877 len 40960
12317 020/300 Read block 3235 len 8192
12318 021/300 Flush
12319 022/300 Read block 3610 len 32768
12320 023/300 Write block 3848 len 4096 data: 2
12321 024/300 Flush
12322 025/300 Read block 2291 len 28672
12323 026/300 Read block 2532 len 8192
12324 027/300 Write block 839 len 16384 data: 2 2 2 2
12325 028/300 Read block 227 len 4096
12326 029/300 Write block 175 len 16384 data: 2 2 2 2
12327 030/300 Write block 1342 len 4096 data: 2
12328 031/300 Write block 3182 len 32768 data: 3 3 4 4 4 4 3 3
12329 032/300 Write block 2434 len 24576 data: 4 3 3 4 4 4
12330 033/300 Read block 819 len 16384
12331 034/300 Read block 2517 len 24576
12332 035/300 Read block 2212 len 12288
12333 036/300 Write block 2236 len 36864 data: 2 3 3 4 4 3 3 3 2
12334 037/300 Flush
12335 038/300 Read block 4278 len 28672
12336 039/300 Read block 1343 len 12288
12337 040/300 Read block 2773 len 20480
12338 041/300 Write block 2361 len 4096 data: 3
12339 042/300 Write block 3406 len 20480 data: 6 5 5 4 4
12340 043/300 Flush
12341 044/300 Read block 668 len 40960
12342 045/300 Write block 2261 len 28672 data: 4 4 3 3 2 2 2
12343 046/300 Write block 1731 len 16384 data: 2 2 3 2
12344 047/300 Write block 4241 len 20480 data: 3 4 3 3 3
12345 048/300 Write block 4979 len 20480 data: 2 2 2 2 2
12346 049/300 Flush
12347 050/300 Write block 119 len 24576 data: 3 4 5 5 5 4
12348 051/300 Write block 3127 len 28672 data: 6 6 6 6 6 5 4
12349 052/300 Write block 3396 len 8192 data: 4 4
12350 053/300 Write block 3711 len 40960 data: 2 2 3 3 3 3 3 3 3 3
12351 054/300 Write block 4155 len 12288 data: 3 3 3
12352 055/300 Write block 3002 len 32768 data: 2 2 3 3 3 3 3 3
12353 056/300 Write block 4133 len 28672 data: 3 3 4 3 4 3 4
12354 057/300 Read block 1395 len 8192
12355 058/300 Read block 1403 len 12288
12356 059/300 Read block 442 len 20480
12357 060/300 Write block 3589 len 4096 data: 5
12358 061/300 Read block 3401 len 20480
12359 062/300 Read block 1762 len 16384
12360 063/300 Flush
12361 064/300 Read block 3531 len 28672
12362 065/300 Read block 4037 len 28672
12363 066/300 Write block 4224 len 36864 data: 4 5 5 5 3 3 3 4 3
12364 067/300 Write block 2564 len 20480 data: 2 2 2 2 3
12365 068/300 Read block 1002 len 36864
12366 069/300 Write block 2858 len 8192 data: 3 3
12367 070/300 Write block 1929 len 20480 data: 5 4 5 6 7
12368 071/300 Read block 2789 len 40960
12369 072/300 Read block 2731 len 24576
12370 073/300 Read block 2410 len 20480
12371 074/300 Write block 2483 len 24576 data: 2 2 2 2 2 2
12372 075/300 Write block 4020 len 24576 data: 2 2 2 2 2 2
12373 076/300 Read block 2081 len 24576
12374 077/300 Write block 3115 len 20480 data: 5 5 5 5 5
12375 078/300 Write block 1955 len 20480 data: 2 2 3 3 3
12376 079/300 Write block 3519 len 20480 data: 5 5 3 3 4
12377 080/300 Read block 351 len 12288
12378 081/300 Write block 3102 len 40960 data: 3 3 2 2 2 2 2 2 2 3
12379 082/300 Read block 3716 len 24576
12380 083/300 Read block 4801 len 20480
12381 084/300 Write block 4538 len 12288 data: 6 5 5
12382 085/300 Write block 692 len 36864 data: 3 2 2 2 2 2 2 3 4
12383 086/300 Write block 780 len 40960 data: 5 6 7 7 6 6 6 5 5 4
12384 087/300 Write block 3019 len 32768 data: 4 4 4 4 4 4 3 3
12385 088/300 Write block 1891 len 28672 data: 2 3 2 2 3 3 3
12386 089/300 Write block 3587 len 32768 data: 4 4 6 5 5 3 3 3
12387 090/300 Read block 929 len 12288
12388 091/300 Write block 71 len 32768 data: 2 2 2 2 2 2 2 3
12389 092/300 Read block 3160 len 12288
12390 093/300 Write block 1984 len 36864 data: 4 4 4 4 4 4 4 4 4
12391 094/300 Flush
12392 095/300 Read block 821 len 4096
12393 096/300 Write block 2045 len 8192 data: 3 3
12394 097/300 Read block 487 len 28672
12395 098/300 Read block 1147 len 20480
12396 099/300 Write block 2317 len 28672 data: 5 5 4 3 2 2 2
12397 100/300 Write block 1791 len 4096 data: 3
12398 101/300 Read block 2720 len 40960
12399 102/300 Read block 1878 len 32768
12400 103/300 Read block 3219 len 8192
12401 104/300 Write block 1480 len 24576 data: 4 3 3 3 2 2
12402 105/300 Write block 4904 len 32768 data: 3 3 3 3 2 2 3 3
12403 106/300 Write block 3312 len 40960 data: 3 3 3 3 3 2 2 2 2 4
12404 107/300 Write block 1513 len 36864 data: 4 3 3 3 3 3 3 3 4
12405 108/300 Flush
12406 109/300 Flush
12407 110/300 Flush
12408 111/300 Read block 2178 len 32768
12409 112/300 Read block 2134 len 4096
12410 113/300 Read block 2261 len 4096
12411 114/300 Write block 2597 len 24576 data: 3 3 3 3 2 2
12412 115/300 Read block 4869 len 40960
12413 116/300 Read block 3427 len 36864
12414 117/300 Write block 2249 len 24576 data: 3 3 3 3 3 2
12415 118/300 Read block 2384 len 32768
12416 119/300 Read block 2181 len 24576
12417 120/300 Flush
12418 121/300 Write block 464 len 36864 data: 4 3 3 4 4 3 3 2 3
12419 122/300 Read block 4542 len 4096
12420 123/300 Read block 3895 len 36864
12421 124/300 Flush
12422 125/300 Write block 867 len 36864 data: 2 2 2 3 3 3 3 4 4
12423 126/300 Write block 4318 len 36864 data: 4 3 2 3 3 3 3 3 4
12424 127/300 Read block 1594 len 28672
12425 128/300 Read block 2269 len 4096
12426 129/300 Read block 3152 len 32768
12427 130/300 Read block 2427 len 24576
12428 131/300 Read block 4143 len 24576
12429 132/300 Read block 429 len 20480
12430 133/300 Write block 701 len 20480 data: 4 4 4 3 3
12431 134/300 Read block 3440 len 40960
12432 135/300 Flush
12433 136/300 Read block 607 len 12288
12434 137/300 Flush
12435 138/300 Write block 3752 len 32768 data: 4 4 4 4 4 4 4 5
12436 139/300 Write block 4814 len 20480 data: 3 4 4 4 4
12437 140/300 Read block 97 len 16384
12438 141/300 Write block 620 len 4096 data: 3
12439 142/300 Read block 1284 len 40960
12440 143/300 Read block 2331 len 24576
12441 144/300 Write block 2864 len 24576 data: 4 4 3 3 3 3
12442 145/300 Write block 3806 len 40960 data: 3 3 3 3 3 3 3 3 3 3
12443 146/300 Read block 2372 len 12288
12444 147/300 Read block 1171 len 32768
12445 148/300 Read block 4071 len 8192
12446 149/300 Write block 1663 len 36864 data: 3 3 3 3 4 3 3 4 4
12447 150/300 Write block 1471 len 8192 data: 4 4
12448 151/300 Write block 1653 len 28672 data: 2 2 2 2 2 2 2
12449 152/300 Write block 833 len 8192 data: 2 2
12450 153/300 Read block 3151 len 20480
12451 154/300 Read block 4932 len 32768
12452 155/300 Write block 318 len 16384 data: 4 3 3 3
12453 156/300 Read block 1419 len 40960
12454 157/300 Write block 4176 len 36864 data: 4 4 5 5 4 4 4 3 2
12455 158/300 Read block 4277 len 40960
12456 159/300 Write block 3537 len 24576 data: 4 5 4 4 4 5
12457 160/300 Write block 660 len 12288 data: 3 3 3
12458 161/300 Write block 4712 len 16384 data: 3 3 4 4
12459 162/300 Write block 1726 len 24576 data: 4 2 3 2 2 3
12460 163/300 Write block 509 len 24576 data: 4 3 3 3 2 2
12461 164/300 Read block 2678 len 12288
12462 165/300 Read block 964 len 8192
12463 166/300 Write block 3993 len 36864 data: 2 2 2 2 2 3 3 3 3
12464 167/300 Read block 671 len 40960
12465 168/300 Write block 2404 len 20480 data: 4 4 4 3 2
12466 169/300 Write block 3915 len 12288 data: 2 2 2
12467 170/300 Flush
12468 171/300 Write block 2273 len 36864 data: 2 2 3 3 3 3 3 3 4
12469 172/300 Read block 1648 len 4096
12470 173/300 Read block 4904 len 12288
12471 174/300 Write block 2065 len 24576 data: 3 4 4 4 4 4
12472 175/300 Write block 1687 len 20480 data: 3 3 3 3 2
12473 176/300 Read block 1585 len 20480
12474 177/300 Write block 1962 len 28672 data: 3 3 3 4 5 5 5
12475 178/300 Write block 2923 len 40960 data: 3 4 4 3 3 3 3 2 2 3
12476 179/300 Write block 2996 len 36864 data: 2 2 3 2 2 3 3 3 4
12477 180/300 Write block 2900 len 20480 data: 4 4 4 3 3
12478 181/300 Read block 3152 len 28672
12479 182/300 Read block 1743 len 24576
12480 183/300 Write block 3045 len 4096 data: 4
12481 184/300 Write block 154 len 16384 data: 8 7 5 5
12482 185/300 Read block 872 len 24576
12483 186/300 Write block 4941 len 40960 data: 5 5 4 4 3 4 3 3 3 3
12484 187/300 Read block 847 len 28672
12485 188/300 Write block 4750 len 12288 data: 3 4 4
12486 189/300 Read block 1790 len 40960
12487 190/300 Write block 1265 len 12288 data: 2 2 2
12488 191/300 Read block 1568 len 16384
12489 192/300 Write block 588 len 8192 data: 3 2
12490 193/300 Read block 2091 len 28672
12491 194/300 Read block 3052 len 8192
12492 195/300 Write block 4229 len 32768 data: 4 4 5 4 3 3 2 2
12493 196/300 Read block 2811 len 32768
12494 197/300 Read block 2886 len 36864
12495 198/300 Read block 115 len 8192
12496 199/300 Read block 2556 len 36864
12497 200/300 Write block 3595 len 32768 data: 3 2 2 2 2 3 3 3
12498 201/300 Read block 4780 len 40960
12499 202/300 Write block 5 len 24576 data: 4 4 3 3 3 3
12500 203/300 Write block 2947 len 36864 data: 3 3 3 3 3 3 3 2 2
12501 204/300 Write block 4706 len 20480 data: 2 2 2 2 3
12502 205/300 Write block 2577 len 40960 data: 5 5 3 3 3 4 5 6 6 6
12503 206/300 Write block 2098 len 36864 data: 2 2 2 2 2 2 2 2 2
12504 207/300 Read block 1369 len 8192
12505 208/300 Write block 3017 len 4096 data: 5
12506 209/300 Read block 4757 len 12288
12507 210/300 Read block 625 len 4096
12508 211/300 Read block 3001 len 36864
12509 212/300 Write block 4842 len 8192 data: 3 4
12510 213/300 Read block 2201 len 28672
12511 214/300 Read block 3143 len 36864
12512 215/300 Read block 4804 len 4096
12513 216/300 Write block 3437 len 32768 data: 3 3 2 3 2 2 3 3
12514 217/300 Flush
12515 218/300 Write block 1905 len 40960 data: 3 4 5 6 6 5 3 2 2 2
12516 219/300 Read block 171 len 12288
12517 220/300 Read block 1878 len 40960
12518 221/300 Write block 2158 len 24576 data: 2 2 2 2 2 2
12519 222/300 Write block 4279 len 4096 data: 2
12520 223/300 Read block 3773 len 4096
12521 224/300 Flush
12522 225/300 Read block 1942 len 36864
12523 226/300 Read block 696 len 36864
12524 227/300 Write block 2997 len 4096 data: 3
12525 228/300 Read block 3359 len 28672
12526 229/300 Write block 1362 len 4096 data: 4
12527 230/300 Write block 3313 len 16384 data: 4 4 4 4
12528 231/300 Read block 4805 len 32768
12529 232/300 Read block 2318 len 4096
12530 233/300 Write block 4020 len 12288 data: 3 3 3
12531 234/300 Read block 115 len 32768
12532 235/300 Write block 4776 len 16384 data: 2 2 2 2
12533 236/300 Write block 4140 len 12288 data: 5 4 5
12534 237/300 Write block 452 len 8192 data: 5 5
12535 238/300 Read block 4924 len 4096
12536 239/300 Read block 3694 len 16384
12537 240/300 Read block 1026 len 16384
12538 241/300 Write block 3802 len 8192 data: 2 2
12539 242/300 Read block 148 len 16384
12540 243/300 Flush
12541 244/300 Read block 4060 len 20480
12542 245/300 Read block 3349 len 8192
12543 246/300 Flush
12544 247/300 Flush
12545 248/300 Write block 1651 len 24576 data: 2 2 3 3 3 3
12546 249/300 Read block 220 len 16384
12547 250/300 Read block 2711 len 32768
12548 251/300 Read block 444 len 20480
12549 252/300 Read block 2931 len 28672
12550 253/300 Flush
12551 254/300 Write block 3649 len 8192 data: 2 2
12552 255/300 Write block 1705 len 24576 data: 3 2 2 2 4 7
12553 256/300 Write block 2563 len 12288 data: 2 3 3
12554 257/300 Flush
12555 258/300 Read block 2235 len 36864
12556 259/300 Write block 3097 len 16384 data: 3 3 3 3
12557 260/300 Write block 4650 len 8192 data: 3 4
12558 261/300 Write block 4739 len 12288 data: 5 5 5
12559 262/300 Read block 956 len 28672
12560 263/300 Read block 3585 len 28672
12561 264/300 Write block 4280 len 20480 data: 2 2 2 2 2
12562 265/300 Write block 3559 len 20480 data: 2 2 3 3 3
12563 266/300 Flush
12564 267/300 Write block 2854 len 40960 data: 3 3 3 3 4 4 4 4 3 3
12565 268/300 Write block 4894 len 32768 data: 2 3 3 3 3 3 3 3
12566 269/300 Write block 4583 len 12288 data: 3 3 2
12567 270/300 Read block 3956 len 8192
12568 271/300 Read block 3959 len 20480
12569 272/300 Read block 4766 len 8192
12570 273/300 Read block 3312 len 8192
12571 274/300 Write block 715 len 20480 data: 2 2 2 2 2
12572 275/300 Read block 3422 len 16384
12573 276/300 Read block 3275 len 8192
12574 277/300 Read block 4454 len 4096
12575 278/300 Read block 3985 len 8192
12576 279/300 Read block 2887 len 24576
12577 280/300 Read block 2103 len 40960
12578 281/300 Write block 4985 len 28672 data: 2 2 4 4 4 4 4
12579 282/300 Write block 1425 len 4096 data: 3
12580 283/300 Read block 4690 len 36864
12581 284/300 Write block 2898 len 16384 data: 3 3 5 5
12582 285/300 Read block 2003 len 36864
12583 286/300 Flush
12584 287/300 Read block 4157 len 16384
12585 288/300 Read block 745 len 12288
12586 289/300 Write block 2823 len 40960 data: 2 2 2 2 3 3 3 3 4 3
12587 290/300 Read block 4978 len 16384
12588 291/300 Write block 819 len 20480 data: 2 2 2 2 3
12589 292/300 Write block 2358 len 12288 data: 2 3 3
12590 293/300 Write block 3107 len 36864 data: 3 3 3 3 4 4 5 6 6
12591 294/300 Write block 987 len 32768 data: 3 3 3 3 3 2 2 2
12592 295/300 Write block 2286 len 12288 data: 3 4 5
12593 296/300 Write block 359 len 8192 data: 2 2
12594 297/300 Read block 2545 len 40960
12595 298/300 Read block 2501 len 32768
12596 299/300 Read block 128 len 16384
12597 300/300 Read block 4017 len 40960
12598 301/300 Read block 3244 len 36864
12599 Main task has work to do, go find it
12600 got dsc Start(1)
12601 start 1
12602 Mar 17 22:21:48.609 INFO request completed, latency_us: 210, response_code: 204, uri: /start/cid/1, method: POST, req_id: 5461aa91-1cc8-4697-bf62-4c43bdde5dc0, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
12603 Replay: started 1, returned:Ok(())
12604 ----------------------------------------------------------------
12605 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12606 States: Active Offline Active
12607 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12608 3717 Acked Read 6 Done Sent Done false
12609 3718 Acked Read 5 Done Sent Done false
12610 3719 Acked Read 10 Done Sent Done false
12611 3720 Acked Read 8 Done Sent Done false
12612 3721 Acked Read 1 Done Sent Done false
12613 3722 Acked Write 1 Done Sent Done false
12614 3723 Acked Read 3 Done Sent Done false
12615 3724 Acked Write 8 Done Sent Done false
12616 3725 Acked Read 8 Done Sent Done false
12617 3726 Acked Write 2 Done Sent Done false
12618 3727 Acked Read 1 Done Sent Done false
12619 3728 Acked Write 7 Done Sent Done false
12620 3729 Acked Read 6 Done Sent Done false
12621 3730 Acked Write 10 Done Sent Done false
12622 3731 Acked Write 8 Done Sent Done false
12623 3732 Acked Write 6 Done Sent Done false
12624 3733 Acked Read 3 Done Sent Done false
12625 3734 Acked Read 5 Done Sent Done false
12626 3735 Acked Read 10 Done Sent Done false
12627 3736 Acked Read 2 Done Sent Done false
12628 3737 Acked Flush 0 Done Sent Done false
12629 3738 Acked Read 8 Done Sent Done false
12630 3739 Acked Write 1 Done Sent Done false
12631 3740 Acked Flush 0 Done Sent Done false
12632 3741 Acked Read 7 Done Sent Done false
12633 3742 Acked Read 2 Done Sent Done false
12634 3743 Acked Write 4 Done Sent Done false
12635 3744 Acked Read 1 Done Sent Done false
12636 3745 Acked Write 4 Done Sent Done false
12637 3746 Acked Write 1 Done Sent Done false
12638 3747 Acked Write 8 Done Sent Done false
12639 3748 Acked Write 6 Done Sent Done false
12640 3749 Acked Read 4 Done Sent Done false
12641 3750 Acked Read 6 Done Sent Done false
12642 3751 Acked Read 3 Done Sent Done false
12643 3752 Acked Write 9 Done Sent Done false
12644 3753 Acked Flush 0 Done Sent Done false
12645 3754 Acked Read 7 Done Sent Done false
12646 3755 Acked Read 3 Done Sent Done false
12647 3756 Acked Read 5 Done Sent Done false
12648 3757 Acked Write 1 Done Sent Done false
12649 3758 Acked Write 5 Done Sent Done false
12650 3759 Acked Flush 0 Done Sent Done false
12651 3760 Acked Read 10 Done Sent Done false
12652 3761 Acked Write 7 Done Sent Done false
12653 3762 Acked Write 4 Done Sent Done false
12654 3763 Acked Write 5 Done Sent Done false
12655 3764 Acked Write 5 Done Sent Done false
12656 3765 Acked Flush 0 Done Sent Done false
12657 3766 Acked Write 6 Done Sent Done false
12658 3767 Acked Write 7 Done Sent Done false
12659 3768 Acked Write 2 Done Sent Done false
12660 3769 Acked Write 10 Done Sent Done false
12661 3770 Acked Write 3 Done Sent Done false
12662 3771 Acked Write 8 Done Sent Done false
12663 3772 Acked Write 7 Done Sent Done false
12664 3773 Acked Read 2 Done Sent Done false
12665 3774 Acked Read 3 Done Sent Done false
12666 3775 Acked Read 5 Done Sent Done false
12667 3776 Acked Write 1 Done Sent Done false
12668 3777 Acked Read 5 Done Sent Done false
12669 3778 Acked Read 4 Done Sent Done false
12670 3779 Acked Flush 0 Done Sent Done false
12671 3780 Acked Read 7 Done Sent Done false
12672 3781 Acked Read 7 Done Sent Done false
12673 3782 Acked Write 9 Done Sent Done false
12674 3783 Acked Write 5 Done Sent Done false
12675 3784 Acked Read 9 Done Sent Done false
12676 3785 Acked Write 2 Done Sent Done false
12677 3786 Acked Write 5 Done Sent Done false
12678 3787 Acked Read 10 Done Sent Done false
12679 3788 Acked Read 6 Done Sent Done false
12680 3789 Acked Read 5 Done Sent Done false
12681 3790 Acked Write 6 Done Sent Done false
12682 3791 Acked Write 6 Done Sent Done false
12683 3792 Acked Read 6 Done Sent Done false
12684 3793 Acked Write 5 Done Sent Done false
12685 3794 Acked Write 5 Done Sent Done false
12686 3795 Acked Write 5 Done Sent Done false
12687 3796 Acked Read 3 Done Sent Done false
12688 3797 Acked Write 10 Done Sent Done false
12689 3798 Acked Read 6 Done Sent Done false
12690 3799 Acked Read 5 Done Sent Done false
12691 3800 Acked Write 3 Done Sent Done false
12692 3801 Acked Write 9 Done Sent Done false
12693 3802 Acked Write 10 Done Sent Done false
12694 3803 Acked Write 8 Done Sent Done false
12695 3804 Acked Write 7 Done Sent Done false
12696 3805 Acked Write 8 Done Sent Done false
12697 3806 Acked Read 3 Done Sent Done false
12698 3807 Acked Write 8 Done Sent Done false
12699 3808 Acked Read 3 Done Sent Done false
12700 3809 Acked Write 9 Done Sent Done false
12701 3810 Acked Flush 0 Done Sent Done false
12702 3811 Acked Read 1 Done Sent Done false
12703 3812 Acked Write 2 Done Sent Done false
12704 3813 Acked Read 7 Done Sent Done false
12705 3814 Acked Read 5 Done Sent Done false
12706 3815 Acked Write 7 Done Sent Done false
12707 3816 Acked Write 1 Done Sent Done false
12708 3817 Acked Read 10 Done Sent Done false
12709 3818 Acked Read 8 Done Sent Done false
12710 3819 Acked Read 2 Done Sent Done false
12711 3820 Acked Write 6 Done Sent Done false
12712 3821 Acked Write 8 Done Sent Done false
12713 3822 Acked Write 10 Done Sent Done false
12714 3823 Acked Write 9 Done Sent Done false
12715 3824 Acked Flush 0 Done Sent Done false
12716 3825 Acked Flush 0 Done Sent Done false
12717 3826 Acked Flush 0 Done Sent Done false
12718 3827 Acked Read 8 Done Sent Done false
12719 3828 Acked Read 1 Done Sent Done false
12720 3829 Acked Read 1 Done Sent Done false
12721 3830 Acked Write 6 Done Sent Done false
12722 3831 Acked Read 10 Done Sent Done false
12723 3832 Acked Read 9 Done Sent Done false
12724 3833 Acked Write 6 Done Sent Done false
12725 3834 Acked Read 8 Done Sent Done false
12726 3835 Acked Read 6 Done Sent Done false
12727 3836 Acked Flush 0 Done Sent Done false
12728 3837 Acked Write 9 Done Sent Done false
12729 3838 Acked Read 1 Done Sent Done false
12730 3839 Acked Read 9 Done Sent Done false
12731 3840 Acked Flush 0 Done Sent Done false
12732 3841 Acked Write 9 Done Sent Done false
12733 3842 Acked Write 9 Done Sent Done false
12734 3843 Acked Read 7 Done Sent Done false
12735 3844 Acked Read 1 Done Sent Done false
12736 3845 Acked Read 8 Done Sent Done false
12737 3846 Acked Read 6 Done Sent Done false
12738 3847 Acked Read 6 Done Sent Done false
12739 3848 Acked Read 5 Done Sent Done false
12740 3849 Acked Write 5 Done Sent Done false
12741 3850 Acked Read 10 Done Sent Done false
12742 3851 Acked Flush 0 Done Sent Done false
12743 3852 Acked Read 3 Done Sent Done false
12744 3853 Acked Flush 0 Done Sent Done false
12745 3854 Acked Write 8 Done Sent Done false
12746 3855 Acked Write 5 Done Sent Done false
12747 3856 Acked Read 4 Done Sent Done false
12748 3857 Acked Write 1 Done Sent Done false
12749 3858 Acked Read 10 Done Sent Done false
12750 3859 Acked Read 6 Done Sent Done false
12751 3860 Acked Write 6 Done Sent Done false
12752 3861 Acked Write 10 Done Sent Done false
12753 3862 Acked Read 3 Done Sent Done false
12754 3863 Acked Read 8 Done Sent Done false
12755 3864 Acked Read 2 Done Sent Done false
12756 3865 Acked Write 9 Done Sent Done false
12757 3866 Acked Write 2 Done Sent Done false
12758 3867 Acked Write 7 Done Sent Done false
12759 3868 Acked Write 2 Done Sent Done false
12760 3869 Acked Read 5 Done Sent Done false
12761 3870 Acked Read 8 Done Sent Done false
12762 3871 Acked Write 4 Done Sent Done false
12763 3872 Acked Read 10 Done Sent Done false
12764 3873 Acked Write 9 Done Sent Done false
12765 3874 Acked Read 10 Done Sent Done false
12766 3875 Acked Write 6 Done Sent Done false
12767 3876 Acked Write 3 Done Sent Done false
12768 3877 Acked Write 4 Done Sent Done false
12769 3878 Acked Write 6 Done Sent Done false
12770 3879 Acked Write 6 Done Sent Done false
12771 3880 Acked Read 3 Done Sent Done false
12772 3881 Acked Read 2 Done Sent Done false
12773 3882 Acked Write 9 Done Sent Done false
12774 3883 Acked Read 10 Done Sent Done false
12775 3884 Acked Write 5 Done Sent Done false
12776 3885 Acked Write 3 Done Sent Done false
12777 3886 Acked Flush 0 Done Sent Done false
12778 3887 Acked Write 9 Done Sent Done false
12779 3888 Acked Read 1 Done Sent Done false
12780 3889 Acked Read 3 Done Sent Done false
12781 3890 Acked Write 6 Done Sent Done false
12782 3891 Acked Write 5 Done Sent Done false
12783 3892 Acked Read 5 Done Sent Done false
12784 3893 Acked Write 7 Done Sent Done false
12785 3894 Acked Write 10 Done Sent Done false
12786 3895 Acked Write 9 Done Sent Done false
12787 3896 Acked Write 5 Done Sent Done false
12788 3897 Acked Read 7 Done Sent Done false
12789 3898 Acked Read 6 Done Sent Done false
12790 3899 Acked Write 1 Done Sent Done false
12791 3900 Acked Write 4 Done Sent Done false
12792 3901 Acked Read 6 Done Sent Done false
12793 3902 Acked Write 10 Done Sent Done false
12794 3903 Acked Read 7 Done Sent Done false
12795 3904 Acked Write 3 Done Sent Done false
12796 3905 Acked Read 10 Done Sent Done false
12797 3906 Acked Write 3 Done Sent Done false
12798 3907 Acked Read 4 Done Sent Done false
12799 3908 Acked Write 2 Done Sent Done false
12800 3909 Acked Read 7 Done Sent Done false
12801 3910 Acked Read 2 Done Sent Done false
12802 3911 Acked Write 8 Done Sent Done false
12803 3912 Acked Read 8 Done Sent Done false
12804 3913 Acked Read 9 Done Sent Done false
12805 3914 Acked Read 2 Done Sent Done false
12806 3915 Acked Read 9 Done Sent Done false
12807 3916 Acked Write 8 Done Sent Done false
12808 3917 Acked Read 10 Done Sent Done false
12809 3918 Acked Write 6 Done Sent Done false
12810 3919 Acked Write 9 Done Sent Done false
12811 3920 Acked Write 5 Done Sent Done false
12812 3921 Acked Write 10 Done Sent Done false
12813 3922 Acked Write 9 Done Sent Done false
12814 3923 Acked Read 2 Done Sent Done false
12815 3924 Acked Write 1 Done Sent Done false
12816 3925 Acked Read 3 Done Sent Done false
12817 3926 Acked Read 1 Done Sent Done false
12818 3927 Acked Read 9 Done Sent Done false
12819 3928 Acked Write 2 Done Sent Done false
12820 3929 Acked Read 7 Done Sent Done false
12821 3930 Acked Read 9 Done Sent Done false
12822 3931 Acked Read 1 Done Sent Done false
12823 3932 Acked Write 8 Done Sent Done false
12824 3933 Acked Flush 0 Done Sent Done false
12825 3934 Acked Write 10 Done Sent Done false
12826 3935 Acked Read 3 Done Sent Done false
12827 3936 Acked Read 10 Done Sent Done false
12828 3937 Acked Write 6 Done Sent Done false
12829 3938 Acked Write 1 Done Sent Done false
12830 3939 Acked Read 1 Done Sent Done false
12831 3940 Acked Flush 0 Done Sent Done false
12832 3941 Acked Read 9 Done Sent Done false
12833 3942 Acked Read 9 Done Sent Done false
12834 3943 Acked Write 1 Done Sent Done false
12835 3944 Acked Read 7 Done Sent Done false
12836 3945 Acked Write 1 Done Sent Done false
12837 3946 Acked Write 4 Done Sent Done false
12838 3947 Acked Read 8 Done Sent Done false
12839 3948 Acked Read 1 Done Sent Done false
12840 3949 Acked Write 3 Done Sent Done false
12841 3950 Acked Read 8 Done Sent Done false
12842 3951 Acked Write 4 Done Sent Done false
12843 3952 Acked Write 3 Done Sent Done false
12844 3953 Acked Write 2 Done Sent Done false
12845 3954 Acked Read 1 Done Sent Done false
12846 3955 Acked Read 4 Done Sent Done false
12847 3956 Acked Read 4 Done Sent Done false
12848 3957 Acked Write 2 Done Sent Done false
12849 3958 Acked Read 4 Done Sent Done false
12850 3959 Acked Flush 0 Done Sent Done false
12851 3960 Acked Read 5 Done Sent Done false
12852 3961 Acked Read 2 Done Sent Done false
12853 3962 Acked Flush 0 Done Sent Done false
12854 3963 Acked Flush 0 Done Sent Done false
12855 3964 Acked Write 6 Done Sent Done false
12856 3965 Acked Read 4 Done Sent Done false
12857 3966 Acked Read 8 Done Sent Done false
12858 3967 Acked Read 5 Done Sent Done false
12859 3968 Acked Read 7 Done Sent Done false
12860 3969 Acked Flush 0 Done Sent Done false
12861 3970 Acked Write 2 Done Sent Done false
12862 3971 Acked Write 6 Done Sent Done false
12863 3972 Acked Write 3 Done Sent Done false
12864 3973 Acked Flush 0 Done Sent Done false
12865 3974 Acked Read 9 Done Sent Done false
12866 3975 Acked Write 4 Done Sent Done false
12867 3976 Acked Write 2 Done Sent Done false
12868 3977 Acked Write 3 Done Sent Done false
12869 3978 Acked Read 7 Done Sent Done false
12870 3979 Acked Read 7 Done Sent Done false
12871 3980 Acked Write 5 Done Sent Done false
12872 3981 Acked Write 5 Done Sent Done false
12873 3982 Acked Flush 0 Done Sent Done false
12874 3983 Acked Write 10 Done Sent Done false
12875 3984 Acked Write 8 Done Sent Done false
12876 3985 Acked Write 3 Done Sent Done false
12877 3986 Acked Read 2 Done Sent Done false
12878 3987 Acked Read 5 Done Sent Done false
12879 3988 Acked Read 2 Done Sent Done false
12880 3989 Acked Read 2 Done Sent Done false
12881 3990 Acked Write 5 Done Sent Done false
12882 3991 Acked Read 4 Done Sent Done false
12883 3992 Acked Read 2 Done Sent Done false
12884 3993 Acked Read 1 Done Sent Done false
12885 3994 Acked Read 2 Done Sent Done false
12886 3995 Acked Read 6 Done Sent Done false
12887 3996 Acked Read 10 Done Sent Done false
12888 3997 Acked Write 7 Done Sent Done false
12889 3998 Acked Write 1 Done Sent Done false
12890 3999 Acked Read 9 Done Sent Done false
12891 4000 Acked Write 4 Done Sent Done false
12892 4001 Acked Read 9 Done Sent Done false
12893 4002 Acked Flush 0 Done Sent Done false
12894 4003 Acked Read 4 Done Sent Done false
12895 4004 Acked Read 3 Done Sent Done false
12896 4005 Acked Write 10 Done Sent Done false
12897 4006 Acked Read 4 Done Sent Done false
12898 4007 Acked Write 5 Done Sent Done false
12899 4008 Acked Write 3 Done Sent Done false
12900 4009 Acked Write 9 Done Sent Done false
12901 4010 Acked Write 8 Done Sent Done false
12902 4011 Acked Write 3 Done Sent Done false
12903 4012 Acked Write 2 Done Sent Done false
12904 4013 Acked Read 10 Done Sent Done false
12905 4014 Acked Read 8 Done Sent Done false
12906 4015 Acked Read 4 Done Sent Done false
12907 4016 Acked Read 10 Done Sent Done false
12908 4017 Acked Read 9 Done Sent Done false
12909 STATES DS:0 DS:1 DS:2 TOTAL
12910 Sent 0 301 0 301
12911 Done 301 0 301 602
12912 Skipped 0 0 0 0
12913 Error 0 0 0 0
12914 Last Flush: Some(JobId(4002)) Some(JobId(3716)) Some(JobId(4002))
12915 Downstairs last five completed: 3716 3715 3714 3713 3712
12916 Upstairs last five completed: 4017 4016 4015 4014 4013
12917 CLIENT: Up:0 ds:301 act:2
129182025-03-17T22:21:49.111ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
129192025-03-17T22:21:49.833ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
129202025-03-17T22:21:50.334ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
12921 [8820] got start action
12922 [8820] I am going to restart
12923 Starting downstairs at port 8820
12924 Make output file at "/tmp/dsc/downstairs-8820.txt"
12925 [8820][1] reports Starting
12926 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1221)
12927 [8820][1] reports Running
129282025-03-17T22:21:50.835ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
129292025-03-17T22:21:51.837ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
12930 ----------------------------------------------------------------
12931 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12932 States: Active Offline Active
12933 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12934 3717 Acked Read 6 Done Sent Done false
12935 3718 Acked Read 5 Done Sent Done false
12936 3719 Acked Read 10 Done Sent Done false
12937 3720 Acked Read 8 Done Sent Done false
12938 3721 Acked Read 1 Done Sent Done false
12939 3722 Acked Write 1 Done Sent Done false
12940 3723 Acked Read 3 Done Sent Done false
12941 3724 Acked Write 8 Done Sent Done false
12942 3725 Acked Read 8 Done Sent Done false
12943 3726 Acked Write 2 Done Sent Done false
12944 3727 Acked Read 1 Done Sent Done false
12945 3728 Acked Write 7 Done Sent Done false
12946 3729 Acked Read 6 Done Sent Done false
12947 3730 Acked Write 10 Done Sent Done false
12948 3731 Acked Write 8 Done Sent Done false
12949 3732 Acked Write 6 Done Sent Done false
12950 3733 Acked Read 3 Done Sent Done false
12951 3734 Acked Read 5 Done Sent Done false
12952 3735 Acked Read 10 Done Sent Done false
12953 3736 Acked Read 2 Done Sent Done false
12954 3737 Acked Flush 0 Done Sent Done false
12955 3738 Acked Read 8 Done Sent Done false
12956 3739 Acked Write 1 Done Sent Done false
12957 3740 Acked Flush 0 Done Sent Done false
12958 3741 Acked Read 7 Done Sent Done false
12959 3742 Acked Read 2 Done Sent Done false
12960 3743 Acked Write 4 Done Sent Done false
12961 3744 Acked Read 1 Done Sent Done false
12962 3745 Acked Write 4 Done Sent Done false
12963 3746 Acked Write 1 Done Sent Done false
12964 3747 Acked Write 8 Done Sent Done false
12965 3748 Acked Write 6 Done Sent Done false
12966 3749 Acked Read 4 Done Sent Done false
12967 3750 Acked Read 6 Done Sent Done false
12968 3751 Acked Read 3 Done Sent Done false
12969 3752 Acked Write 9 Done Sent Done false
12970 3753 Acked Flush 0 Done Sent Done false
12971 3754 Acked Read 7 Done Sent Done false
12972 3755 Acked Read 3 Done Sent Done false
12973 3756 Acked Read 5 Done Sent Done false
12974 3757 Acked Write 1 Done Sent Done false
12975 3758 Acked Write 5 Done Sent Done false
12976 3759 Acked Flush 0 Done Sent Done false
12977 3760 Acked Read 10 Done Sent Done false
12978 3761 Acked Write 7 Done Sent Done false
12979 3762 Acked Write 4 Done Sent Done false
12980 3763 Acked Write 5 Done Sent Done false
12981 3764 Acked Write 5 Done Sent Done false
12982 3765 Acked Flush 0 Done Sent Done false
12983 3766 Acked Write 6 Done Sent Done false
12984 3767 Acked Write 7 Done Sent Done false
12985 3768 Acked Write 2 Done Sent Done false
12986 3769 Acked Write 10 Done Sent Done false
12987 3770 Acked Write 3 Done Sent Done false
12988 3771 Acked Write 8 Done Sent Done false
12989 3772 Acked Write 7 Done Sent Done false
12990 3773 Acked Read 2 Done Sent Done false
12991 3774 Acked Read 3 Done Sent Done false
12992 3775 Acked Read 5 Done Sent Done false
12993 3776 Acked Write 1 Done Sent Done false
12994 3777 Acked Read 5 Done Sent Done false
12995 3778 Acked Read 4 Done Sent Done false
12996 3779 Acked Flush 0 Done Sent Done false
12997 3780 Acked Read 7 Done Sent Done false
12998 3781 Acked Read 7 Done Sent Done false
12999 3782 Acked Write 9 Done Sent Done false
13000 3783 Acked Write 5 Done Sent Done false
13001 3784 Acked Read 9 Done Sent Done false
13002 3785 Acked Write 2 Done Sent Done false
13003 3786 Acked Write 5 Done Sent Done false
13004 3787 Acked Read 10 Done Sent Done false
13005 3788 Acked Read 6 Done Sent Done false
13006 3789 Acked Read 5 Done Sent Done false
13007 3790 Acked Write 6 Done Sent Done false
13008 3791 Acked Write 6 Done Sent Done false
13009 3792 Acked Read 6 Done Sent Done false
13010 3793 Acked Write 5 Done Sent Done false
13011 3794 Acked Write 5 Done Sent Done false
13012 3795 Acked Write 5 Done Sent Done false
13013 3796 Acked Read 3 Done Sent Done false
13014 3797 Acked Write 10 Done Sent Done false
13015 3798 Acked Read 6 Done Sent Done false
13016 3799 Acked Read 5 Done Sent Done false
13017 3800 Acked Write 3 Done Sent Done false
13018 3801 Acked Write 9 Done Sent Done false
13019 3802 Acked Write 10 Done Sent Done false
13020 3803 Acked Write 8 Done Sent Done false
13021 3804 Acked Write 7 Done Sent Done false
13022 3805 Acked Write 8 Done Sent Done false
13023 3806 Acked Read 3 Done Sent Done false
13024 3807 Acked Write 8 Done Sent Done false
13025 3808 Acked Read 3 Done Sent Done false
13026 3809 Acked Write 9 Done Sent Done false
13027 3810 Acked Flush 0 Done Sent Done false
13028 3811 Acked Read 1 Done Sent Done false
13029 3812 Acked Write 2 Done Sent Done false
13030 3813 Acked Read 7 Done Sent Done false
13031 3814 Acked Read 5 Done Sent Done false
13032 3815 Acked Write 7 Done Sent Done false
13033 3816 Acked Write 1 Done Sent Done false
13034 3817 Acked Read 10 Done Sent Done false
13035 3818 Acked Read 8 Done Sent Done false
13036 3819 Acked Read 2 Done Sent Done false
13037 3820 Acked Write 6 Done Sent Done false
13038 3821 Acked Write 8 Done Sent Done false
13039 3822 Acked Write 10 Done Sent Done false
13040 3823 Acked Write 9 Done Sent Done false
13041 3824 Acked Flush 0 Done Sent Done false
13042 3825 Acked Flush 0 Done Sent Done false
13043 3826 Acked Flush 0 Done Sent Done false
13044 3827 Acked Read 8 Done Sent Done false
13045 3828 Acked Read 1 Done Sent Done false
13046 3829 Acked Read 1 Done Sent Done false
13047 3830 Acked Write 6 Done Sent Done false
13048 3831 Acked Read 10 Done Sent Done false
13049 3832 Acked Read 9 Done Sent Done false
13050 3833 Acked Write 6 Done Sent Done false
13051 3834 Acked Read 8 Done Sent Done false
13052 3835 Acked Read 6 Done Sent Done false
13053 3836 Acked Flush 0 Done Sent Done false
13054 3837 Acked Write 9 Done Sent Done false
13055 3838 Acked Read 1 Done Sent Done false
13056 3839 Acked Read 9 Done Sent Done false
13057 3840 Acked Flush 0 Done Sent Done false
13058 3841 Acked Write 9 Done Sent Done false
13059 3842 Acked Write 9 Done Sent Done false
13060 3843 Acked Read 7 Done Sent Done false
13061 3844 Acked Read 1 Done Sent Done false
13062 3845 Acked Read 8 Done Sent Done false
13063 3846 Acked Read 6 Done Sent Done false
13064 3847 Acked Read 6 Done Sent Done false
13065 3848 Acked Read 5 Done Sent Done false
13066 3849 Acked Write 5 Done Sent Done false
13067 3850 Acked Read 10 Done Sent Done false
13068 3851 Acked Flush 0 Done Sent Done false
13069 3852 Acked Read 3 Done Sent Done false
13070 3853 Acked Flush 0 Done Sent Done false
13071 3854 Acked Write 8 Done Sent Done false
13072 3855 Acked Write 5 Done Sent Done false
13073 3856 Acked Read 4 Done Sent Done false
13074 3857 Acked Write 1 Done Sent Done false
13075 3858 Acked Read 10 Done Sent Done false
13076 3859 Acked Read 6 Done Sent Done false
13077 3860 Acked Write 6 Done Sent Done false
13078 3861 Acked Write 10 Done Sent Done false
13079 3862 Acked Read 3 Done Sent Done false
13080 3863 Acked Read 8 Done Sent Done false
13081 3864 Acked Read 2 Done Sent Done false
13082 3865 Acked Write 9 Done Sent Done false
13083 3866 Acked Write 2 Done Sent Done false
13084 3867 Acked Write 7 Done Sent Done false
13085 3868 Acked Write 2 Done Sent Done false
13086 3869 Acked Read 5 Done Sent Done false
13087 3870 Acked Read 8 Done Sent Done false
13088 3871 Acked Write 4 Done Sent Done false
13089 3872 Acked Read 10 Done Sent Done false
13090 3873 Acked Write 9 Done Sent Done false
13091 3874 Acked Read 10 Done Sent Done false
13092 3875 Acked Write 6 Done Sent Done false
13093 3876 Acked Write 3 Done Sent Done false
13094 3877 Acked Write 4 Done Sent Done false
13095 3878 Acked Write 6 Done Sent Done false
13096 3879 Acked Write 6 Done Sent Done false
13097 3880 Acked Read 3 Done Sent Done false
13098 3881 Acked Read 2 Done Sent Done false
13099 3882 Acked Write 9 Done Sent Done false
13100 3883 Acked Read 10 Done Sent Done false
13101 3884 Acked Write 5 Done Sent Done false
13102 3885 Acked Write 3 Done Sent Done false
13103 3886 Acked Flush 0 Done Sent Done false
13104 3887 Acked Write 9 Done Sent Done false
13105 3888 Acked Read 1 Done Sent Done false
13106 3889 Acked Read 3 Done Sent Done false
13107 3890 Acked Write 6 Done Sent Done false
13108 3891 Acked Write 5 Done Sent Done false
13109 3892 Acked Read 5 Done Sent Done false
13110 3893 Acked Write 7 Done Sent Done false
13111 3894 Acked Write 10 Done Sent Done false
13112 3895 Acked Write 9 Done Sent Done false
13113 3896 Acked Write 5 Done Sent Done false
13114 3897 Acked Read 7 Done Sent Done false
13115 3898 Acked Read 6 Done Sent Done false
13116 3899 Acked Write 1 Done Sent Done false
13117 3900 Acked Write 4 Done Sent Done false
13118 3901 Acked Read 6 Done Sent Done false
13119 3902 Acked Write 10 Done Sent Done false
13120 3903 Acked Read 7 Done Sent Done false
13121 3904 Acked Write 3 Done Sent Done false
13122 3905 Acked Read 10 Done Sent Done false
13123 3906 Acked Write 3 Done Sent Done false
13124 3907 Acked Read 4 Done Sent Done false
13125 3908 Acked Write 2 Done Sent Done false
13126 3909 Acked Read 7 Done Sent Done false
13127 3910 Acked Read 2 Done Sent Done false
13128 3911 Acked Write 8 Done Sent Done false
13129 3912 Acked Read 8 Done Sent Done false
13130 3913 Acked Read 9 Done Sent Done false
13131 3914 Acked Read 2 Done Sent Done false
13132 3915 Acked Read 9 Done Sent Done false
13133 3916 Acked Write 8 Done Sent Done false
13134 3917 Acked Read 10 Done Sent Done false
13135 3918 Acked Write 6 Done Sent Done false
13136 3919 Acked Write 9 Done Sent Done false
13137 3920 Acked Write 5 Done Sent Done false
13138 3921 Acked Write 10 Done Sent Done false
13139 3922 Acked Write 9 Done Sent Done false
13140 3923 Acked Read 2 Done Sent Done false
13141 3924 Acked Write 1 Done Sent Done false
13142 3925 Acked Read 3 Done Sent Done false
13143 3926 Acked Read 1 Done Sent Done false
13144 3927 Acked Read 9 Done Sent Done false
13145 3928 Acked Write 2 Done Sent Done false
13146 3929 Acked Read 7 Done Sent Done false
13147 3930 Acked Read 9 Done Sent Done false
13148 3931 Acked Read 1 Done Sent Done false
13149 3932 Acked Write 8 Done Sent Done false
13150 3933 Acked Flush 0 Done Sent Done false
13151 3934 Acked Write 10 Done Sent Done false
13152 3935 Acked Read 3 Done Sent Done false
13153 3936 Acked Read 10 Done Sent Done false
13154 3937 Acked Write 6 Done Sent Done false
13155 3938 Acked Write 1 Done Sent Done false
13156 3939 Acked Read 1 Done Sent Done false
13157 3940 Acked Flush 0 Done Sent Done false
13158 3941 Acked Read 9 Done Sent Done false
13159 3942 Acked Read 9 Done Sent Done false
13160 3943 Acked Write 1 Done Sent Done false
13161 3944 Acked Read 7 Done Sent Done false
13162 3945 Acked Write 1 Done Sent Done false
13163 3946 Acked Write 4 Done Sent Done false
13164 3947 Acked Read 8 Done Sent Done false
13165 3948 Acked Read 1 Done Sent Done false
13166 3949 Acked Write 3 Done Sent Done false
13167 3950 Acked Read 8 Done Sent Done false
13168 3951 Acked Write 4 Done Sent Done false
13169 3952 Acked Write 3 Done Sent Done false
13170 3953 Acked Write 2 Done Sent Done false
13171 3954 Acked Read 1 Done Sent Done false
13172 3955 Acked Read 4 Done Sent Done false
13173 3956 Acked Read 4 Done Sent Done false
13174 3957 Acked Write 2 Done Sent Done false
13175 3958 Acked Read 4 Done Sent Done false
13176 3959 Acked Flush 0 Done Sent Done false
13177 3960 Acked Read 5 Done Sent Done false
13178 3961 Acked Read 2 Done Sent Done false
13179 3962 Acked Flush 0 Done Sent Done false
13180 3963 Acked Flush 0 Done Sent Done false
13181 3964 Acked Write 6 Done Sent Done false
13182 3965 Acked Read 4 Done Sent Done false
13183 3966 Acked Read 8 Done Sent Done false
13184 3967 Acked Read 5 Done Sent Done false
13185 3968 Acked Read 7 Done Sent Done false
13186 3969 Acked Flush 0 Done Sent Done false
13187 3970 Acked Write 2 Done Sent Done false
13188 3971 Acked Write 6 Done Sent Done false
13189 3972 Acked Write 3 Done Sent Done false
13190 3973 Acked Flush 0 Done Sent Done false
13191 3974 Acked Read 9 Done Sent Done false
13192 3975 Acked Write 4 Done Sent Done false
13193 3976 Acked Write 2 Done Sent Done false
13194 3977 Acked Write 3 Done Sent Done false
13195 3978 Acked Read 7 Done Sent Done false
13196 3979 Acked Read 7 Done Sent Done false
13197 3980 Acked Write 5 Done Sent Done false
13198 3981 Acked Write 5 Done Sent Done false
13199 3982 Acked Flush 0 Done Sent Done false
13200 3983 Acked Write 10 Done Sent Done false
13201 3984 Acked Write 8 Done Sent Done false
13202 3985 Acked Write 3 Done Sent Done false
13203 3986 Acked Read 2 Done Sent Done false
13204 3987 Acked Read 5 Done Sent Done false
13205 3988 Acked Read 2 Done Sent Done false
13206 3989 Acked Read 2 Done Sent Done false
13207 3990 Acked Write 5 Done Sent Done false
13208 3991 Acked Read 4 Done Sent Done false
13209 3992 Acked Read 2 Done Sent Done false
13210 3993 Acked Read 1 Done Sent Done false
13211 3994 Acked Read 2 Done Sent Done false
13212 3995 Acked Read 6 Done Sent Done false
13213 3996 Acked Read 10 Done Sent Done false
13214 3997 Acked Write 7 Done Sent Done false
13215 3998 Acked Write 1 Done Sent Done false
13216 3999 Acked Read 9 Done Sent Done false
13217 4000 Acked Write 4 Done Sent Done false
13218 4001 Acked Read 9 Done Sent Done false
13219 4002 Acked Flush 0 Done Sent Done false
13220 4003 Acked Read 4 Done Sent Done false
13221 4004 Acked Read 3 Done Sent Done false
13222 4005 Acked Write 10 Done Sent Done false
13223 4006 Acked Read 4 Done Sent Done false
13224 4007 Acked Write 5 Done Sent Done false
13225 4008 Acked Write 3 Done Sent Done false
13226 4009 Acked Write 9 Done Sent Done false
13227 4010 Acked Write 8 Done Sent Done false
13228 4011 Acked Write 3 Done Sent Done false
13229 4012 Acked Write 2 Done Sent Done false
13230 4013 Acked Read 10 Done Sent Done false
13231 4014 Acked Read 8 Done Sent Done false
13232 4015 Acked Read 4 Done Sent Done false
13233 4016 Acked Read 10 Done Sent Done false
13234 4017 Acked Read 9 Done Sent Done false
13235 STATES DS:0 DS:1 DS:2 TOTAL
13236 Sent 0 301 0 301
13237 Done 301 0 301 602
13238 Skipped 0 0 0 0
13239 Error 0 0 0 0
13240 Last Flush: Some(JobId(4002)) Some(JobId(3716)) Some(JobId(4002))
13241 Downstairs last five completed: 3716 3715 3714 3713 3712
13242 Upstairs last five completed: 4017 4016 4015 4014 4013
13243 CLIENT: Up:0 ds:301 act:2
132442025-03-17T22:21:53.114ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
132452025-03-17T22:21:53.841ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
132462025-03-17T22:21:54.371ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
132472025-03-17T22:21:54.371ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:50727) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
132482025-03-17T22:21:54.372ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
132492025-03-17T22:21:54.372ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
132502025-03-17T22:21:54.372ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
132512025-03-17T22:21:54.372ZINFOcrucible: send last flush ID to this DS: Some(JobId(3716)) = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
132522025-03-17T22:21:54.372ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
132532025-03-17T22:21:54.372ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(3716)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
132542025-03-17T22:21:54.373ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(3716)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
13255 ----------------------------------------------------------------
13256 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
13257 Downstairs last five completed: 4018 4017 4016 4015 4014
13258 Upstairs last five completed: 4018 4017 4016 4015 4014
13259 CLIENT: Up:0 ds:0 act:3
13260 Replay: All jobs finished
13261 Main task has work to do, go find it
13262 got dsc Stop(2)
13263 stop 2
13264 Mar 17 22:21:56.616 INFO request completed[8830] Got stop action so:false kr:false
13265 , latency_us: 351, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 5b115384-bd42-4936-8ad9-c38ad98388f1, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
132662025-03-17T22:21:56.616ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
132672025-03-17T22:21:56.616ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
132682025-03-17T22:21:56.617ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
132692025-03-17T22:21:56.617ZINFOcrucible: client task is exiting = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
13270 Mar 17 22:21:56.617 INFO request completed, latency_us: 171, response_code: 200, uri: /state/cid/2, method: GET, req_id: be84e116-dbbb-411c-ae5a-610fb9e952bf, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
13271 [8830] Exited with: ExitStatus(unix_wait_status(9))
13272 [8830][2] reports Exit
13273 Mar 17 22:22:00.620 INFO request completed, latency_us: 197, response_code: 200, uri: /state/cid/2, method: GET, req_id: d513bc07-58ab-4aab-8e61-bd763ac6bfaf, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
13274 001/300 Flush
13275 002/300 Flush
13276 003/300 Write block 2602 len 16384 data: 3 2 2 2
13277 004/300 Write block 728 len 8192 data: 3 3
13278 005/300 Write block 4164 len 24576 data: 5 5 6 6 5 5
13279 006/300 Read block 2982 len 8192
13280 007/300 Write block 4921 len 36864 data: 3 3 3 3 3 3 2 2 2
13281 008/300 Write block 4284 len 20480 data: 3 2 2 2 2
13282 009/300 Write block 3698 len 24576 data: 4 3 3 3 3 2
13283 010/300 Read block 3856 len 4096
13284 011/300 Read block 763 len 32768
13285 012/300 Read block 180 len 40960
13286 013/300 Write block 4691 len 16384 data: 2 3 3 3
13287 014/300 Read block 3207 len 32768
13288 015/300 Read block 1629 len 16384
13289 016/300 Read block 3768 len 28672
13290 017/300 Read block 3357 len 24576
13291 018/300 Read block 1695 len 4096
13292 019/300 Read block 2784 len 24576
13293 020/300 Read block 4383 len 24576
13294 021/300 Read block 529 len 20480
13295 022/300 Read block 947 len 36864
13296 023/300 Write block 947 len 40960 data: 6 6 6 4 4 3 3 3 2 2
13297 024/300 Write block 4430 len 32768 data: 4 4 4 3 3 3 2 2
13298 025/300 Read block 3141 len 20480
13299 026/300 Read block 4222 len 24576
13300 027/300 Write block 3742 len 28672 data: 4 4 4 4 4 4 3
13301 028/300 Write block 2413 len 32768 data: 5 5 4 4 5 4 5 5
13302 029/300 Read block 1490 len 8192
13303 030/300 Read block 1356 len 24576
13304 031/300 Write block 2392 len 12288 data: 4 4 4
13305 032/300 Read block 1715 len 16384
13306 033/300 Read block 1759 len 4096
13307 034/300 Read block 1366 len 28672
13308 035/300 Read block 2104 len 4096
13309 036/300 Read block 919 len 24576
13310 037/300 Read block 355 len 12288
13311 038/300 Write block 3318 len 24576 data: 3 3 3 5 4 4
13312 039/300 Write block 911 len 12288 data: 5 4 4
13313 040/300 Write block 3784 len 40960 data: 3 3 3 3 4 4 4 4 3 2
13314 041/300 Write block 14 len 36864 data: 2 2 2 2 2 2 2 2 2
13315 042/300 Write block 3652 len 24576 data: 3 4 4 4 4 4
13316 043/300 Flush
13317 044/300 Read block 1126 len 12288
13318 045/300 Read block 2469 len 36864
13319 046/300 Flush
13320 047/300 Write block 2342 len 20480 data: 5 5 5 5 4
13321 048/300 Write block 4152 len 36864 data: 4 4 3 4 4 4 2 2 3
13322 049/300 Write block 4409 len 32768 data: 3 3 3 3 4 4 3 3
13323 050/300 Read block 824 len 40960
13324 051/300 Write block 1987 len 32768 data: 5 5 5 5 5 5 3 4
13325 052/300 Write block 1548 len 32768 data: 3 4 4 4 4 5 5 4
13326 053/300 Read block 462 len 28672
13327 054/300 Read block 742 len 8192
13328 055/300 Read block 3159 len 16384
13329 056/300 Read block 2764 len 28672
13330 057/300 Read block 176 len 24576
13331 058/300 Read block 3654 len 36864
13332 059/300 Read block 543 len 8192
13333 060/300 Read block 1619 len 8192
13334 061/300 Write block 3066 len 40960 data: 2 2 2 2 2 2 2 2 2 2
13335 062/300 Write block 2052 len 24576 data: 3 4 3 3 4 5
13336 063/300 Write block 4179 len 24576 data: 6 5 5 5 4 3
13337 064/300 Write block 4562 len 12288 data: 3 3 3
13338 065/300 Write block 406 len 12288 data: 3 3 2
13339 066/300 Write block 73 len 28672 data: 3 3 3 3 3 4 3
13340 067/300 Read block 614 len 40960
13341 068/300 Flush
13342 069/300 Read block 3844 len 4096
13343 070/300 Write block 3029 len 4096 data: 2
13344 071/300 Write block 4831 len 4096 data: 3
13345 072/300 Read block 3017 len 24576
13346 073/300 Write block 262 len 12288 data: 3 3 3
13347 074/300 Read block 3155 len 32768
13348 075/300 Read block 2754 len 28672
13349 076/300 Read block 3727 len 40960
13350 077/300 Read block 3579 len 40960
13351 078/300 Write block 2597 len 36864 data: 4 4 4 4 3 4 3 3 3
13352 079/300 Read block 349 len 8192
13353 080/300 Write block 730 len 4096 data: 3
13354 081/300 Write block 4169 len 16384 data: 6 3 3 3
13355 082/300 Read block 3722 len 4096
13356 083/300 Write block 2765 len 12288 data: 4 4 3
13357 084/300 Read block 3856 len 4096
13358 085/300 Write block 4867 len 20480 data: 3 4 4 3 3
13359 086/300 Write block 559 len 4096 data: 2
13360 087/300 Read block 4388 len 20480
13361 088/300 Write block 1185 len 12288 data: 2 2 2
13362 089/300 Read block 525 len 40960
13363 090/300 Write block 4166 len 32768 data: 7 7 6 7 4 4 4 3
13364 091/300 Flush
13365 092/300 Read block 1298 len 40960
13366 093/300 Write block 108 len 32768 data: 3 3 3 3 3 3 2 3
13367 094/300 Read block 3642 len 8192
13368 095/300 Write block 3428 len 16384 data: 3 3 3 2
13369 096/300 Write block 2033 len 4096 data: 3
13370 097/300 Write block 1879 len 4096 data: 5
13371 098/300 Read block 291 len 28672
13372 099/300 Read block 3373 len 32768
13373 100/300 Write block 1152 len 32768 data: 3 3 3 3 4 4 5 5
13374 101/300 Read block 2488 len 12288
13375 102/300 Read block 4464 len 24576
13376 103/300 Write block 649 len 12288 data: 2 2 2
13377 104/300 Read block 3384 len 16384
13378 105/300 Read block 1872 len 8192
13379 106/300 Write block 3566 len 20480 data: 4 2 2 2 2
13380 107/300 Read block 3185 len 32768
13381 108/300 Read block 2342 len 12288
13382 109/300 Read block 3547 len 16384
13383 110/300 Flush
13384 111/300 Read block 1115 len 40960
13385 112/300 Write block 1706 len 40960 data: 3 3 3 5 8 6 5 5 5 5
13386 113/300 Write block 3178 len 16384 data: 5 3 4 4
13387 114/300 Write block 4660 len 32768 data: 4 4 4 4 3 2 3 3
13388 115/300 Read block 3142 len 8192
13389 116/300 Write block 3334 len 4096 data: 2
13390 117/300 Read block 2218 len 20480
13391 118/300 Flush
13392 119/300 Read block 848 len 4096
13393 120/300 Read block 4845 len 4096
13394 121/300 Read block 3976 len 20480
13395 122/300 Read block 3538 len 24576
13396 123/300 Flush
13397 124/300 Write block 2229 len 16384 data: 3 3 3 3
13398 125/300 Read block 3140 len 28672
13399 126/300 Flush
13400 127/300 Read block 1118 len 24576
13401 128/300 Write block 2899 len 28672 data: 4 6 6 5 4 4 3
13402 129/300 Flush
13403 130/300 Read block 3824 len 20480
13404 131/300 Write block 1434 len 24576 data: 2 2 2 2 2 2
13405 132/300 Read block 2266 len 32768
13406 133/300 Read block 1205 len 20480
13407 134/300 Write block 4854 len 32768 data: 2 2 2 3 4 3 4 4
13408 135/300 Read block 803 len 12288
13409 136/300 Flush
13410 137/300 Read block 4709 len 12288
13411 138/300 Read block 2201 len 32768
13412 139/300 Read block 4471 len 40960
13413 140/300 Write block 1148 len 32768 data: 2 2 3 3 4 4 4 4
13414 141/300 Read block 4046 len 4096
13415 142/300 Read block 2676 len 12288
13416 143/300 Read block 1551 len 24576
13417 144/300 Write block 4648 len 28672 data: 5 3 4 5 5 3 3
13418 145/300 Write block 3882 len 40960 data: 2 2 2 2 2 2 2 2 2 3
13419 146/300 Read block 741 len 36864
13420 147/300 Flush
13421 148/300 Write block 2979 len 4096 data: 2
13422 149/300 Read block 3261 len 12288
13423 150/300 Flush
13424 151/300 Read block 3852 len 32768
13425 152/300 Read block 235 len 32768
13426 153/300 Read block 369 len 40960
13427 154/300 Write block 644 len 16384 data: 2 2 4 3
13428 155/300 Write block 651 len 4096 data: 3
13429 156/300 Flush
13430 157/300 Read block 1327 len 32768
13431 158/300 Read block 1783 len 4096
13432 159/300 Read block 2822 len 24576
13433 160/300 Read block 4869 len 12288
13434 161/300 Read block 1821 len 28672
13435 162/300 Write block 4047 len 36864 data: 2 2 2 2 2 2 2 2 2
13436 163/300 Write block 3653 len 12288 data: 5 5 5
13437 164/300 Write block 3941 len 16384 data: 5 7 6 5
13438 165/300 Read block 4534 len 4096
13439 166/300 Write block 2099 len 4096 data: 3
13440 167/300 Write block 4188 len 40960 data: 3 3 4 5 4 3 4 4 6 6
13441 168/300 Read block 649 len 12288
13442 169/300 Flush
13443 170/300 Write block 1955 len 20480 data: 3 3 4 4 4
13444 171/300 Write block 3614 len 4096 data: 2
13445 172/300 Write block 329 len 4096 data: 3
13446 173/300 Read block 2906 len 12288
13447 174/300 Read block 249 len 40960
13448 175/300 Flush
13449 176/300 Write block 68 len 24576 data: 2 2 2 3 3 4
13450 177/300 Read block 2733 len 24576
13451 178/300 Write block 346 len 8192 data: 2 2
13452 179/300 Flush
13453 180/300 Write block 723 len 12288 data: 2 2 3
13454 181/300 Read block 3226 len 28672
13455 182/300 Read block 375 len 32768
13456 183/300 Read block 1649 len 32768
13457 184/300 Read block 2265 len 24576
13458 185/300 Write block 3848 len 36864 data: 3 4 4 3 4 4 4 3 3
13459 186/300 Write block 4705 len 24576 data: 2 3 3 3 3 4
13460 187/300 Write block 475 len 40960 data: 4 4 3 3 3 2 2 2 3 3
13461 188/300 Read block 4926 len 32768
13462 189/300 Flush
13463 190/300 Write block 1181 len 36864 data: 2 2 2 2 3 3 3 2 2
13464 191/300 Read block 1214 len 16384
13465 192/300 Read block 251 len 24576
13466 193/300 Read block 1699 len 32768
13467 194/300 Write block 1727 len 32768 data: 3 4 3 3 4 3 4 3
13468 195/300 Read block 4693 len 24576
13469 196/300 Read block 1197 len 4096
13470 197/300 Write block 3268 len 4096 data: 3
13471 198/300 Write block 4325 len 32768 data: 4 5 4 4 3 3 3 4
13472 199/300 Write block 1180 len 8192 data: 3 3
13473 200/300 Read block 2606 len 20480
13474 201/300 Read block 444 len 36864
13475 202/300 Flush
13476 203/300 Read block 2029 len 8192
13477 204/300 Flush
13478 205/300 Write block 1715 len 4096 data: 6
13479 206/300 Write block 4391 len 32768 data: 2 2 2 2 2 2 2 2
13480 207/300 Write block 553 len 8192 data: 3 3
13481 208/300 Write block 3802 len 36864 data: 3 3 2 2 4 4 4 4 4
13482 209/300 Write block 1869 len 40960 data: 5 5 6 6 3 2 4 4 4 4
13483 210/300 Read block 4883 len 4096
13484 211/300 Flush
13485 212/300 Read block 3674 len 8192
13486 213/300 Read block 2036 len 20480
13487 214/300 Read block 4874 len 16384
13488 215/300 Write block 4505 len 28672 data: 2 2 2 3 3 4 5
13489 216/300 Read block 3437 len 28672
13490 217/300 Write block 4254 len 16384 data: 3 3 4 5
13491 218/300 Write block 1915 len 8192 data: 2 2
13492 219/300 Flush
13493 220/300 Read block 4133 len 24576
13494 221/300 Flush
13495 222/300 Read block 896 len 4096
13496 223/300 Write block 2156 len 16384 data: 3 2 3 3
13497 224/300 Read block 2943 len 32768
13498 225/300 Write block 4746 len 16384 data: 4 3 3 3
13499 226/300 Read block 2556 len 16384
13500 227/300 Read block 3307 len 28672
13501 228/300 Read block 4212 len 32768
13502 229/300 Read block 1364 len 28672
13503 230/300 Write block 1811 len 12288 data: 4 4 4
13504 231/300 Write block 3827 len 16384 data: 2 3 3 4
13505 232/300 Read block 3610 len 16384
13506 233/300 Write block 1249 len 24576 data: 2 2 3 3 3 3
13507 234/300 Write block 3622 len 16384 data: 4 4 3 3
13508 235/300 Write block 2118 len 40960 data: 3 3 3 3 3 3 3 3 2 2
13509 236/300 Write block 1567 len 16384 data: 3 3 3 3
13510 237/300 Flush
13511 238/300 Write block 250 len 16384 data: 2 2 2 2
13512 239/300 Read block 1437 len 40960
13513 240/300 Read block 534 len 16384
13514 241/300 Read block 2197 len 32768
13515 242/300 Write block 1085 len 36864 data: 2 2 2 2 2 2 2 3 3
13516 243/300 Write block 1961 len 36864 data: 2 4 4 4 5 6 6 6 5
13517 244/300 Read block 266 len 36864
13518 245/300 Write block 2864 len 36864 data: 5 5 4 4 4 4 4 4 3
13519 246/300 Write block 956 len 20480 data: 3 3 2 2 2
13520 247/300 Flush
13521 248/300 Read block 1580 len 12288
13522 249/300 Read block 3062 len 24576
13523 250/300 Write block 4054 len 4096 data: 3
13524 251/300 Write block 1980 len 8192 data: 4 4
13525 252/300 Read block 652 len 32768
13526 253/300 Read block 2505 len 36864
13527 254/300 Write block 4471 len 16384 data: 3 3 4 3
13528 255/300 Read block 3363 len 4096
13529 256/300 Write block 4450 len 20480 data: 6 5 4 2 3
13530 257/300 Read block 637 len 16384
13531 258/300 Read block 1230 len 8192
13532 259/300 Write block 346 len 32768 data: 3 3 2 3 2 3 3 3
13533 260/300 Write block 1583 len 28672 data: 4 4 4 3 4 4 5
13534 261/300 Flush
13535 262/300 Write block 4866 len 28672 data: 5 4 5 5 4 4 3
13536 263/300 Read block 4679 len 24576
13537 264/300 Read block 1991 len 40960
13538 265/300 Read block 476 len 4096
13539 266/300 Read block 4655 len 28672
13540 267/300 Write block 430 len 36864 data: 3 3 3 3 3 4 4 3 2
13541 268/300 Write block 748 len 8192 data: 4 4
13542 269/300 Read block 2988 len 16384
13543 270/300 Write block 3116 len 8192 data: 6 6
13544 271/300 Read block 4752 len 32768
13545 272/300 Write block 1947 len 16384 data: 2 2 2 2
13546 273/300 Flush
13547 274/300 Read block 2077 len 16384
13548 275/300 Read block 92 len 12288
13549 276/300 Write block 1248 len 40960 data: 2 3 3 4 4 4 4 4 4 4
13550 277/300 Read block 4049 len 40960
13551 278/300 Write block 3749 len 40960 data: 3 3 3 5 5 5 5 5 5 5
13552 279/300 Write block 1771 len 28672 data: 2 2 2 3 3 4 4
13553 280/300 Write block 2602 len 28672 data: 5 4 4 4 2 3 3
13554 281/300 Read block 2659 len 28672
13555 282/300 Write block 3360 len 20480 data: 2 2 2 2 2
13556 283/300 Flush
13557 284/300 Read block 1593 len 28672
13558 285/300 Read block 2721 len 36864
13559 286/300 Flush
13560 287/300 Write block 1126 len 36864 data: 4 4 3 2 2 2 3 4 2
13561 288/300 Write block 2293 len 8192 data: 4 2
13562 289/300 Write block 2137 len 28672 data: 2 2 3 4 5 4 3
13563 290/300 Read block 1732 len 16384
13564 291/300 Read block 4253 len 28672
13565 292/300 Read block 4194 len 24576
13566 293/300 Write block 314 len 20480 data: 3 3 3 3 5
13567 294/300 Read block 4917 len 36864
13568 295/300 Read block 2841 len 8192
13569 296/300 Read block 4150 len 24576
13570 297/300 Flush
13571 298/300 Read block 2398 len 40960
13572 299/300 Read block 3609 len 36864
13573 300/300 Read block 262 len 32768
13574 301/300 Read block 1088 len 20480
13575 Main task has work to do, go find it
13576 got dsc Start(2)
13577 start 2
13578 Mar 17 22:22:00.864 INFO request completed, latency_us: 207, response_code: 204, uri: /start/cid/2, method: POST, req_id: d10653c3-ad67-4beb-9035-6771c13866ed, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
13579 Replay: started 2, returned:Ok(())
13580 ----------------------------------------------------------------
13581 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13582 States: Active Active Offline
13583 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13584 4019 Acked Flush 0 Done Done Sent false
13585 4020 Acked Flush 0 Done Done Sent false
13586 4021 Acked Write 4 Done Done Sent false
13587 4022 Acked Write 2 Done Done Sent false
13588 4023 Acked Write 6 Done Done Sent false
13589 4024 Acked Read 2 Done Done Sent false
13590 4025 Acked Write 9 Done Done Sent false
13591 4026 Acked Write 5 Done Done Sent false
13592 4027 Acked Write 6 Done Done Sent false
13593 4028 Acked Read 1 Done Done Sent false
13594 4029 Acked Read 8 Done Done Sent false
13595 4030 Acked Read 10 Done Done Sent false
13596 4031 Acked Write 4 Done Done Sent false
13597 4032 Acked Read 8 Done Done Sent false
13598 4033 Acked Read 4 Done Done Sent false
13599 4034 Acked Read 7 Done Done Sent false
13600 4035 Acked Read 6 Done Done Sent false
13601 4036 Acked Read 1 Done Done Sent false
13602 4037 Acked Read 6 Done Done Sent false
13603 4038 Acked Read 6 Done Done Sent false
13604 4039 Acked Read 5 Done Done Sent false
13605 4040 Acked Read 9 Done Done Sent false
13606 4041 Acked Write 10 Done Done Sent false
13607 4042 Acked Write 8 Done Done Sent false
13608 4043 Acked Read 5 Done Done Sent false
13609 4044 Acked Read 6 Done Done Sent false
13610 4045 Acked Write 7 Done Done Sent false
13611 4046 Acked Write 8 Done Done Sent false
13612 4047 Acked Read 2 Done Done Sent false
13613 4048 Acked Read 6 Done Done Sent false
13614 4049 Acked Write 3 Done Done Sent false
13615 4050 Acked Read 4 Done Done Sent false
13616 4051 Acked Read 1 Done Done Sent false
13617 4052 Acked Read 7 Done Done Sent false
13618 4053 Acked Read 1 Done Done Sent false
13619 4054 Acked Read 6 Done Done Sent false
13620 4055 Acked Read 3 Done Done Sent false
13621 4056 Acked Write 6 Done Done Sent false
13622 4057 Acked Write 3 Done Done Sent false
13623 4058 Acked Write 10 Done Done Sent false
13624 4059 Acked Write 9 Done Done Sent false
13625 4060 Acked Write 6 Done Done Sent false
13626 4061 Acked Flush 0 Done Done Sent false
13627 4062 Acked Read 3 Done Done Sent false
13628 4063 Acked Read 9 Done Done Sent false
13629 4064 Acked Flush 0 Done Done Sent false
13630 4065 Acked Write 5 Done Done Sent false
13631 4066 Acked Write 9 Done Done Sent false
13632 4067 Acked Write 8 Done Done Sent false
13633 4068 Acked Read 10 Done Done Sent false
13634 4069 Acked Write 8 Done Done Sent false
13635 4070 Acked Write 8 Done Done Sent false
13636 4071 Acked Read 7 Done Done Sent false
13637 4072 Acked Read 2 Done Done Sent false
13638 4073 Acked Read 4 Done Done Sent false
13639 4074 Acked Read 7 Done Done Sent false
13640 4075 Acked Read 6 Done Done Sent false
13641 4076 Acked Read 9 Done Done Sent false
13642 4077 Acked Read 2 Done Done Sent false
13643 4078 Acked Read 2 Done Done Sent false
13644 4079 Acked Write 10 Done Done Sent false
13645 4080 Acked Write 6 Done Done Sent false
13646 4081 Acked Write 6 Done Done Sent false
13647 4082 Acked Write 3 Done Done Sent false
13648 4083 Acked Write 3 Done Done Sent false
13649 4084 Acked Write 7 Done Done Sent false
13650 4085 Acked Read 10 Done Done Sent false
13651 4086 Acked Flush 0 Done Done Sent false
13652 4087 Acked Read 1 Done Done Sent false
13653 4088 Acked Write 1 Done Done Sent false
13654 4089 Acked Write 1 Done Done Sent false
13655 4090 Acked Read 6 Done Done Sent false
13656 4091 Acked Write 3 Done Done Sent false
13657 4092 Acked Read 8 Done Done Sent false
13658 4093 Acked Read 7 Done Done Sent false
13659 4094 Acked Read 10 Done Done Sent false
13660 4095 Acked Read 10 Done Done Sent false
13661 4096 Acked Write 9 Done Done Sent false
13662 4097 Acked Read 2 Done Done Sent false
13663 4098 Acked Write 1 Done Done Sent false
13664 4099 Acked Write 4 Done Done Sent false
13665 4100 Acked Read 1 Done Done Sent false
13666 4101 Acked Write 3 Done Done Sent false
13667 4102 Acked Read 1 Done Done Sent false
13668 4103 Acked Write 5 Done Done Sent false
13669 4104 Acked Write 1 Done Done Sent false
13670 4105 Acked Read 5 Done Done Sent false
13671 4106 Acked Write 3 Done Done Sent false
13672 4107 Acked Read 10 Done Done Sent false
13673 4108 Acked Write 8 Done Done Sent false
13674 4109 Acked Flush 0 Done Done Sent false
13675 4110 Acked Read 10 Done Done Sent false
13676 4111 Acked Write 8 Done Done Sent false
13677 4112 Acked Read 2 Done Done Sent false
13678 4113 Acked Write 4 Done Done Sent false
13679 4114 Acked Write 1 Done Done Sent false
13680 4115 Acked Write 1 Done Done Sent false
13681 4116 Acked Read 7 Done Done Sent false
13682 4117 Acked Read 8 Done Done Sent false
13683 4118 Acked Write 8 Done Done Sent false
13684 4119 Acked Read 3 Done Done Sent false
13685 4120 Acked Read 6 Done Done Sent false
13686 4121 Acked Write 3 Done Done Sent false
13687 4122 Acked Read 4 Done Done Sent false
13688 4123 Acked Read 2 Done Done Sent false
13689 4124 Acked Write 5 Done Done Sent false
13690 4125 Acked Read 8 Done Done Sent false
13691 4126 Acked Read 3 Done Done Sent false
13692 4127 Acked Read 4 Done Done Sent false
13693 4128 Acked Flush 0 Done Done Sent false
13694 4129 Acked Read 10 Done Done Sent false
13695 4130 Acked Write 10 Done Done Sent false
13696 4131 Acked Write 4 Done Done Sent false
13697 4132 Acked Write 8 Done Done Sent false
13698 4133 Acked Read 2 Done Done Sent false
13699 4134 Acked Write 1 Done Done Sent false
13700 4135 Acked Read 5 Done Done Sent false
13701 4136 Acked Flush 0 Done Done Sent false
13702 4137 Acked Read 1 Done Done Sent false
13703 4138 Acked Read 1 Done Done Sent false
13704 4139 Acked Read 5 Done Done Sent false
13705 4140 Acked Read 6 Done Done Sent false
13706 4141 Acked Flush 0 Done Done Sent false
13707 4142 Acked Write 4 Done Done Sent false
13708 4143 Acked Read 7 Done Done Sent false
13709 4144 Acked Flush 0 Done Done Sent false
13710 4145 Acked Read 6 Done Done Sent false
13711 4146 Acked Write 7 Done Done Sent false
13712 4147 Acked Flush 0 Done Done Sent false
13713 4148 Acked Read 5 Done Done Sent false
13714 4149 Acked Write 6 Done Done Sent false
13715 4150 Acked Read 8 Done Done Sent false
13716 4151 Acked Read 5 Done Done Sent false
13717 4152 Acked Write 8 Done Done Sent false
13718 4153 Acked Read 3 Done Done Sent false
13719 4154 Acked Flush 0 Done Done Sent false
13720 4155 Acked Read 3 Done Done Sent false
13721 4156 Acked Read 8 Done Done Sent false
13722 4157 Acked Read 10 Done Done Sent false
13723 4158 Acked Write 8 Done Done Sent false
13724 4159 Acked Read 1 Done Done Sent false
13725 4160 Acked Read 3 Done Done Sent false
13726 4161 Acked Read 6 Done Done Sent false
13727 4162 Acked Write 7 Done Done Sent false
13728 4163 Acked Write 10 Done Done Sent false
13729 4164 Acked Read 9 Done Done Sent false
13730 4165 Acked Flush 0 Done Done Sent false
13731 4166 Acked Write 1 Done Done Sent false
13732 4167 Acked Read 3 Done Done Sent false
13733 4168 Acked Flush 0 Done Done Sent false
13734 4169 Acked Read 8 Done Done Sent false
13735 4170 Acked Read 8 Done Done Sent false
13736 4171 Acked Read 10 Done Done Sent false
13737 4172 Acked Write 4 Done Done Sent false
13738 4173 Acked Write 1 Done Done Sent false
13739 4174 Acked Flush 0 Done Done Sent false
13740 4175 Acked Read 8 Done Done Sent false
13741 4176 Acked Read 1 Done Done Sent false
13742 4177 Acked Read 6 Done Done Sent false
13743 4178 Acked Read 3 Done Done Sent false
13744 4179 Acked Read 7 Done Done Sent false
13745 4180 Acked Write 9 Done Done Sent false
13746 4181 Acked Write 3 Done Done Sent false
13747 4182 Acked Write 4 Done Done Sent false
13748 4183 Acked Read 1 Done Done Sent false
13749 4184 Acked Write 1 Done Done Sent false
13750 4185 Acked Write 10 Done Done Sent false
13751 4186 Acked Read 3 Done Done Sent false
13752 4187 Acked Flush 0 Done Done Sent false
13753 4188 Acked Write 5 Done Done Sent false
13754 4189 Acked Write 1 Done Done Sent false
13755 4190 Acked Write 1 Done Done Sent false
13756 4191 Acked Read 3 Done Done Sent false
13757 4192 Acked Read 10 Done Done Sent false
13758 4193 Acked Flush 0 Done Done Sent false
13759 4194 Acked Write 6 Done Done Sent false
13760 4195 Acked Read 6 Done Done Sent false
13761 4196 Acked Write 2 Done Done Sent false
13762 4197 Acked Flush 0 Done Done Sent false
13763 4198 Acked Write 3 Done Done Sent false
13764 4199 Acked Read 7 Done Done Sent false
13765 4200 Acked Read 8 Done Done Sent false
13766 4201 Acked Read 8 Done Done Sent false
13767 4202 Acked Read 6 Done Done Sent false
13768 4203 Acked Write 9 Done Done Sent false
13769 4204 Acked Write 6 Done Done Sent false
13770 4205 Acked Write 10 Done Done Sent false
13771 4206 Acked Read 8 Done Done Sent false
13772 4207 Acked Flush 0 Done Done Sent false
13773 4208 Acked Write 9 Done Done Sent false
13774 4209 Acked Read 4 Done Done Sent false
13775 4210 Acked Read 6 Done Done Sent false
13776 4211 Acked Read 8 Done Done Sent false
13777 4212 Acked Write 8 Done Done Sent false
13778 4213 Acked Read 6 Done Done Sent false
13779 4214 Acked Read 1 Done Done Sent false
13780 4215 Acked Write 1 Done Done Sent false
13781 4216 Acked Write 8 Done Done Sent false
13782 4217 Acked Write 2 Done Done Sent false
13783 4218 Acked Read 5 Done Done Sent false
13784 4219 Acked Read 9 Done Done Sent false
13785 4220 Acked Flush 0 Done Done Sent false
13786 4221 Acked Read 2 Done Done Sent false
13787 4222 Acked Flush 0 Done Done Sent false
13788 4223 Acked Write 1 Done Done Sent false
13789 4224 Acked Write 8 Done Done Sent false
13790 4225 Acked Write 2 Done Done Sent false
13791 4226 Acked Write 9 Done Done Sent false
13792 4227 Acked Write 10 Done Done Sent false
13793 4228 Acked Read 1 Done Done Sent false
13794 4229 Acked Flush 0 Done Done Sent false
13795 4230 Acked Read 2 Done Done Sent false
13796 4231 Acked Read 5 Done Done Sent false
13797 4232 Acked Read 4 Done Done Sent false
13798 4233 Acked Write 7 Done Done Sent false
13799 4234 Acked Read 7 Done Done Sent false
13800 4235 Acked Write 4 Done Done Sent false
13801 4236 Acked Write 2 Done Done Sent false
13802 4237 Acked Flush 0 Done Done Sent false
13803 4238 Acked Read 6 Done Done Sent false
13804 4239 Acked Flush 0 Done Done Sent false
13805 4240 Acked Read 1 Done Done Sent false
13806 4241 Acked Write 4 Done Done Sent false
13807 4242 Acked Read 8 Done Done Sent false
13808 4243 Acked Write 4 Done Done Sent false
13809 4244 Acked Read 4 Done Done Sent false
13810 4245 Acked Read 7 Done Done Sent false
13811 4246 Acked Read 8 Done Done Sent false
13812 4247 Acked Read 7 Done Done Sent false
13813 4248 Acked Write 3 Done Done Sent false
13814 4249 Acked Write 4 Done Done Sent false
13815 4250 Acked Read 4 Done Done Sent false
13816 4251 Acked Write 6 Done Done Sent false
13817 4252 Acked Write 4 Done Done Sent false
13818 4253 Acked Write 10 Done Done Sent false
13819 4254 Acked Write 4 Done Done Sent false
13820 4255 Acked Flush 0 Done Done Sent false
13821 4256 Acked Write 4 Done Done Sent false
13822 4257 Acked Read 10 Done Done Sent false
13823 4258 Acked Read 4 Done Done Sent false
13824 4259 Acked Read 8 Done Done Sent false
13825 4260 Acked Write 9 Done Done Sent false
13826 4261 Acked Write 9 Done Done Sent false
13827 4262 Acked Read 9 Done Done Sent false
13828 4263 Acked Write 9 Done Done Sent false
13829 4264 Acked Write 5 Done Done Sent false
13830 4265 Acked Flush 0 Done Done Sent false
13831 4266 Acked Read 3 Done Done Sent false
13832 4267 Acked Read 6 Done Done Sent false
13833 4268 Acked Write 1 Done Done Sent false
13834 4269 Acked Write 2 Done Done Sent false
13835 4270 Acked Read 8 Done Done Sent false
13836 4271 Acked Read 9 Done Done Sent false
13837 4272 Acked Write 4 Done Done Sent false
13838 4273 Acked Read 1 Done Done Sent false
13839 4274 Acked Write 5 Done Done Sent false
13840 4275 Acked Read 4 Done Done Sent false
13841 4276 Acked Read 2 Done Done Sent false
13842 4277 Acked Write 8 Done Done Sent false
13843 4278 Acked Write 7 Done Done Sent false
13844 4279 Acked Flush 0 Done Done Sent false
13845 4280 Acked Write 7 Done Done Sent false
13846 4281 Acked Read 6 Done Done Sent false
13847 4282 Acked Read 10 Done Done Sent false
13848 4283 Acked Read 1 Done Done Sent false
13849 4284 Acked Read 7 Done Done Sent false
13850 4285 Acked Write 9 Done Done Sent false
13851 4286 Acked Write 2 Done Done Sent false
13852 4287 Acked Read 4 Done Done Sent false
13853 4288 Acked Write 2 Done Done Sent false
13854 4289 Acked Read 8 Done Done Sent false
13855 4290 Acked Write 4 Done Done Sent false
13856 4291 Acked Flush 0 Done Done Sent false
13857 4292 Acked Read 4 Done Done Sent false
13858 4293 Acked Read 3 Done Done Sent false
13859 4294 Acked Write 10 Done Done Sent false
13860 4295 Acked Read 10 Done Done Sent false
13861 4296 Acked Write 10 Done Done Sent false
13862 4297 Acked Write 7 Done Done Sent false
13863 4298 Acked Write 7 Done Done Sent false
13864 4299 Acked Read 7 Done Done Sent false
13865 4300 Acked Write 5 Done Done Sent false
13866 4301 Acked Flush 0 Done Done Sent false
13867 4302 Acked Read 7 Done Done Sent false
13868 4303 Acked Read 9 Done Done Sent false
13869 4304 Acked Flush 0 Done Done Sent false
13870 4305 Acked Write 9 Done Done Sent false
13871 4306 Acked Write 2 Done Done Sent false
13872 4307 Acked Write 7 Done Done Sent false
13873 4308 Acked Read 4 Done Done Sent false
13874 4309 Acked Read 7 Done Done Sent false
13875 4310 Acked Read 6 Done Done Sent false
13876 4311 Acked Write 5 Done Done Sent false
13877 4312 Acked Read 9 Done Done Sent false
13878 4313 Acked Read 2 Done Done Sent false
13879 4314 Acked Read 6 Done Done Sent false
13880 4315 Acked Flush 0 Done Done Sent false
13881 4316 Acked Read 10 Done Done Sent false
13882 4317 Acked Read 9 Done Done Sent false
13883 4318 Acked Read 8 Done Done Sent false
13884 4319 Acked Read 5 Done Done Sent false
13885 STATES DS:0 DS:1 DS:2 TOTAL
13886 Sent 0 0 301 301
13887 Done 301 301 0 602
13888 Skipped 0 0 0 0
13889 Error 0 0 0 0
13890 Last Flush: Some(JobId(4315)) Some(JobId(4315)) Some(JobId(4018))
13891 Downstairs last five completed: 4018 4017 4016 4015 4014
13892 Upstairs last five completed: 4319 4318 4317 4316 4315
13893 CLIENT: Up:0 ds:301 act:2
138942025-03-17T22:22:01.856ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
138952025-03-17T22:22:02.858ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
138962025-03-17T22:22:03.859ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
13897 ----------------------------------------------------------------
13898 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13899 States: Active Active Offline
13900 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13901 4019 Acked Flush 0 Done Done Sent false
13902 4020 Acked Flush 0 Done Done Sent false
13903 4021 Acked Write 4 Done Done Sent false
13904 4022 Acked Write 2 Done Done Sent false
13905 4023 Acked Write 6 Done Done Sent false
13906 4024 Acked Read 2 Done Done Sent false
13907 4025 Acked Write 9 Done Done Sent false
13908 4026 Acked Write 5 Done Done Sent false
13909 4027 Acked Write 6 Done Done Sent false
13910 4028 Acked Read 1 Done Done Sent false
13911 4029 Acked Read 8 Done Done Sent false
13912 4030 Acked Read 10 Done Done Sent false
13913 4031 Acked Write 4 Done Done Sent false
13914 4032 Acked Read 8 Done Done Sent false
13915 4033 Acked Read 4 Done Done Sent false
13916 4034 Acked Read 7 Done Done Sent false
13917 4035 Acked Read 6 Done Done Sent false
13918 4036 Acked Read 1 Done Done Sent false
13919 4037 Acked Read 6 Done Done Sent false
13920 4038 Acked Read 6 Done Done Sent false
13921 4039 Acked Read 5 Done Done Sent false
13922 4040 Acked Read 9 Done Done Sent false
13923 4041 Acked Write 10 Done Done Sent false
13924 4042 Acked Write 8 Done Done Sent false
13925 4043 Acked Read 5 Done Done Sent false
13926 4044 Acked Read 6 Done Done Sent false
13927 4045 Acked Write 7 Done Done Sent false
13928 4046 Acked Write 8 Done Done Sent false
13929 4047 Acked Read 2 Done Done Sent false
13930 4048 Acked Read 6 Done Done Sent false
13931 4049 Acked Write 3 Done Done Sent false
13932 4050 Acked Read 4 Done Done Sent false
13933 4051 Acked Read 1 Done Done Sent false
13934 4052 Acked Read 7 Done Done Sent false
13935 4053 Acked Read 1 Done Done Sent false
13936 4054 Acked Read 6 Done Done Sent false
13937 4055 Acked Read 3 Done Done Sent false
13938 4056 Acked Write 6 Done Done Sent false
13939 4057 Acked Write 3 Done Done Sent false
13940 4058 Acked Write 10 Done Done Sent false
13941 4059 Acked Write 9 Done Done Sent false
13942 4060 Acked Write 6 Done Done Sent false
13943 4061 Acked Flush 0 Done Done Sent false
13944 4062 Acked Read 3 Done Done Sent false
13945 4063 Acked Read 9 Done Done Sent false
13946 4064 Acked Flush 0 Done Done Sent false
13947 4065 Acked Write 5 Done Done Sent false
13948 4066 Acked Write 9 Done Done Sent false
13949 4067 Acked Write 8 Done Done Sent false
13950 4068 Acked Read 10 Done Done Sent false
13951 4069 Acked Write 8 Done Done Sent false
13952 4070 Acked Write 8 Done Done Sent false
13953 4071 Acked Read 7 Done Done Sent false
13954 4072 Acked Read 2 Done Done Sent false
13955 4073 Acked Read 4 Done Done Sent false
13956 4074 Acked Read 7 Done Done Sent false
13957 4075 Acked Read 6 Done Done Sent false
13958 4076 Acked Read 9 Done Done Sent false
13959 4077 Acked Read 2 Done Done Sent false
13960 4078 Acked Read 2 Done Done Sent false
13961 4079 Acked Write 10 Done Done Sent false
13962 4080 Acked Write 6 Done Done Sent false
13963 4081 Acked Write 6 Done Done Sent false
13964 4082 Acked Write 3 Done Done Sent false
13965 4083 Acked Write 3 Done Done Sent false
13966 4084 Acked Write 7 Done Done Sent false
13967 4085 Acked Read 10 Done Done Sent false
13968 4086 Acked Flush 0 Done Done Sent false
13969 4087 Acked Read 1 Done Done Sent false
13970 4088 Acked Write 1 Done Done Sent false
13971 4089 Acked Write 1 Done Done Sent false
13972 4090 Acked Read 6 Done Done Sent false
13973 4091 Acked Write 3 Done Done Sent false
13974 4092 Acked Read 8 Done Done Sent false
13975 4093 Acked Read 7 Done Done Sent false
13976 4094 Acked Read 10 Done Done Sent false
13977 4095 Acked Read 10 Done Done Sent false
13978 4096 Acked Write 9 Done Done Sent false
13979 4097 Acked Read 2 Done Done Sent false
13980 4098 Acked Write 1 Done Done Sent false
13981 4099 Acked Write 4 Done Done Sent false
13982 4100 Acked Read 1 Done Done Sent false
13983 4101 Acked Write 3 Done Done Sent false
13984 4102 Acked Read 1 Done Done Sent false
13985 4103 Acked Write 5 Done Done Sent false
13986 4104 Acked Write 1 Done Done Sent false
13987 4105 Acked Read 5 Done Done Sent false
13988 4106 Acked Write 3 Done Done Sent false
13989 4107 Acked Read 10 Done Done Sent false
13990 4108 Acked Write 8 Done Done Sent false
13991 4109 Acked Flush 0 Done Done Sent false
13992 4110 Acked Read 10 Done Done Sent false
13993 4111 Acked Write 8 Done Done Sent false
13994 4112 Acked Read 2 Done Done Sent false
13995 4113 Acked Write 4 Done Done Sent false
13996 4114 Acked Write 1 Done Done Sent false
13997 4115 Acked Write 1 Done Done Sent false
13998 4116 Acked Read 7 Done Done Sent false
13999 4117 Acked Read 8 Done Done Sent false
14000 4118 Acked Write 8 Done Done Sent false
14001 4119 Acked Read 3 Done Done Sent false
14002 4120 Acked Read 6 Done Done Sent false
14003 4121 Acked Write 3 Done Done Sent false
14004 4122 Acked Read 4 Done Done Sent false
14005 4123 Acked Read 2 Done Done Sent false
14006 4124 Acked Write 5 Done Done Sent false
14007 4125 Acked Read 8 Done Done Sent false
14008 4126 Acked Read 3 Done Done Sent false
14009 4127 Acked Read 4 Done Done Sent false
14010 4128 Acked Flush 0 Done Done Sent false
14011 4129 Acked Read 10 Done Done Sent false
14012 4130 Acked Write 10 Done Done Sent false
14013 4131 Acked Write 4 Done Done Sent false
14014 4132 Acked Write 8 Done Done Sent false
14015 4133 Acked Read 2 Done Done Sent false
14016 4134 Acked Write 1 Done Done Sent false
14017 4135 Acked Read 5 Done Done Sent false
14018 4136 Acked Flush 0 Done Done Sent false
14019 4137 Acked Read 1 Done Done Sent false
14020 4138 Acked Read 1 Done Done Sent false
14021 4139 Acked Read 5 Done Done Sent false
14022 4140 Acked Read 6 Done Done Sent false
14023 4141 Acked Flush 0 Done Done Sent false
14024 4142 Acked Write 4 Done Done Sent false
14025 4143 Acked Read 7 Done Done Sent false
14026 4144 Acked Flush 0 Done Done Sent false
14027 4145 Acked Read 6 Done Done Sent false
14028 4146 Acked Write 7 Done Done Sent false
14029 4147 Acked Flush 0 Done Done Sent false
14030 4148 Acked Read 5 Done Done Sent false
14031 4149 Acked Write 6 Done Done Sent false
14032 4150 Acked Read 8 Done Done Sent false
14033 4151 Acked Read 5 Done Done Sent false
14034 4152 Acked Write 8 Done Done Sent false
14035 4153 Acked Read 3 Done Done Sent false
14036 4154 Acked Flush 0 Done Done Sent false
14037 4155 Acked Read 3 Done Done Sent false
14038 4156 Acked Read 8 Done Done Sent false
14039 4157 Acked Read 10 Done Done Sent false
14040 4158 Acked Write 8 Done Done Sent false
14041 4159 Acked Read 1 Done Done Sent false
14042 4160 Acked Read 3 Done Done Sent false
14043 4161 Acked Read 6 Done Done Sent false
14044 4162 Acked Write 7 Done Done Sent false
14045 4163 Acked Write 10 Done Done Sent false
14046 4164 Acked Read 9 Done Done Sent false
14047 4165 Acked Flush 0 Done Done Sent false
14048 4166 Acked Write 1 Done Done Sent false
14049 4167 Acked Read 3 Done Done Sent false
14050 4168 Acked Flush 0 Done Done Sent false
14051 4169 Acked Read 8 Done Done Sent false
14052 4170 Acked Read 8 Done Done Sent false
14053 4171 Acked Read 10 Done Done Sent false
14054 4172 Acked Write 4 Done Done Sent false
14055 4173 Acked Write 1 Done Done Sent false
14056 4174 Acked Flush 0 Done Done Sent false
14057 4175 Acked Read 8 Done Done Sent false
14058 4176 Acked Read 1 Done Done Sent false
14059 4177 Acked Read 6 Done Done Sent false
14060 4178 Acked Read 3 Done Done Sent false
14061 4179 Acked Read 7 Done Done Sent false
14062 4180 Acked Write 9 Done Done Sent false
14063 4181 Acked Write 3 Done Done Sent false
14064 4182 Acked Write 4 Done Done Sent false
14065 4183 Acked Read 1 Done Done Sent false
14066 4184 Acked Write 1 Done Done Sent false
14067 4185 Acked Write 10 Done Done Sent false
14068 4186 Acked Read 3 Done Done Sent false
14069 4187 Acked Flush 0 Done Done Sent false
14070 4188 Acked Write 5 Done Done Sent false
14071 4189 Acked Write 1 Done Done Sent false
14072 4190 Acked Write 1 Done Done Sent false
14073 4191 Acked Read 3 Done Done Sent false
14074 4192 Acked Read 10 Done Done Sent false
14075 4193 Acked Flush 0 Done Done Sent false
14076 4194 Acked Write 6 Done Done Sent false
14077 4195 Acked Read 6 Done Done Sent false
14078 4196 Acked Write 2 Done Done Sent false
14079 4197 Acked Flush 0 Done Done Sent false
14080 4198 Acked Write 3 Done Done Sent false
14081 4199 Acked Read 7 Done Done Sent false
14082 4200 Acked Read 8 Done Done Sent false
14083 4201 Acked Read 8 Done Done Sent false
14084 4202 Acked Read 6 Done Done Sent false
14085 4203 Acked Write 9 Done Done Sent false
14086 4204 Acked Write 6 Done Done Sent false
14087 4205 Acked Write 10 Done Done Sent false
14088 4206 Acked Read 8 Done Done Sent false
14089 4207 Acked Flush 0 Done Done Sent false
14090 4208 Acked Write 9 Done Done Sent false
14091 4209 Acked Read 4 Done Done Sent false
14092 4210 Acked Read 6 Done Done Sent false
14093 4211 Acked Read 8 Done Done Sent false
14094 4212 Acked Write 8 Done Done Sent false
14095 4213 Acked Read 6 Done Done Sent false
14096 4214 Acked Read 1 Done Done Sent false
14097 4215 Acked Write 1 Done Done Sent false
14098 4216 Acked Write 8 Done Done Sent false
14099 4217 Acked Write 2 Done Done Sent false
14100 4218 Acked Read 5 Done Done Sent false
14101 4219 Acked Read 9 Done Done Sent false
14102 4220 Acked Flush 0 Done Done Sent false
14103 4221 Acked Read 2 Done Done Sent false
14104 4222 Acked Flush 0 Done Done Sent false
14105 4223 Acked Write 1 Done Done Sent false
14106 4224 Acked Write 8 Done Done Sent false
14107 4225 Acked Write 2 Done Done Sent false
14108 4226 Acked Write 9 Done Done Sent false
14109 4227 Acked Write 10 Done Done Sent false
14110 4228 Acked Read 1 Done Done Sent false
14111 4229 Acked Flush 0 Done Done Sent false
14112 4230 Acked Read 2 Done Done Sent false
14113 4231 Acked Read 5 Done Done Sent false
14114 4232 Acked Read 4 Done Done Sent false
14115 4233 Acked Write 7 Done Done Sent false
14116 4234 Acked Read 7 Done Done Sent false
14117 4235 Acked Write 4 Done Done Sent false
14118 4236 Acked Write 2 Done Done Sent false
14119 4237 Acked Flush 0 Done Done Sent false
14120 4238 Acked Read 6 Done Done Sent false
14121 4239 Acked Flush 0 Done Done Sent false
14122 4240 Acked Read 1 Done Done Sent false
14123 4241 Acked Write 4 Done Done Sent false
14124 4242 Acked Read 8 Done Done Sent false
14125 4243 Acked Write 4 Done Done Sent false
14126 4244 Acked Read 4 Done Done Sent false
14127 4245 Acked Read 7 Done Done Sent false
14128 4246 Acked Read 8 Done Done Sent false
14129 4247 Acked Read 7 Done Done Sent false
14130 4248 Acked Write 3 Done Done Sent false
14131 4249 Acked Write 4 Done Done Sent false
14132 4250 Acked Read 4 Done Done Sent false
14133 4251 Acked Write 6 Done Done Sent false
14134 4252 Acked Write 4 Done Done Sent false
14135 4253 Acked Write 10 Done Done Sent false
14136 4254 Acked Write 4 Done Done Sent false
14137 4255 Acked Flush 0 Done Done Sent false
14138 4256 Acked Write 4 Done Done Sent false
14139 4257 Acked Read 10 Done Done Sent false
14140 4258 Acked Read 4 Done Done Sent false
14141 4259 Acked Read 8 Done Done Sent false
14142 4260 Acked Write 9 Done Done Sent false
14143 4261 Acked Write 9 Done Done Sent false
14144 4262 Acked Read 9 Done Done Sent false
14145 4263 Acked Write 9 Done Done Sent false
14146 4264 Acked Write 5 Done Done Sent false
14147 4265 Acked Flush 0 Done Done Sent false
14148 4266 Acked Read 3 Done Done Sent false
14149 4267 Acked Read 6 Done Done Sent false
14150 4268 Acked Write 1 Done Done Sent false
14151 4269 Acked Write 2 Done Done Sent false
14152 4270 Acked Read 8 Done Done Sent false
14153 4271 Acked Read 9 Done Done Sent false
14154 4272 Acked Write 4 Done Done Sent false
14155 4273 Acked Read 1 Done Done Sent false
14156 4274 Acked Write 5 Done Done Sent false
14157 4275 Acked Read 4 Done Done Sent false
14158 4276 Acked Read 2 Done Done Sent false
14159 4277 Acked Write 8 Done Done Sent false
14160 4278 Acked Write 7 Done Done Sent false
14161 4279 Acked Flush 0 Done Done Sent false
14162 4280 Acked Write 7 Done Done Sent false
14163 4281 Acked Read 6 Done Done Sent false
14164 4282 Acked Read 10 Done Done Sent false
14165 4283 Acked Read 1 Done Done Sent false
14166 4284 Acked Read 7 Done Done Sent false
14167 4285 Acked Write 9 Done Done Sent false
14168 4286 Acked Write 2 Done Done Sent false
14169 4287 Acked Read 4 Done Done Sent false
14170 4288 Acked Write 2 Done Done Sent false
14171 4289 Acked Read 8 Done Done Sent false
14172 4290 Acked Write 4 Done Done Sent false
14173 4291 Acked Flush 0 Done Done Sent false
14174 4292 Acked Read 4 Done Done Sent false
14175 4293 Acked Read 3 Done Done Sent false
14176 4294 Acked Write 10 Done Done Sent false
14177 4295 Acked Read 10 Done Done Sent false
14178 4296 Acked Write 10 Done Done Sent false
14179 4297 Acked Write 7 Done Done Sent false
14180 4298 Acked Write 7 Done Done Sent false
14181 4299 Acked Read 7 Done Done Sent false
14182 4300 Acked Write 5 Done Done Sent false
14183 4301 Acked Flush 0 Done Done Sent false
14184 4302 Acked Read 7 Done Done Sent false
14185 4303 Acked Read 9 Done Done Sent false
14186 4304 Acked Flush 0 Done Done Sent false
14187 4305 Acked Write 9 Done Done Sent false
14188 4306 Acked Write 2 Done Done Sent false
14189 4307 Acked Write 7 Done Done Sent false
14190 4308 Acked Read 4 Done Done Sent false
14191 4309 Acked Read 7 Done Done Sent false
14192 4310 Acked Read 6 Done Done Sent false
14193 4311 Acked Write 5 Done Done Sent false
14194 4312 Acked Read 9 Done Done Sent false
14195 4313 Acked Read 2 Done Done Sent false
14196 4314 Acked Read 6 Done Done Sent false
14197 4315 Acked Flush 0 Done Done Sent false
14198 4316 Acked Read 10 Done Done Sent false
14199 4317 Acked Read 9 Done Done Sent false
14200 4318 Acked Read 8 Done Done Sent false
14201 4319 Acked Read 5 Done Done Sent false
14202 STATES DS:0 DS:1 DS:2 TOTAL
14203 Sent 0 0 301 301
14204 Done 301 301 0 602
14205 Skipped 0 0 0 0
14206 Error 0 0 0 0
14207 Last Flush: Some(JobId(4315)) Some(JobId(4315)) Some(JobId(4018))
14208 Downstairs last five completed: 4018 4017 4016 4015 4014
14209 Upstairs last five completed: 4319 4318 4317 4316 4315
14210 CLIENT: Up:0 ds:301 act:2
142112025-03-17T22:22:05.862ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
142122025-03-17T22:22:06.619ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
142132025-03-17T22:22:06.619ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
142142025-03-17T22:22:06.619ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
142152025-03-17T22:22:06.619ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
142162025-03-17T22:22:06.619ZINFOcrucible: client task is exiting = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
142172025-03-17T22:22:07.120ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
142182025-03-17T22:22:07.864ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
14219 [8830] got start action
14220 [8830] I am going to restart
14221 Starting downstairs at port 8830
14222 Make output file at "/tmp/dsc/downstairs-8830.txt"
14223 [8830][2] reports Starting
14224 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1222)
14225 [8830][2] reports Running
142262025-03-17T22:22:08.866ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
14227 ----------------------------------------------------------------
14228 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14229 States: Active Active Offline
14230 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14231 4019 Acked Flush 0 Done Done Sent false
14232 4020 Acked Flush 0 Done Done Sent false
14233 4021 Acked Write 4 Done Done Sent false
14234 4022 Acked Write 2 Done Done Sent false
14235 4023 Acked Write 6 Done Done Sent false
14236 4024 Acked Read 2 Done Done Sent false
14237 4025 Acked Write 9 Done Done Sent false
14238 4026 Acked Write 5 Done Done Sent false
14239 4027 Acked Write 6 Done Done Sent false
14240 4028 Acked Read 1 Done Done Sent false
14241 4029 Acked Read 8 Done Done Sent false
14242 4030 Acked Read 10 Done Done Sent false
14243 4031 Acked Write 4 Done Done Sent false
14244 4032 Acked Read 8 Done Done Sent false
14245 4033 Acked Read 4 Done Done Sent false
14246 4034 Acked Read 7 Done Done Sent false
14247 4035 Acked Read 6 Done Done Sent false
14248 4036 Acked Read 1 Done Done Sent false
14249 4037 Acked Read 6 Done Done Sent false
14250 4038 Acked Read 6 Done Done Sent false
14251 4039 Acked Read 5 Done Done Sent false
14252 4040 Acked Read 9 Done Done Sent false
14253 4041 Acked Write 10 Done Done Sent false
14254 4042 Acked Write 8 Done Done Sent false
14255 4043 Acked Read 5 Done Done Sent false
14256 4044 Acked Read 6 Done Done Sent false
14257 4045 Acked Write 7 Done Done Sent false
14258 4046 Acked Write 8 Done Done Sent false
14259 4047 Acked Read 2 Done Done Sent false
14260 4048 Acked Read 6 Done Done Sent false
14261 4049 Acked Write 3 Done Done Sent false
14262 4050 Acked Read 4 Done Done Sent false
14263 4051 Acked Read 1 Done Done Sent false
14264 4052 Acked Read 7 Done Done Sent false
14265 4053 Acked Read 1 Done Done Sent false
14266 4054 Acked Read 6 Done Done Sent false
14267 4055 Acked Read 3 Done Done Sent false
14268 4056 Acked Write 6 Done Done Sent false
14269 4057 Acked Write 3 Done Done Sent false
14270 4058 Acked Write 10 Done Done Sent false
14271 4059 Acked Write 9 Done Done Sent false
14272 4060 Acked Write 6 Done Done Sent false
14273 4061 Acked Flush 0 Done Done Sent false
14274 4062 Acked Read 3 Done Done Sent false
14275 4063 Acked Read 9 Done Done Sent false
14276 4064 Acked Flush 0 Done Done Sent false
14277 4065 Acked Write 5 Done Done Sent false
14278 4066 Acked Write 9 Done Done Sent false
14279 4067 Acked Write 8 Done Done Sent false
14280 4068 Acked Read 10 Done Done Sent false
14281 4069 Acked Write 8 Done Done Sent false
14282 4070 Acked Write 8 Done Done Sent false
14283 4071 Acked Read 7 Done Done Sent false
14284 4072 Acked Read 2 Done Done Sent false
14285 4073 Acked Read 4 Done Done Sent false
14286 4074 Acked Read 7 Done Done Sent false
14287 4075 Acked Read 6 Done Done Sent false
14288 4076 Acked Read 9 Done Done Sent false
14289 4077 Acked Read 2 Done Done Sent false
14290 4078 Acked Read 2 Done Done Sent false
14291 4079 Acked Write 10 Done Done Sent false
14292 4080 Acked Write 6 Done Done Sent false
14293 4081 Acked Write 6 Done Done Sent false
14294 4082 Acked Write 3 Done Done Sent false
14295 4083 Acked Write 3 Done Done Sent false
14296 4084 Acked Write 7 Done Done Sent false
14297 4085 Acked Read 10 Done Done Sent false
14298 4086 Acked Flush 0 Done Done Sent false
14299 4087 Acked Read 1 Done Done Sent false
14300 4088 Acked Write 1 Done Done Sent false
14301 4089 Acked Write 1 Done Done Sent false
14302 4090 Acked Read 6 Done Done Sent false
14303 4091 Acked Write 3 Done Done Sent false
14304 4092 Acked Read 8 Done Done Sent false
14305 4093 Acked Read 7 Done Done Sent false
14306 4094 Acked Read 10 Done Done Sent false
14307 4095 Acked Read 10 Done Done Sent false
14308 4096 Acked Write 9 Done Done Sent false
14309 4097 Acked Read 2 Done Done Sent false
14310 4098 Acked Write 1 Done Done Sent false
14311 4099 Acked Write 4 Done Done Sent false
14312 4100 Acked Read 1 Done Done Sent false
14313 4101 Acked Write 3 Done Done Sent false
14314 4102 Acked Read 1 Done Done Sent false
14315 4103 Acked Write 5 Done Done Sent false
14316 4104 Acked Write 1 Done Done Sent false
14317 4105 Acked Read 5 Done Done Sent false
14318 4106 Acked Write 3 Done Done Sent false
14319 4107 Acked Read 10 Done Done Sent false
14320 4108 Acked Write 8 Done Done Sent false
14321 4109 Acked Flush 0 Done Done Sent false
14322 4110 Acked Read 10 Done Done Sent false
14323 4111 Acked Write 8 Done Done Sent false
14324 4112 Acked Read 2 Done Done Sent false
14325 4113 Acked Write 4 Done Done Sent false
14326 4114 Acked Write 1 Done Done Sent false
14327 4115 Acked Write 1 Done Done Sent false
14328 4116 Acked Read 7 Done Done Sent false
14329 4117 Acked Read 8 Done Done Sent false
14330 4118 Acked Write 8 Done Done Sent false
14331 4119 Acked Read 3 Done Done Sent false
14332 4120 Acked Read 6 Done Done Sent false
14333 4121 Acked Write 3 Done Done Sent false
14334 4122 Acked Read 4 Done Done Sent false
14335 4123 Acked Read 2 Done Done Sent false
14336 4124 Acked Write 5 Done Done Sent false
14337 4125 Acked Read 8 Done Done Sent false
14338 4126 Acked Read 3 Done Done Sent false
14339 4127 Acked Read 4 Done Done Sent false
14340 4128 Acked Flush 0 Done Done Sent false
14341 4129 Acked Read 10 Done Done Sent false
14342 4130 Acked Write 10 Done Done Sent false
14343 4131 Acked Write 4 Done Done Sent false
14344 4132 Acked Write 8 Done Done Sent false
14345 4133 Acked Read 2 Done Done Sent false
14346 4134 Acked Write 1 Done Done Sent false
14347 4135 Acked Read 5 Done Done Sent false
14348 4136 Acked Flush 0 Done Done Sent false
14349 4137 Acked Read 1 Done Done Sent false
14350 4138 Acked Read 1 Done Done Sent false
14351 4139 Acked Read 5 Done Done Sent false
14352 4140 Acked Read 6 Done Done Sent false
14353 4141 Acked Flush 0 Done Done Sent false
14354 4142 Acked Write 4 Done Done Sent false
14355 4143 Acked Read 7 Done Done Sent false
14356 4144 Acked Flush 0 Done Done Sent false
14357 4145 Acked Read 6 Done Done Sent false
14358 4146 Acked Write 7 Done Done Sent false
14359 4147 Acked Flush 0 Done Done Sent false
14360 4148 Acked Read 5 Done Done Sent false
14361 4149 Acked Write 6 Done Done Sent false
14362 4150 Acked Read 8 Done Done Sent false
14363 4151 Acked Read 5 Done Done Sent false
14364 4152 Acked Write 8 Done Done Sent false
14365 4153 Acked Read 3 Done Done Sent false
14366 4154 Acked Flush 0 Done Done Sent false
14367 4155 Acked Read 3 Done Done Sent false
14368 4156 Acked Read 8 Done Done Sent false
14369 4157 Acked Read 10 Done Done Sent false
14370 4158 Acked Write 8 Done Done Sent false
14371 4159 Acked Read 1 Done Done Sent false
14372 4160 Acked Read 3 Done Done Sent false
14373 4161 Acked Read 6 Done Done Sent false
14374 4162 Acked Write 7 Done Done Sent false
14375 4163 Acked Write 10 Done Done Sent false
14376 4164 Acked Read 9 Done Done Sent false
14377 4165 Acked Flush 0 Done Done Sent false
14378 4166 Acked Write 1 Done Done Sent false
14379 4167 Acked Read 3 Done Done Sent false
14380 4168 Acked Flush 0 Done Done Sent false
14381 4169 Acked Read 8 Done Done Sent false
14382 4170 Acked Read 8 Done Done Sent false
14383 4171 Acked Read 10 Done Done Sent false
14384 4172 Acked Write 4 Done Done Sent false
14385 4173 Acked Write 1 Done Done Sent false
14386 4174 Acked Flush 0 Done Done Sent false
14387 4175 Acked Read 8 Done Done Sent false
14388 4176 Acked Read 1 Done Done Sent false
14389 4177 Acked Read 6 Done Done Sent false
14390 4178 Acked Read 3 Done Done Sent false
14391 4179 Acked Read 7 Done Done Sent false
14392 4180 Acked Write 9 Done Done Sent false
14393 4181 Acked Write 3 Done Done Sent false
14394 4182 Acked Write 4 Done Done Sent false
14395 4183 Acked Read 1 Done Done Sent false
14396 4184 Acked Write 1 Done Done Sent false
14397 4185 Acked Write 10 Done Done Sent false
14398 4186 Acked Read 3 Done Done Sent false
14399 4187 Acked Flush 0 Done Done Sent false
14400 4188 Acked Write 5 Done Done Sent false
14401 4189 Acked Write 1 Done Done Sent false
14402 4190 Acked Write 1 Done Done Sent false
14403 4191 Acked Read 3 Done Done Sent false
14404 4192 Acked Read 10 Done Done Sent false
14405 4193 Acked Flush 0 Done Done Sent false
14406 4194 Acked Write 6 Done Done Sent false
14407 4195 Acked Read 6 Done Done Sent false
14408 4196 Acked Write 2 Done Done Sent false
14409 4197 Acked Flush 0 Done Done Sent false
14410 4198 Acked Write 3 Done Done Sent false
14411 4199 Acked Read 7 Done Done Sent false
14412 4200 Acked Read 8 Done Done Sent false
14413 4201 Acked Read 8 Done Done Sent false
14414 4202 Acked Read 6 Done Done Sent false
14415 4203 Acked Write 9 Done Done Sent false
14416 4204 Acked Write 6 Done Done Sent false
14417 4205 Acked Write 10 Done Done Sent false
14418 4206 Acked Read 8 Done Done Sent false
14419 4207 Acked Flush 0 Done Done Sent false
14420 4208 Acked Write 9 Done Done Sent false
14421 4209 Acked Read 4 Done Done Sent false
14422 4210 Acked Read 6 Done Done Sent false
14423 4211 Acked Read 8 Done Done Sent false
14424 4212 Acked Write 8 Done Done Sent false
14425 4213 Acked Read 6 Done Done Sent false
14426 4214 Acked Read 1 Done Done Sent false
14427 4215 Acked Write 1 Done Done Sent false
14428 4216 Acked Write 8 Done Done Sent false
14429 4217 Acked Write 2 Done Done Sent false
14430 4218 Acked Read 5 Done Done Sent false
14431 4219 Acked Read 9 Done Done Sent false
14432 4220 Acked Flush 0 Done Done Sent false
14433 4221 Acked Read 2 Done Done Sent false
14434 4222 Acked Flush 0 Done Done Sent false
14435 4223 Acked Write 1 Done Done Sent false
14436 4224 Acked Write 8 Done Done Sent false
14437 4225 Acked Write 2 Done Done Sent false
14438 4226 Acked Write 9 Done Done Sent false
14439 4227 Acked Write 10 Done Done Sent false
14440 4228 Acked Read 1 Done Done Sent false
14441 4229 Acked Flush 0 Done Done Sent false
14442 4230 Acked Read 2 Done Done Sent false
14443 4231 Acked Read 5 Done Done Sent false
14444 4232 Acked Read 4 Done Done Sent false
14445 4233 Acked Write 7 Done Done Sent false
14446 4234 Acked Read 7 Done Done Sent false
14447 4235 Acked Write 4 Done Done Sent false
14448 4236 Acked Write 2 Done Done Sent false
14449 4237 Acked Flush 0 Done Done Sent false
14450 4238 Acked Read 6 Done Done Sent false
14451 4239 Acked Flush 0 Done Done Sent false
14452 4240 Acked Read 1 Done Done Sent false
14453 4241 Acked Write 4 Done Done Sent false
14454 4242 Acked Read 8 Done Done Sent false
14455 4243 Acked Write 4 Done Done Sent false
14456 4244 Acked Read 4 Done Done Sent false
14457 4245 Acked Read 7 Done Done Sent false
14458 4246 Acked Read 8 Done Done Sent false
14459 4247 Acked Read 7 Done Done Sent false
14460 4248 Acked Write 3 Done Done Sent false
14461 4249 Acked Write 4 Done Done Sent false
14462 4250 Acked Read 4 Done Done Sent false
14463 4251 Acked Write 6 Done Done Sent false
14464 4252 Acked Write 4 Done Done Sent false
14465 4253 Acked Write 10 Done Done Sent false
14466 4254 Acked Write 4 Done Done Sent false
14467 4255 Acked Flush 0 Done Done Sent false
14468 4256 Acked Write 4 Done Done Sent false
14469 4257 Acked Read 10 Done Done Sent false
14470 4258 Acked Read 4 Done Done Sent false
14471 4259 Acked Read 8 Done Done Sent false
14472 4260 Acked Write 9 Done Done Sent false
14473 4261 Acked Write 9 Done Done Sent false
14474 4262 Acked Read 9 Done Done Sent false
14475 4263 Acked Write 9 Done Done Sent false
14476 4264 Acked Write 5 Done Done Sent false
14477 4265 Acked Flush 0 Done Done Sent false
14478 4266 Acked Read 3 Done Done Sent false
14479 4267 Acked Read 6 Done Done Sent false
14480 4268 Acked Write 1 Done Done Sent false
14481 4269 Acked Write 2 Done Done Sent false
14482 4270 Acked Read 8 Done Done Sent false
14483 4271 Acked Read 9 Done Done Sent false
14484 4272 Acked Write 4 Done Done Sent false
14485 4273 Acked Read 1 Done Done Sent false
14486 4274 Acked Write 5 Done Done Sent false
14487 4275 Acked Read 4 Done Done Sent false
14488 4276 Acked Read 2 Done Done Sent false
14489 4277 Acked Write 8 Done Done Sent false
14490 4278 Acked Write 7 Done Done Sent false
14491 4279 Acked Flush 0 Done Done Sent false
14492 4280 Acked Write 7 Done Done Sent false
14493 4281 Acked Read 6 Done Done Sent false
14494 4282 Acked Read 10 Done Done Sent false
14495 4283 Acked Read 1 Done Done Sent false
14496 4284 Acked Read 7 Done Done Sent false
14497 4285 Acked Write 9 Done Done Sent false
14498 4286 Acked Write 2 Done Done Sent false
14499 4287 Acked Read 4 Done Done Sent false
14500 4288 Acked Write 2 Done Done Sent false
14501 4289 Acked Read 8 Done Done Sent false
14502 4290 Acked Write 4 Done Done Sent false
14503 4291 Acked Flush 0 Done Done Sent false
14504 4292 Acked Read 4 Done Done Sent false
14505 4293 Acked Read 3 Done Done Sent false
14506 4294 Acked Write 10 Done Done Sent false
14507 4295 Acked Read 10 Done Done Sent false
14508 4296 Acked Write 10 Done Done Sent false
14509 4297 Acked Write 7 Done Done Sent false
14510 4298 Acked Write 7 Done Done Sent false
14511 4299 Acked Read 7 Done Done Sent false
14512 4300 Acked Write 5 Done Done Sent false
14513 4301 Acked Flush 0 Done Done Sent false
14514 4302 Acked Read 7 Done Done Sent false
14515 4303 Acked Read 9 Done Done Sent false
14516 4304 Acked Flush 0 Done Done Sent false
14517 4305 Acked Write 9 Done Done Sent false
14518 4306 Acked Write 2 Done Done Sent false
14519 4307 Acked Write 7 Done Done Sent false
14520 4308 Acked Read 4 Done Done Sent false
14521 4309 Acked Read 7 Done Done Sent false
14522 4310 Acked Read 6 Done Done Sent false
14523 4311 Acked Write 5 Done Done Sent false
14524 4312 Acked Read 9 Done Done Sent false
14525 4313 Acked Read 2 Done Done Sent false
14526 4314 Acked Read 6 Done Done Sent false
14527 4315 Acked Flush 0 Done Done Sent false
14528 4316 Acked Read 10 Done Done Sent false
14529 4317 Acked Read 9 Done Done Sent false
14530 4318 Acked Read 8 Done Done Sent false
14531 4319 Acked Read 5 Done Done Sent false
14532 STATES DS:0 DS:1 DS:2 TOTAL
14533 Sent 0 0 301 301
14534 Done 301 301 0 602
14535 Skipped 0 0 0 0
14536 Error 0 0 0 0
14537 Last Flush: Some(JobId(4315)) Some(JobId(4315)) Some(JobId(4018))
14538 Downstairs last five completed: 4018 4017 4016 4015 4014
14539 Upstairs last five completed: 4319 4318 4317 4316 4315
14540 CLIENT: Up:0 ds:301 act:2
145412025-03-17T22:22:09.876ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
145422025-03-17T22:22:10.870ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
145432025-03-17T22:22:11.870ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
145442025-03-17T22:22:12.872ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
14545 ----------------------------------------------------------------
14546 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14547 States: Active Active Offline
14548 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14549 4019 Acked Flush 0 Done Done Sent false
14550 4020 Acked Flush 0 Done Done Sent false
14551 4021 Acked Write 4 Done Done Sent false
14552 4022 Acked Write 2 Done Done Sent false
14553 4023 Acked Write 6 Done Done Sent false
14554 4024 Acked Read 2 Done Done Sent false
14555 4025 Acked Write 9 Done Done Sent false
14556 4026 Acked Write 5 Done Done Sent false
14557 4027 Acked Write 6 Done Done Sent false
14558 4028 Acked Read 1 Done Done Sent false
14559 4029 Acked Read 8 Done Done Sent false
14560 4030 Acked Read 10 Done Done Sent false
14561 4031 Acked Write 4 Done Done Sent false
14562 4032 Acked Read 8 Done Done Sent false
14563 4033 Acked Read 4 Done Done Sent false
14564 4034 Acked Read 7 Done Done Sent false
14565 4035 Acked Read 6 Done Done Sent false
14566 4036 Acked Read 1 Done Done Sent false
14567 4037 Acked Read 6 Done Done Sent false
14568 4038 Acked Read 6 Done Done Sent false
14569 4039 Acked Read 5 Done Done Sent false
14570 4040 Acked Read 9 Done Done Sent false
14571 4041 Acked Write 10 Done Done Sent false
14572 4042 Acked Write 8 Done Done Sent false
14573 4043 Acked Read 5 Done Done Sent false
14574 4044 Acked Read 6 Done Done Sent false
14575 4045 Acked Write 7 Done Done Sent false
14576 4046 Acked Write 8 Done Done Sent false
14577 4047 Acked Read 2 Done Done Sent false
14578 4048 Acked Read 6 Done Done Sent false
14579 4049 Acked Write 3 Done Done Sent false
14580 4050 Acked Read 4 Done Done Sent false
14581 4051 Acked Read 1 Done Done Sent false
14582 4052 Acked Read 7 Done Done Sent false
14583 4053 Acked Read 1 Done Done Sent false
14584 4054 Acked Read 6 Done Done Sent false
14585 4055 Acked Read 3 Done Done Sent false
14586 4056 Acked Write 6 Done Done Sent false
14587 4057 Acked Write 3 Done Done Sent false
14588 4058 Acked Write 10 Done Done Sent false
14589 4059 Acked Write 9 Done Done Sent false
14590 4060 Acked Write 6 Done Done Sent false
14591 4061 Acked Flush 0 Done Done Sent false
14592 4062 Acked Read 3 Done Done Sent false
14593 4063 Acked Read 9 Done Done Sent false
14594 4064 Acked Flush 0 Done Done Sent false
14595 4065 Acked Write 5 Done Done Sent false
14596 4066 Acked Write 9 Done Done Sent false
14597 4067 Acked Write 8 Done Done Sent false
14598 4068 Acked Read 10 Done Done Sent false
14599 4069 Acked Write 8 Done Done Sent false
14600 4070 Acked Write 8 Done Done Sent false
14601 4071 Acked Read 7 Done Done Sent false
14602 4072 Acked Read 2 Done Done Sent false
14603 4073 Acked Read 4 Done Done Sent false
14604 4074 Acked Read 7 Done Done Sent false
14605 4075 Acked Read 6 Done Done Sent false
14606 4076 Acked Read 9 Done Done Sent false
14607 4077 Acked Read 2 Done Done Sent false
14608 4078 Acked Read 2 Done Done Sent false
14609 4079 Acked Write 10 Done Done Sent false
14610 4080 Acked Write 6 Done Done Sent false
14611 4081 Acked Write 6 Done Done Sent false
14612 4082 Acked Write 3 Done Done Sent false
14613 4083 Acked Write 3 Done Done Sent false
14614 4084 Acked Write 7 Done Done Sent false
14615 4085 Acked Read 10 Done Done Sent false
14616 4086 Acked Flush 0 Done Done Sent false
14617 4087 Acked Read 1 Done Done Sent false
14618 4088 Acked Write 1 Done Done Sent false
14619 4089 Acked Write 1 Done Done Sent false
14620 4090 Acked Read 6 Done Done Sent false
14621 4091 Acked Write 3 Done Done Sent false
14622 4092 Acked Read 8 Done Done Sent false
14623 4093 Acked Read 7 Done Done Sent false
14624 4094 Acked Read 10 Done Done Sent false
14625 4095 Acked Read 10 Done Done Sent false
14626 4096 Acked Write 9 Done Done Sent false
14627 4097 Acked Read 2 Done Done Sent false
14628 4098 Acked Write 1 Done Done Sent false
14629 4099 Acked Write 4 Done Done Sent false
14630 4100 Acked Read 1 Done Done Sent false
14631 4101 Acked Write 3 Done Done Sent false
14632 4102 Acked Read 1 Done Done Sent false
14633 4103 Acked Write 5 Done Done Sent false
14634 4104 Acked Write 1 Done Done Sent false
14635 4105 Acked Read 5 Done Done Sent false
14636 4106 Acked Write 3 Done Done Sent false
14637 4107 Acked Read 10 Done Done Sent false
14638 4108 Acked Write 8 Done Done Sent false
14639 4109 Acked Flush 0 Done Done Sent false
14640 4110 Acked Read 10 Done Done Sent false
14641 4111 Acked Write 8 Done Done Sent false
14642 4112 Acked Read 2 Done Done Sent false
14643 4113 Acked Write 4 Done Done Sent false
14644 4114 Acked Write 1 Done Done Sent false
14645 4115 Acked Write 1 Done Done Sent false
14646 4116 Acked Read 7 Done Done Sent false
14647 4117 Acked Read 8 Done Done Sent false
14648 4118 Acked Write 8 Done Done Sent false
14649 4119 Acked Read 3 Done Done Sent false
14650 4120 Acked Read 6 Done Done Sent false
14651 4121 Acked Write 3 Done Done Sent false
14652 4122 Acked Read 4 Done Done Sent false
14653 4123 Acked Read 2 Done Done Sent false
14654 4124 Acked Write 5 Done Done Sent false
14655 4125 Acked Read 8 Done Done Sent false
14656 4126 Acked Read 3 Done Done Sent false
14657 4127 Acked Read 4 Done Done Sent false
14658 4128 Acked Flush 0 Done Done Sent false
14659 4129 Acked Read 10 Done Done Sent false
14660 4130 Acked Write 10 Done Done Sent false
14661 4131 Acked Write 4 Done Done Sent false
14662 4132 Acked Write 8 Done Done Sent false
14663 4133 Acked Read 2 Done Done Sent false
14664 4134 Acked Write 1 Done Done Sent false
14665 4135 Acked Read 5 Done Done Sent false
14666 4136 Acked Flush 0 Done Done Sent false
14667 4137 Acked Read 1 Done Done Sent false
14668 4138 Acked Read 1 Done Done Sent false
14669 4139 Acked Read 5 Done Done Sent false
14670 4140 Acked Read 6 Done Done Sent false
14671 4141 Acked Flush 0 Done Done Sent false
14672 4142 Acked Write 4 Done Done Sent false
14673 4143 Acked Read 7 Done Done Sent false
14674 4144 Acked Flush 0 Done Done Sent false
14675 4145 Acked Read 6 Done Done Sent false
14676 4146 Acked Write 7 Done Done Sent false
14677 4147 Acked Flush 0 Done Done Sent false
14678 4148 Acked Read 5 Done Done Sent false
14679 4149 Acked Write 6 Done Done Sent false
14680 4150 Acked Read 8 Done Done Sent false
14681 4151 Acked Read 5 Done Done Sent false
14682 4152 Acked Write 8 Done Done Sent false
14683 4153 Acked Read 3 Done Done Sent false
14684 4154 Acked Flush 0 Done Done Sent false
14685 4155 Acked Read 3 Done Done Sent false
14686 4156 Acked Read 8 Done Done Sent false
14687 4157 Acked Read 10 Done Done Sent false
14688 4158 Acked Write 8 Done Done Sent false
14689 4159 Acked Read 1 Done Done Sent false
14690 4160 Acked Read 3 Done Done Sent false
14691 4161 Acked Read 6 Done Done Sent false
14692 4162 Acked Write 7 Done Done Sent false
14693 4163 Acked Write 10 Done Done Sent false
14694 4164 Acked Read 9 Done Done Sent false
14695 4165 Acked Flush 0 Done Done Sent false
14696 4166 Acked Write 1 Done Done Sent false
14697 4167 Acked Read 3 Done Done Sent false
14698 4168 Acked Flush 0 Done Done Sent false
14699 4169 Acked Read 8 Done Done Sent false
14700 4170 Acked Read 8 Done Done Sent false
14701 4171 Acked Read 10 Done Done Sent false
14702 4172 Acked Write 4 Done Done Sent false
14703 4173 Acked Write 1 Done Done Sent false
14704 4174 Acked Flush 0 Done Done Sent false
14705 4175 Acked Read 8 Done Done Sent false
14706 4176 Acked Read 1 Done Done Sent false
14707 4177 Acked Read 6 Done Done Sent false
14708 4178 Acked Read 3 Done Done Sent false
14709 4179 Acked Read 7 Done Done Sent false
14710 4180 Acked Write 9 Done Done Sent false
14711 4181 Acked Write 3 Done Done Sent false
14712 4182 Acked Write 4 Done Done Sent false
14713 4183 Acked Read 1 Done Done Sent false
14714 4184 Acked Write 1 Done Done Sent false
14715 4185 Acked Write 10 Done Done Sent false
14716 4186 Acked Read 3 Done Done Sent false
14717 4187 Acked Flush 0 Done Done Sent false
14718 4188 Acked Write 5 Done Done Sent false
14719 4189 Acked Write 1 Done Done Sent false
14720 4190 Acked Write 1 Done Done Sent false
14721 4191 Acked Read 3 Done Done Sent false
14722 4192 Acked Read 10 Done Done Sent false
14723 4193 Acked Flush 0 Done Done Sent false
14724 4194 Acked Write 6 Done Done Sent false
14725 4195 Acked Read 6 Done Done Sent false
14726 4196 Acked Write 2 Done Done Sent false
14727 4197 Acked Flush 0 Done Done Sent false
14728 4198 Acked Write 3 Done Done Sent false
14729 4199 Acked Read 7 Done Done Sent false
14730 4200 Acked Read 8 Done Done Sent false
14731 4201 Acked Read 8 Done Done Sent false
14732 4202 Acked Read 6 Done Done Sent false
14733 4203 Acked Write 9 Done Done Sent false
14734 4204 Acked Write 6 Done Done Sent false
14735 4205 Acked Write 10 Done Done Sent false
14736 4206 Acked Read 8 Done Done Sent false
14737 4207 Acked Flush 0 Done Done Sent false
14738 4208 Acked Write 9 Done Done Sent false
14739 4209 Acked Read 4 Done Done Sent false
14740 4210 Acked Read 6 Done Done Sent false
14741 4211 Acked Read 8 Done Done Sent false
14742 4212 Acked Write 8 Done Done Sent false
14743 4213 Acked Read 6 Done Done Sent false
14744 4214 Acked Read 1 Done Done Sent false
14745 4215 Acked Write 1 Done Done Sent false
14746 4216 Acked Write 8 Done Done Sent false
14747 4217 Acked Write 2 Done Done Sent false
14748 4218 Acked Read 5 Done Done Sent false
14749 4219 Acked Read 9 Done Done Sent false
14750 4220 Acked Flush 0 Done Done Sent false
14751 4221 Acked Read 2 Done Done Sent false
14752 4222 Acked Flush 0 Done Done Sent false
14753 4223 Acked Write 1 Done Done Sent false
14754 4224 Acked Write 8 Done Done Sent false
14755 4225 Acked Write 2 Done Done Sent false
14756 4226 Acked Write 9 Done Done Sent false
14757 4227 Acked Write 10 Done Done Sent false
14758 4228 Acked Read 1 Done Done Sent false
14759 4229 Acked Flush 0 Done Done Sent false
14760 4230 Acked Read 2 Done Done Sent false
14761 4231 Acked Read 5 Done Done Sent false
14762 4232 Acked Read 4 Done Done Sent false
14763 4233 Acked Write 7 Done Done Sent false
14764 4234 Acked Read 7 Done Done Sent false
14765 4235 Acked Write 4 Done Done Sent false
14766 4236 Acked Write 2 Done Done Sent false
14767 4237 Acked Flush 0 Done Done Sent false
14768 4238 Acked Read 6 Done Done Sent false
14769 4239 Acked Flush 0 Done Done Sent false
14770 4240 Acked Read 1 Done Done Sent false
14771 4241 Acked Write 4 Done Done Sent false
14772 4242 Acked Read 8 Done Done Sent false
14773 4243 Acked Write 4 Done Done Sent false
14774 4244 Acked Read 4 Done Done Sent false
14775 4245 Acked Read 7 Done Done Sent false
14776 4246 Acked Read 8 Done Done Sent false
14777 4247 Acked Read 7 Done Done Sent false
14778 4248 Acked Write 3 Done Done Sent false
14779 4249 Acked Write 4 Done Done Sent false
14780 4250 Acked Read 4 Done Done Sent false
14781 4251 Acked Write 6 Done Done Sent false
14782 4252 Acked Write 4 Done Done Sent false
14783 4253 Acked Write 10 Done Done Sent false
14784 4254 Acked Write 4 Done Done Sent false
14785 4255 Acked Flush 0 Done Done Sent false
14786 4256 Acked Write 4 Done Done Sent false
14787 4257 Acked Read 10 Done Done Sent false
14788 4258 Acked Read 4 Done Done Sent false
14789 4259 Acked Read 8 Done Done Sent false
14790 4260 Acked Write 9 Done Done Sent false
14791 4261 Acked Write 9 Done Done Sent false
14792 4262 Acked Read 9 Done Done Sent false
14793 4263 Acked Write 9 Done Done Sent false
14794 4264 Acked Write 5 Done Done Sent false
14795 4265 Acked Flush 0 Done Done Sent false
14796 4266 Acked Read 3 Done Done Sent false
14797 4267 Acked Read 6 Done Done Sent false
14798 4268 Acked Write 1 Done Done Sent false
14799 4269 Acked Write 2 Done Done Sent false
14800 4270 Acked Read 8 Done Done Sent false
14801 4271 Acked Read 9 Done Done Sent false
14802 4272 Acked Write 4 Done Done Sent false
14803 4273 Acked Read 1 Done Done Sent false
14804 4274 Acked Write 5 Done Done Sent false
14805 4275 Acked Read 4 Done Done Sent false
14806 4276 Acked Read 2 Done Done Sent false
14807 4277 Acked Write 8 Done Done Sent false
14808 4278 Acked Write 7 Done Done Sent false
14809 4279 Acked Flush 0 Done Done Sent false
14810 4280 Acked Write 7 Done Done Sent false
14811 4281 Acked Read 6 Done Done Sent false
14812 4282 Acked Read 10 Done Done Sent false
14813 4283 Acked Read 1 Done Done Sent false
14814 4284 Acked Read 7 Done Done Sent false
14815 4285 Acked Write 9 Done Done Sent false
14816 4286 Acked Write 2 Done Done Sent false
14817 4287 Acked Read 4 Done Done Sent false
14818 4288 Acked Write 2 Done Done Sent false
14819 4289 Acked Read 8 Done Done Sent false
14820 4290 Acked Write 4 Done Done Sent false
14821 4291 Acked Flush 0 Done Done Sent false
14822 4292 Acked Read 4 Done Done Sent false
14823 4293 Acked Read 3 Done Done Sent false
14824 4294 Acked Write 10 Done Done Sent false
14825 4295 Acked Read 10 Done Done Sent false
14826 4296 Acked Write 10 Done Done Sent false
14827 4297 Acked Write 7 Done Done Sent false
14828 4298 Acked Write 7 Done Done Sent false
14829 4299 Acked Read 7 Done Done Sent false
14830 4300 Acked Write 5 Done Done Sent false
14831 4301 Acked Flush 0 Done Done Sent false
14832 4302 Acked Read 7 Done Done Sent false
14833 4303 Acked Read 9 Done Done Sent false
14834 4304 Acked Flush 0 Done Done Sent false
14835 4305 Acked Write 9 Done Done Sent false
14836 4306 Acked Write 2 Done Done Sent false
14837 4307 Acked Write 7 Done Done Sent false
14838 4308 Acked Read 4 Done Done Sent false
14839 4309 Acked Read 7 Done Done Sent false
14840 4310 Acked Read 6 Done Done Sent false
14841 4311 Acked Write 5 Done Done Sent false
14842 4312 Acked Read 9 Done Done Sent false
14843 4313 Acked Read 2 Done Done Sent false
14844 4314 Acked Read 6 Done Done Sent false
14845 4315 Acked Flush 0 Done Done Sent false
14846 4316 Acked Read 10 Done Done Sent false
14847 4317 Acked Read 9 Done Done Sent false
14848 4318 Acked Read 8 Done Done Sent false
14849 4319 Acked Read 5 Done Done Sent false
14850 STATES DS:0 DS:1 DS:2 TOTAL
14851 Sent 0 0 301 301
14852 Done 301 301 0 602
14853 Skipped 0 0 0 0
14854 Error 0 0 0 0
14855 Last Flush: Some(JobId(4315)) Some(JobId(4315)) Some(JobId(4018))
14856 Downstairs last five completed: 4018 4017 4016 4015 4014
14857 Upstairs last five completed: 4319 4318 4317 4316 4315
14858 CLIENT: Up:0 ds:301 act:2
148592025-03-17T22:22:13.872ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
148602025-03-17T22:22:14.878ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
148612025-03-17T22:22:15.876ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
148622025-03-17T22:22:16.620ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
148632025-03-17T22:22:16.620ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:51598) = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
148642025-03-17T22:22:16.622ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
148652025-03-17T22:22:16.622ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
148662025-03-17T22:22:16.622ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
148672025-03-17T22:22:16.622ZINFOcrucible: send last flush ID to this DS: Some(JobId(4018)) = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
148682025-03-17T22:22:16.622ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
148692025-03-17T22:22:16.622ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(4018)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
148702025-03-17T22:22:16.622ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(4018)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
14871 ----------------------------------------------------------------
14872 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:4
14873 States: Active Active Active
14874 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14875 4316 Acked Read 10 Done Done Done true
14876 4317 Acked Read 9 Done Done Done true
14877 4318 Acked Read 8 Done Done Done true
14878 4319 Acked Read 5 Done Done Done true
14879 STATES DS:0 DS:1 DS:2 TOTAL
14880 Sent 0 0 0 0
14881 Done 4 4 4 12
14882 Skipped 0 0 0 0
14883 Error 0 0 0 0
14884 Last Flush: Some(JobId(4315)) Some(JobId(4315)) Some(JobId(4315))
14885 Downstairs last five completed: 4315 4314 4313 4312 4311
14886 Upstairs last five completed: 4319 4318 4317 4316 4315
14887 CLIENT: Up:0 ds:4 act:3
14888 ----------------------------------------------------------------
14889 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
14890 Downstairs last five completed: 4320 4319 4318 4317 4316
14891 Upstairs last five completed: 4320 4319 4318 4317 4316
14892 CLIENT: Up:0 ds:0 act:3
14893 Replay: All jobs finished
14894 Main task has work to do, go find it
14895 got dsc Stop(0)
14896 stop 0
14897 Mar 17 22:22:20.878 INFO request completed[8810] Got stop action so:false kr:false
14898 , latency_us: 362, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 0a0c9d60-514c-4766-b6c3-10803c38a5af, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
14899 Mar 17 22:22:20.879 INFO request completed, latency_us: 132, response_code: 200, uri: /state/cid/0, method: GET, req_id: 31dd4d83-729a-455d-bb79-c51d0e7e744e, remote_addr: 127.0.0.1:58317, local_addr: 127.{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:22:20.879269153Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
14900 0.0.1:9998
149012025-03-17T22:22:20.879ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
149022025-03-17T22:22:20.879ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
149032025-03-17T22:22:20.879ZINFOcrucible: client task is exiting = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
14904 [8810] Exited with: ExitStatus(unix_wait_status(9))
14905 [8810][0] reports Exit
14906 Mar 17 22:22:24.881 INFO request completed, latency_us: 193, response_code: 200, uri: /state/cid/0, method: GET, req_id: a239c082-591f-4803-b707-7bf269b3ade2, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
14907 001/300 Read block 2099 len 24576
14908 002/300 Read block 2135 len 40960
14909 003/300 Write block 2979 len 12288 data: 3 3 3
14910 004/300 Write block 2066 len 8192 data: 5 5
14911 005/300 Flush
14912 006/300 Write block 3432 len 40960 data: 2 2 3 2 2 4 4 3 4 3
14913 007/300 Read block 3507 len 12288
14914 008/300 Write block 205 len 40960 data: 4 4 4 4 4 3 2 2 2 2
14915 009/300 Write block 4555 len 36864 data: 4 4 5 4 4 3 3 4 4
14916 010/300 Write block 4227 len 20480 data: 6 4 5 5 6
14917 011/300 Write block 3037 len 36864 data: 4 4 5 5 5 5 4 4 5
14918 012/300 Read block 628 len 20480
14919 013/300 Read block 75 len 40960
14920 014/300 Flush
14921 015/300 Read block 3432 len 40960
14922 016/300 Write block 588 len 4096 data: 4
14923 017/300 Write block 2870 len 12288 data: 5 5 4
14924 018/300 Write block 1472 len 40960 data: 5 5 5 5 4 3 4 4 5 4
14925 019/300 Read block 511 len 40960
14926 020/300 Read block 1158 len 4096
14927 021/300 Flush
14928 022/300 Write block 2225 len 36864 data: 4 4 5 4 4 4 4 4 3
14929 023/300 Read block 4344 len 20480
14930 024/300 Write block 82 len 20480 data: 3 4 4 6 6
14931 025/300 Write block 435 len 20480 data: 5 5 4 3 2
14932 026/300 Read block 3178 len 16384
14933 027/300 Read block 1974 len 28672
14934 028/300 Write block 1196 len 20480 data: 3 2 2 2 3
14935 029/300 Read block 1350 len 20480
14936 030/300 Flush
14937 031/300 Read block 3799 len 40960
14938 032/300 Write block 1076 len 40960 data: 4 4 3 3 2 2 2 2 2 3
14939 033/300 Read block 1879 len 32768
14940 034/300 Read block 2142 len 24576
14941 035/300 Read block 2088 len 8192
14942 036/300 Write block 3254 len 12288 data: 3 2 3
14943 037/300 Write block 739 len 20480 data: 4 6 6 6 6
14944 038/300 Write block 162 len 12288 data: 4 5 5
14945 039/300 Read block 1802 len 4096
14946 040/300 Write block 3776 len 40960 data: 5 4 3 3 3 3 2 3 4 4
14947 041/300 Read block 4976 len 4096
14948 042/300 Read block 3765 len 8192
14949 043/300 Read block 2412 len 4096
14950 044/300 Write block 3479 len 24576 data: 2 2 2 2 2 2
14951 045/300 Read block 2269 len 32768
14952 046/300 Read block 3676 len 20480
14953 047/300 Write block 4117 len 12288 data: 3 3 3
14954 048/300 Write block 490 len 24576 data: 2 2 2 3 4 4
14955 049/300 Flush
14956 050/300 Write block 4709 len 36864 data: 4 5 4 4 4 5 5 4 4
14957 051/300 Read block 3345 len 4096
14958 052/300 Write block 4009 len 12288 data: 2 2 2
14959 053/300 Read block 2576 len 24576
14960 054/300 Read block 3808 len 20480
14961 055/300 Write block 3745 len 4096 data: 5
14962 056/300 Write block 2087 len 12288 data: 4 3 3
14963 057/300 Read block 2885 len 16384
14964 058/300 Read block 850 len 24576
14965 059/300 Write block 2523 len 40960 data: 4 3 2 3 4 4 4 4 4 4
14966 060/300 Read block 35 len 20480
14967 061/300 Write block 3926 len 40960 data: 3 3 4 2 2 2 2 3 3 3
14968 062/300 Read block 4130 len 20480
14969 063/300 Read block 909 len 28672
14970 064/300 Read block 1460 len 32768
14971 065/300 Write block 3447 len 28672 data: 3 3 3 3 3 3 2
14972 066/300 Write block 1110 len 16384 data: 5 4 3 3
14973 067/300 Flush
14974 068/300 Read block 309 len 36864
14975 069/300 Write block 2068 len 4096 data: 5
14976 070/300 Write block 1806 len 28672 data: 3 3 3 4 4 5 5
14977 071/300 Write block 2533 len 28672 data: 3 2 2 3 3 4 4
14978 072/300 Read block 4649 len 4096
14979 073/300 Write block 4755 len 16384 data: 4 5 5 3
14980 074/300 Read block 3106 len 36864
14981 075/300 Read block 55 len 40960
14982 076/300 Read block 1779 len 32768
14983 077/300 Write block 2291 len 32768 data: 5 4 5 3 2 2 3 4
14984 078/300 Read block 3943 len 40960
14985 079/300 Write block 504 len 12288 data: 3 3 3
14986 080/300 Write block 443 len 36864 data: 2 2 2 4 5 4 5 6 6
14987 081/300 Write block 3835 len 40960 data: 5 4 3 2 2 2 2 2 2 2
14988 082/300 Read block 2163 len 8192
14989 083/300 Write block 1362 len 8192 data: 5 3
14990 084/300 Flush
14991 085/300 Write block 4716 len 28672 data: 5 5 4 4 3 3 3
14992 086/300 Read block 4788 len 20480
14993 087/300 Read block 497 len 20480
14994 088/300 Read block 10 len 40960
14995 089/300 Read block 2588 len 28672
14996 090/300 Read block 2495 len 36864
14997 091/300 Write block 4802 len 40960 data: 5 6 6 6 4 3 2 2 2 3
14998 092/300 Write block 156 len 36864 data: 6 6 5 5 4 4 5 6 6
14999 093/300 Read block 4793 len 12288
15000 094/300 Read block 2206 len 28672
15001 095/300 Write block 1215 len 4096 data: 3
15002 096/300 Write block 1872 len 36864 data: 7 4 3 5 5 5 5 6 5
15003 097/300 Write block 1879 len 12288 data: 7 6 5
15004 098/300 Read block 1616 len 12288
15005 099/300 Write block 1919 len 16384 data: 4 4 4 5
15006 100/300 Read block 3484 len 12288
15007 101/300 Read block 3621 len 16384
15008 102/300 Write block 3873 len 36864 data: 2 3 3 3 3 3 3 2 2
15009 103/300 Read block 2308 len 20480
15010 104/300 Write block 2014 len 24576 data: 4 4 4 4 5 5
15011 105/300 Read block 968 len 40960
15012 106/300 Write block 407 len 20480 data: 4 3 2 3 3
15013 107/300 Read block 4288 len 20480
15014 108/300 Read block 2044 len 32768
15015 109/300 Write block 1878 len 8192 data: 6 8
15016 110/300 Write block 3779 len 32768 data: 4 4 4 3 4 5 5 4
15017 111/300 Write block 1799 len 24576 data: 4 4 3 4 4 4
15018 112/300 Read block 2975 len 36864
15019 113/300 Write block 1956 len 16384 data: 4 5 5 5
15020 114/300 Read block 2599 len 4096
15021 115/300 Read block 1275 len 28672
15022 116/300 Write block 2430 len 24576 data: 3 3 3 4 5 4
15023 117/300 Read block 3812 len 4096
15024 118/300 Flush
15025 119/300 Read block 3760 len 28672
15026 120/300 Write block 95 len 36864 data: 2 2 3 3 3 3 3 3 3
15027 121/300 Write block 388 len 32768 data: 3 4 4 4 4 4 4 4
15028 122/300 Read block 727 len 16384
15029 123/300 Write block 3819 len 40960 data: 2 2 2 2 2 2 2 2 3 4
15030 124/300 Write block 1275 len 8192 data: 4 3
15031 125/300 Write block 3529 len 40960 data: 6 6 7 5 5 5 5 5 5 6
15032 126/300 Read block 462 len 8192
15033 127/300 Flush
15034 128/300 Read block 1010 len 36864
15035 129/300 Read block 705 len 16384
15036 130/300 Read block 3027 len 32768
15037 131/300 Read block 2561 len 12288
15038 132/300 Read block 4434 len 24576
15039 133/300 Write block 252 len 8192 data: 3 3
15040 134/300 Read block 799 len 36864
15041 135/300 Read block 4440 len 4096
15042 136/300 Read block 1690 len 12288
15043 137/300 Read block 3755 len 28672
15044 138/300 Read block 3742 len 16384
15045 139/300 Read block 2618 len 28672
15046 140/300 Flush
15047 141/300 Read block 1101 len 32768
15048 142/300 Write block 83 len 16384 data: 5 5 7 7
15049 143/300 Flush
15050 144/300 Write block 2379 len 40960 data: 4 4 3 4 4 4 4 5 4 2
15051 145/300 Write block 4453 len 20480 data: 3 4 3 2 2
15052 146/300 Read block 340 len 8192
15053 147/300 Write block 2063 len 36864 data: 2 3 4 6 6 6 5 5 4
15054 148/300 Write block 1548 len 36864 data: 4 5 5 5 5 6 6 5 3
15055 149/300 Write block 459 len 28672 data: 3 4 4 4 4 5 4
15056 150/300 Read block 2646 len 12288
15057 151/300 Write block 2619 len 40960 data: 2 2 2 2 2 2 2 2 2 2
15058 152/300 Read block 3937 len 20480
15059 153/300 Read block 4664 len 28672
15060 154/300 Write block 1271 len 8192 data: 3 4
15061 155/300 Read block 1945 len 40960
15062 156/300 Read block 4764 len 4096
15063 157/300 Read block 2487 len 24576
15064 158/300 Write block 4549 len 32768 data: 4 5 6 5 4 5 5 5
15065 159/300 Write block 690 len 28672 data: 3 3 4 3 3 3 3
15066 160/300 Read block 4965 len 32768
15067 161/300 Read block 42 len 8192
15068 162/300 Read block 1682 len 4096
15069 163/300 Read block 1612 len 16384
15070 164/300 Flush
15071 165/300 Read block 1612 len 28672
15072 166/300 Read block 2634 len 40960
15073 167/300 Write block 2551 len 20480 data: 2 2 3 3 3
15074 168/300 Read block 4921 len 24576
15075 169/300 Write block 536 len 28672 data: 3 3 3 4 3 2 2
15076 170/300 Read block 3563 len 40960
15077 171/300 Read block 360 len 12288
15078 172/300 Write block 1534 len 40960 data: 4 4 4 3 3 2 2 3 2 2
15079 173/300 Write block 3431 len 4096 data: 3
15080 174/300 Write block 1100 len 40960 data: 2 2 2 2 2 3 3 3 4 5
15081 175/300 Read block 2581 len 4096
15082 176/300 Write block 2109 len 32768 data: 3 3 3 3 3 3 2 3
15083 177/300 Write block 4296 len 8192 data: 2 2
15084 178/300 Write block 655 len 28672 data: 2 2 2 2 3 4 4
15085 179/300 Write block 4476 len 24576 data: 3 3 4 5 6 6
15086 180/300 Read block 3623 len 36864
15087 181/300 Write block 2849 len 16384 data: 2 2 2 3
15088 182/300 Write block 1743 len 4096 data: 2
15089 183/300 Write block 3720 len 20480 data: 4 3 3 3 3
15090 184/300 Read block 4686 len 16384
15091 185/300 Write block 275 len 20480 data: 4 4 3 3 3
15092 186/300 Read block 1374 len 4096
15093 187/300 Write block 975 len 28672 data: 3 4 3 3 3 3 3
15094 188/300 Flush
15095 189/300 Read block 4906 len 36864
15096 190/300 Read block 2169 len 4096
15097 191/300 Write block 4513 len 28672 data: 5 5 5 5 3 3 3
15098 192/300 Read block 2801 len 4096
15099 193/300 Read block 1360 len 36864
15100 194/300 Flush
15101 195/300 Write block 1556 len 36864 data: 4 3 4 5 5 4 4 5 5
15102 196/300 Read block 3956 len 20480
15103 197/300 Write block 4849 len 28672 data: 2 2 2 2 2 3 3
15104 198/300 Read block 1958 len 8192
15105 199/300 Write block 1231 len 4096 data: 3
15106 200/300 Flush
15107 201/300 Write block 4281 len 24576 data: 3 3 3 4 3 3
15108 202/300 Write block 558 len 36864 data: 2 3 2 2 2 2 2 2 2
15109 203/300 Read block 880 len 8192
15110 204/300 Read block 3527 len 32768
15111 205/300 Write block 2576 len 40960 data: 4 6 6 4 4 4 5 6 7 7
15112 206/300 Write block 1076 len 32768 data: 5 5 4 4 3 3 3 3
15113 207/300 Read block 844 len 4096
15114 208/300 Write block 3554 len 36864 data: 2 2 2 2 2 3 3 4 4
15115 209/300 Write block 47 len 4096 data: 3
15116 210/300 Read block 4618 len 40960
15117 211/300 Write block 1288 len 40960 data: 8 6 4 3 3 3 3 3 2 2
15118 212/300 Write block 3390 len 20480 data: 2 2 3 3 3
15119 213/300 Write block 4850 len 8192 data: 3 3
15120 214/300 Write block 231 len 32768 data: 3 2 2 2 2 2 2 2
15121 215/300 Read block 4275 len 24576
15122 216/300 Write block 1793 len 12288 data: 3 3 3
15123 217/300 Read block 937 len 16384
15124 218/300 Write block 1639 len 8192 data: 4 4
15125 219/300 Write block 1952 len 8192 data: 3 2
15126 220/300 Write block 641 len 28672 data: 2 3 2 3 3 5 4
15127 221/300 Read block 4266 len 32768
15128 222/300 Read block 2186 len 20480
15129 223/300 Read block 4551 len 20480
15130 224/300 Read block 2627 len 4096
15131 225/300 Write block 1371 len 36864 data: 2 3 3 5 4 3 3 2 2
15132 226/300 Read block 1831 len 24576
15133 227/300 Read block 3366 len 24576
15134 228/300 Write block 3770 len 36864 data: 4 3 3 4 5 5 6 5 4
15135 229/300 Write block 134 len 8192 data: 4 3
15136 230/300 Read block 1410 len 20480
15137 231/300 Read block 4372 len 32768
15138 232/300 Read block 2017 len 20480
15139 233/300 Read block 4429 len 20480
15140 234/300 Read block 4714 len 8192
15141 235/300 Read block 75 len 40960
15142 236/300 Flush
15143 237/300 Read block 2032 len 8192
15144 238/300 Read block 2456 len 12288
15145 239/300 Write block 3988 len 28672 data: 2 2 2 2 2 3 3
15146 240/300 Flush
15147 241/300 Write block 1365 len 20480 data: 2 2 2 2 2
15148 242/300 Read block 2502 len 24576
15149 243/300 Read block 3962 len 16384
15150 244/300 Read block 3029 len 12288
15151 245/300 Write block 2712 len 24576 data: 3 3 3 3 2 2
15152 246/300 Write block 49 len 20480 data: 2 2 2 2 2
15153 247/300 Read block 2721 len 28672
15154 248/300 Write block 802 len 4096 data: 3
15155 249/300 Write block 695 len 40960 data: 4 4 3 3 4 5 5 5 5 4
15156 250/300 Read block 2689 len 40960
15157 251/300 Flush
15158 252/300 Read block 350 len 40960
15159 253/300 Read block 4984 len 28672
15160 254/300 Write block 2687 len 24576 data: 3 4 4 2 2 2
15161 255/300 Read block 390 len 36864
15162 256/300 Read block 4251 len 40960
15163 257/300 Read block 4924 len 16384
15164 258/300 Read block 2314 len 20480
15165 259/300 Flush
15166 260/300 Flush
15167 261/300 Write block 3522 len 16384 data: 4 5 3 5
15168 262/300 Write block 3544 len 40960 data: 4 5 5 5 4 5 5 5 4 3
15169 263/300 Flush
15170 264/300 Write block 4931 len 8192 data: 4 5
15171 265/300 Write block 4814 len 36864 data: 4 5 5 5 5 4 5 5 5
15172 266/300 Read block 4522 len 24576
15173 267/300 Read block 259 len 32768
15174 268/300 Read block 523 len 40960
15175 269/300 Write block 4843 len 40960 data: 5 4 4 4 3 2 3 4 4 3
15176 270/300 Read block 706 len 40960
15177 271/300 Write block 1585 len 24576 data: 5 4 5 5 6 5
15178 272/300 Write block 4310 len 24576 data: 2 3 4 4 4 4
15179 273/300 Flush
15180 274/300 Read block 2322 len 12288
15181 275/300 Write block 693 len 20480 data: 4 4 5 5 4
15182 276/300 Read block 2652 len 20480
15183 277/300 Write block 1695 len 4096 data: 3
15184 278/300 Flush
15185 279/300 Write block 3444 len 4096 data: 4
15186 280/300 Write block 3926 len 24576 data: 4 4 5 3 3 3
15187 281/300 Write block 2820 len 28672 data: 3 3 3 3 3 3 3
15188 282/300 Write block 193 len 4096 data: 3
15189 283/300 Read block 852 len 24576
15190 284/300 Flush
15191 285/300 Flush
15192 286/300 Read block 3418 len 36864
15193 287/300 Write block 2945 len 28672 data: 2 3 4 4 4 4 4
15194 288/300 Read block 4412 len 8192
15195 289/300 Write block 2744 len 4096 data: 4
15196 290/300 Write block 1719 len 32768 data: 3 3 3 4 4 4 4 5
15197 291/300 Read block 171 len 36864
15198 292/300 Write block 1632 len 40960 data: 2 3 3 5 5 4 4 5 5 4
15199 293/300 Write block 2540 len 28672 data: 4 4 3 3 3 4 4
15200 294/300 Read block 2421 len 36864
15201 295/300 Write block 4657 len 8192 data: 3 3
15202 296/300 Read block 2400 len 40960
15203 297/300 Read block 1563 len 36864
15204 298/300 Read block 3492 len 36864
15205 299/300 Flush
15206 300/300 Read block 3432 len 40960
15207 301/300 Read block 104 len 32768
15208 Main task has work to do, go find it
15209 got dsc Start(0)
15210 start 0
15211 Mar 17 22:22:25.132 INFO request completed, latency_us: 205, response_code: 204, uri: /start/cid/0, method: POST, req_id: 507618dd-3ea5-4720-baba-c1cb4c7ddf2f, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
15212 Replay: started 0, returned:Ok(())
15213 ----------------------------------------------------------------
15214 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15215 States: Offline Active Active
15216 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15217 4321 Acked Read 6 Sent Done Done false
15218 4322 Acked Read 10 Sent Done Done false
15219 4323 Acked Write 3 Sent Done Done false
15220 4324 Acked Write 2 Sent Done Done false
15221 4325 Acked Flush 0 Sent Done Done false
15222 4326 Acked Write 10 Sent Done Done false
15223 4327 Acked Read 3 Sent Done Done false
15224 4328 Acked Write 10 Sent Done Done false
15225 4329 Acked Write 9 Sent Done Done false
15226 4330 Acked Write 5 Sent Done Done false
15227 4331 Acked Write 9 Sent Done Done false
15228 4332 Acked Read 5 Sent Done Done false
15229 4333 Acked Read 10 Sent Done Done false
15230 4334 Acked Flush 0 Sent Done Done false
15231 4335 Acked Read 10 Sent Done Done false
15232 4336 Acked Write 1 Sent Done Done false
15233 4337 Acked Write 3 Sent Done Done false
15234 4338 Acked Write 10 Sent Done Done false
15235 4339 Acked Read 10 Sent Done Done false
15236 4340 Acked Read 1 Sent Done Done false
15237 4341 Acked Flush 0 Sent Done Done false
15238 4342 Acked Write 9 Sent Done Done false
15239 4343 Acked Read 5 Sent Done Done false
15240 4344 Acked Write 5 Sent Done Done false
15241 4345 Acked Write 5 Sent Done Done false
15242 4346 Acked Read 4 Sent Done Done false
15243 4347 Acked Read 7 Sent Done Done false
15244 4348 Acked Write 5 Sent Done Done false
15245 4349 Acked Read 5 Sent Done Done false
15246 4350 Acked Flush 0 Sent Done Done false
15247 4351 Acked Read 10 Sent Done Done false
15248 4352 Acked Write 10 Sent Done Done false
15249 4353 Acked Read 8 Sent Done Done false
15250 4354 Acked Read 6 Sent Done Done false
15251 4355 Acked Read 2 Sent Done Done false
15252 4356 Acked Write 3 Sent Done Done false
15253 4357 Acked Write 5 Sent Done Done false
15254 4358 Acked Write 3 Sent Done Done false
15255 4359 Acked Read 1 Sent Done Done false
15256 4360 Acked Write 10 Sent Done Done false
15257 4361 Acked Read 1 Sent Done Done false
15258 4362 Acked Read 2 Sent Done Done false
15259 4363 Acked Read 1 Sent Done Done false
15260 4364 Acked Write 6 Sent Done Done false
15261 4365 Acked Read 8 Sent Done Done false
15262 4366 Acked Read 5 Sent Done Done false
15263 4367 Acked Write 3 Sent Done Done false
15264 4368 Acked Write 6 Sent Done Done false
15265 4369 Acked Flush 0 Sent Done Done false
15266 4370 Acked Write 9 Sent Done Done false
15267 4371 Acked Read 1 Sent Done Done false
15268 4372 Acked Write 3 Sent Done Done false
15269 4373 Acked Read 6 Sent Done Done false
15270 4374 Acked Read 5 Sent Done Done false
15271 4375 Acked Write 1 Sent Done Done false
15272 4376 Acked Write 3 Sent Done Done false
15273 4377 Acked Read 4 Sent Done Done false
15274 4378 Acked Read 6 Sent Done Done false
15275 4379 Acked Write 10 Sent Done Done false
15276 4380 Acked Read 5 Sent Done Done false
15277 4381 Acked Write 10 Sent Done Done false
15278 4382 Acked Read 5 Sent Done Done false
15279 4383 Acked Read 7 Sent Done Done false
15280 4384 Acked Read 8 Sent Done Done false
15281 4385 Acked Write 7 Sent Done Done false
15282 4386 Acked Write 4 Sent Done Done false
15283 4387 Acked Flush 0 Sent Done Done false
15284 4388 Acked Read 9 Sent Done Done false
15285 4389 Acked Write 1 Sent Done Done false
15286 4390 Acked Write 7 Sent Done Done false
15287 4391 Acked Write 7 Sent Done Done false
15288 4392 Acked Read 1 Sent Done Done false
15289 4393 Acked Write 4 Sent Done Done false
15290 4394 Acked Read 9 Sent Done Done false
15291 4395 Acked Read 10 Sent Done Done false
15292 4396 Acked Read 8 Sent Done Done false
15293 4397 Acked Write 8 Sent Done Done false
15294 4398 Acked Read 10 Sent Done Done false
15295 4399 Acked Write 3 Sent Done Done false
15296 4400 Acked Write 9 Sent Done Done false
15297 4401 Acked Write 10 Sent Done Done false
15298 4402 Acked Read 2 Sent Done Done false
15299 4403 Acked Write 2 Sent Done Done false
15300 4404 Acked Flush 0 Sent Done Done false
15301 4405 Acked Write 7 Sent Done Done false
15302 4406 Acked Read 5 Sent Done Done false
15303 4407 Acked Read 5 Sent Done Done false
15304 4408 Acked Read 10 Sent Done Done false
15305 4409 Acked Read 7 Sent Done Done false
15306 4410 Acked Read 9 Sent Done Done false
15307 4411 Acked Write 10 Sent Done Done false
15308 4412 Acked Write 9 Sent Done Done false
15309 4413 Acked Read 3 Sent Done Done false
15310 4414 Acked Read 7 Sent Done Done false
15311 4415 Acked Write 1 Sent Done Done false
15312 4416 Acked Write 9 Sent Done Done false
15313 4417 Acked Write 3 Sent Done Done false
15314 4418 Acked Read 3 Sent Done Done false
15315 4419 Acked Write 4 Sent Done Done false
15316 4420 Acked Read 3 Sent Done Done false
15317 4421 Acked Read 4 Sent Done Done false
15318 4422 Acked Write 9 Sent Done Done false
15319 4423 Acked Read 5 Sent Done Done false
15320 4424 Acked Write 6 Sent Done Done false
15321 4425 Acked Read 10 Sent Done Done false
15322 4426 Acked Write 5 Sent Done Done false
15323 4427 Acked Read 5 Sent Done Done false
15324 4428 Acked Read 8 Sent Done Done false
15325 4429 Acked Write 2 Sent Done Done false
15326 4430 Acked Write 8 Sent Done Done false
15327 4431 Acked Write 6 Sent Done Done false
15328 4432 Acked Read 9 Sent Done Done false
15329 4433 Acked Write 4 Sent Done Done false
15330 4434 Acked Read 1 Sent Done Done false
15331 4435 Acked Read 7 Sent Done Done false
15332 4436 Acked Write 6 Sent Done Done false
15333 4437 Acked Read 1 Sent Done Done false
15334 4438 Acked Flush 0 Sent Done Done false
15335 4439 Acked Read 7 Sent Done Done false
15336 4440 Acked Write 9 Sent Done Done false
15337 4441 Acked Write 8 Sent Done Done false
15338 4442 Acked Read 4 Sent Done Done false
15339 4443 Acked Write 10 Sent Done Done false
15340 4444 Acked Write 2 Sent Done Done false
15341 4445 Acked Write 10 Sent Done Done false
15342 4446 Acked Read 2 Sent Done Done false
15343 4447 Acked Flush 0 Sent Done Done false
15344 4448 Acked Read 9 Sent Done Done false
15345 4449 Acked Read 4 Sent Done Done false
15346 4450 Acked Read 8 Sent Done Done false
15347 4451 Acked Read 3 Sent Done Done false
15348 4452 Acked Read 6 Sent Done Done false
15349 4453 Acked Write 2 Sent Done Done false
15350 4454 Acked Read 9 Sent Done Done false
15351 4455 Acked Read 1 Sent Done Done false
15352 4456 Acked Read 3 Sent Done Done false
15353 4457 Acked Read 7 Sent Done Done false
15354 4458 Acked Read 4 Sent Done Done false
15355 4459 Acked Read 7 Sent Done Done false
15356 4460 Acked Flush 0 Sent Done Done false
15357 4461 Acked Read 8 Sent Done Done false
15358 4462 Acked Write 4 Sent Done Done false
15359 4463 Acked Flush 0 Sent Done Done false
15360 4464 Acked Write 10 Sent Done Done false
15361 4465 Acked Write 5 Sent Done Done false
15362 4466 Acked Read 2 Sent Done Done false
15363 4467 Acked Write 9 Sent Done Done false
15364 4468 Acked Write 9 Sent Done Done false
15365 4469 Acked Write 7 Sent Done Done false
15366 4470 Acked Read 3 Sent Done Done false
15367 4471 Acked Write 10 Sent Done Done false
15368 4472 Acked Read 5 Sent Done Done false
15369 4473 Acked Read 7 Sent Done Done false
15370 4474 Acked Write 2 Sent Done Done false
15371 4475 Acked Read 10 Sent Done Done false
15372 4476 Acked Read 1 Sent Done Done false
15373 4477 Acked Read 6 Sent Done Done false
15374 4478 Acked Write 8 Sent Done Done false
15375 4479 Acked Write 7 Sent Done Done false
15376 4480 Acked Read 8 Sent Done Done false
15377 4481 Acked Read 2 Sent Done Done false
15378 4482 Acked Read 1 Sent Done Done false
15379 4483 Acked Read 4 Sent Done Done false
15380 4484 Acked Flush 0 Sent Done Done false
15381 4485 Acked Read 7 Sent Done Done false
15382 4486 Acked Read 10 Sent Done Done false
15383 4487 Acked Write 5 Sent Done Done false
15384 4488 Acked Read 6 Sent Done Done false
15385 4489 Acked Write 7 Sent Done Done false
15386 4490 Acked Read 10 Sent Done Done false
15387 4491 Acked Read 3 Sent Done Done false
15388 4492 Acked Write 10 Sent Done Done false
15389 4493 Acked Write 1 Sent Done Done false
15390 4494 Acked Write 10 Sent Done Done false
15391 4495 Acked Read 1 Sent Done Done false
15392 4496 Acked Write 8 Sent Done Done false
15393 4497 Acked Write 2 Sent Done Done false
15394 4498 Acked Write 7 Sent Done Done false
15395 4499 Acked Write 6 Sent Done Done false
15396 4500 Acked Read 9 Sent Done Done false
15397 4501 Acked Write 4 Sent Done Done false
15398 4502 Acked Write 1 Sent Done Done false
15399 4503 Acked Write 5 Sent Done Done false
15400 4504 Acked Read 4 Sent Done Done false
15401 4505 Acked Write 5 Sent Done Done false
15402 4506 Acked Read 1 Sent Done Done false
15403 4507 Acked Write 7 Sent Done Done false
15404 4508 Acked Flush 0 Sent Done Done false
15405 4509 Acked Read 9 Sent Done Done false
15406 4510 Acked Read 1 Sent Done Done false
15407 4511 Acked Write 7 Sent Done Done false
15408 4512 Acked Read 1 Sent Done Done false
15409 4513 Acked Read 9 Sent Done Done false
15410 4514 Acked Flush 0 Sent Done Done false
15411 4515 Acked Write 9 Sent Done Done false
15412 4516 Acked Read 5 Sent Done Done false
15413 4517 Acked Write 7 Sent Done Done false
15414 4518 Acked Read 2 Sent Done Done false
15415 4519 Acked Write 1 Sent Done Done false
15416 4520 Acked Flush 0 Sent Done Done false
15417 4521 Acked Write 6 Sent Done Done false
15418 4522 Acked Write 9 Sent Done Done false
15419 4523 Acked Read 2 Sent Done Done false
15420 4524 Acked Read 8 Sent Done Done false
15421 4525 Acked Write 10 Sent Done Done false
15422 4526 Acked Write 8 Sent Done Done false
15423 4527 Acked Read 1 Sent Done Done false
15424 4528 Acked Write 9 Sent Done Done false
15425 4529 Acked Write 1 Sent Done Done false
15426 4530 Acked Read 10 Sent Done Done false
15427 4531 Acked Write 10 Sent Done Done false
15428 4532 Acked Write 5 Sent Done Done false
15429 4533 Acked Write 2 Sent Done Done false
15430 4534 Acked Write 8 Sent Done Done false
15431 4535 Acked Read 6 Sent Done Done false
15432 4536 Acked Write 3 Sent Done Done false
15433 4537 Acked Read 4 Sent Done Done false
15434 4538 Acked Write 2 Sent Done Done false
15435 4539 Acked Write 2 Sent Done Done false
15436 4540 Acked Write 7 Sent Done Done false
15437 4541 Acked Read 8 Sent Done Done false
15438 4542 Acked Read 5 Sent Done Done false
15439 4543 Acked Read 5 Sent Done Done false
15440 4544 Acked Read 1 Sent Done Done false
15441 4545 Acked Write 9 Sent Done Done false
15442 4546 Acked Read 6 Sent Done Done false
15443 4547 Acked Read 6 Sent Done Done false
15444 4548 Acked Write 9 Sent Done Done false
15445 4549 Acked Write 2 Sent Done Done false
15446 4550 Acked Read 5 Sent Done Done false
15447 4551 Acked Read 8 Sent Done Done false
15448 4552 Acked Read 5 Sent Done Done false
15449 4553 Acked Read 5 Sent Done Done false
15450 4554 Acked Read 2 Sent Done Done false
15451 4555 Acked Read 10 Sent Done Done false
15452 4556 Acked Flush 0 Sent Done Done false
15453 4557 Acked Read 2 Sent Done Done false
15454 4558 Acked Read 3 Sent Done Done false
15455 4559 Acked Write 7 Sent Done Done false
15456 4560 Acked Flush 0 Sent Done Done false
15457 4561 Acked Write 5 Sent Done Done false
15458 4562 Acked Read 6 Sent Done Done false
15459 4563 Acked Read 4 Sent Done Done false
15460 4564 Acked Read 3 Sent Done Done false
15461 4565 Acked Write 6 Sent Done Done false
15462 4566 Acked Write 5 Sent Done Done false
15463 4567 Acked Read 7 Sent Done Done false
15464 4568 Acked Write 1 Sent Done Done false
15465 4569 Acked Write 10 Sent Done Done false
15466 4570 Acked Read 10 Sent Done Done false
15467 4571 Acked Flush 0 Sent Done Done false
15468 4572 Acked Read 10 Sent Done Done false
15469 4573 Acked Read 7 Sent Done Done false
15470 4574 Acked Write 6 Sent Done Done false
15471 4575 Acked Read 9 Sent Done Done false
15472 4576 Acked Read 10 Sent Done Done false
15473 4577 Acked Read 4 Sent Done Done false
15474 4578 Acked Read 5 Sent Done Done false
15475 4579 Acked Flush 0 Sent Done Done false
15476 4580 Acked Flush 0 Sent Done Done false
15477 4581 Acked Write 4 Sent Done Done false
15478 4582 Acked Write 10 Sent Done Done false
15479 4583 Acked Flush 0 Sent Done Done false
15480 4584 Acked Write 2 Sent Done Done false
15481 4585 Acked Write 9 Sent Done Done false
15482 4586 Acked Read 6 Sent Done Done false
15483 4587 Acked Read 8 Sent Done Done false
15484 4588 Acked Read 10 Sent Done Done false
15485 4589 Acked Write 10 Sent Done Done false
15486 4590 Acked Read 10 Sent Done Done false
15487 4591 Acked Write 6 Sent Done Done false
15488 4592 Acked Write 6 Sent Done Done false
15489 4593 Acked Flush 0 Sent Done Done false
15490 4594 Acked Read 3 Sent Done Done false
15491 4595 Acked Write 5 Sent Done Done false
15492 4596 Acked Read 5 Sent Done Done false
15493 4597 Acked Write 1 Sent Done Done false
15494 4598 Acked Flush 0 Sent Done Done false
15495 4599 Acked Write 1 Sent Done Done false
15496 4600 Acked Write 6 Sent Done Done false
15497 4601 Acked Write 7 Sent Done Done false
15498 4602 Acked Write 1 Sent Done Done false
15499 4603 Acked Read 6 Sent Done Done false
15500 4604 Acked Flush 0 Sent Done Done false
15501 4605 Acked Flush 0 Sent Done Done false
15502 4606 Acked Read 9 Sent Done Done false
15503 4607 Acked Write 7 Sent Done Done false
15504 4608 Acked Read 2 Sent Done Done false
15505 4609 Acked Write 1 Sent Done Done false
15506 4610 Acked Write 8 Sent Done Done false
15507 4611 Acked Read 9 Sent Done Done false
15508 4612 Acked Write 10 Sent Done Done false
15509 4613 Acked Write 7 Sent Done Done false
15510 4614 Acked Read 9 Sent Done Done false
15511 4615 Acked Write 2 Sent Done Done false
15512 4616 Acked Read 10 Sent Done Done false
15513 4617 Acked Read 9 Sent Done Done false
15514 4618 Acked Read 9 Sent Done Done false
15515 4619 Acked Flush 0 Sent Done Done false
15516 4620 Acked Read 10 Sent Done Done false
15517 4621 Acked Read 8 Sent Done Done false
15518 STATES DS:0 DS:1 DS:2 TOTAL
15519 Sent 301 0 0 301
15520 Done 0 301 301 602
15521 Skipped 0 0 0 0
15522 Error 0 0 0 0
15523 Last Flush: Some(JobId(4320)) Some(JobId(4619)) Some(JobId(4619))
15524 Downstairs last five completed: 4320 4319 4318 4317 4316
15525 Upstairs last five completed: 4621 4620 4619 4618 4617
15526 CLIENT: Up:0 ds:301 act:2
155272025-03-17T22:22:25.895ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
155282025-03-17T22:22:27.126ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
155292025-03-17T22:22:27.896ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
15530 [8810] got start action
15531 [8810] I am going to restart
15532 Starting downstairs at port 8810
15533 Make output file at "/tmp/dsc/downstairs-8810.txt"
15534 [8810][0] reports Starting
15535 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1223)
15536 [8810][0] reports Running
155372025-03-17T22:22:28.898ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
15538 ----------------------------------------------------------------
15539 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15540 States: Offline Active Active
15541 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15542 4321 Acked Read 6 Sent Done Done false
15543 4322 Acked Read 10 Sent Done Done false
15544 4323 Acked Write 3 Sent Done Done false
15545 4324 Acked Write 2 Sent Done Done false
15546 4325 Acked Flush 0 Sent Done Done false
15547 4326 Acked Write 10 Sent Done Done false
15548 4327 Acked Read 3 Sent Done Done false
15549 4328 Acked Write 10 Sent Done Done false
15550 4329 Acked Write 9 Sent Done Done false
15551 4330 Acked Write 5 Sent Done Done false
15552 4331 Acked Write 9 Sent Done Done false
15553 4332 Acked Read 5 Sent Done Done false
15554 4333 Acked Read 10 Sent Done Done false
15555 4334 Acked Flush 0 Sent Done Done false
15556 4335 Acked Read 10 Sent Done Done false
15557 4336 Acked Write 1 Sent Done Done false
15558 4337 Acked Write 3 Sent Done Done false
15559 4338 Acked Write 10 Sent Done Done false
15560 4339 Acked Read 10 Sent Done Done false
15561 4340 Acked Read 1 Sent Done Done false
15562 4341 Acked Flush 0 Sent Done Done false
15563 4342 Acked Write 9 Sent Done Done false
15564 4343 Acked Read 5 Sent Done Done false
15565 4344 Acked Write 5 Sent Done Done false
15566 4345 Acked Write 5 Sent Done Done false
15567 4346 Acked Read 4 Sent Done Done false
15568 4347 Acked Read 7 Sent Done Done false
15569 4348 Acked Write 5 Sent Done Done false
15570 4349 Acked Read 5 Sent Done Done false
15571 4350 Acked Flush 0 Sent Done Done false
15572 4351 Acked Read 10 Sent Done Done false
15573 4352 Acked Write 10 Sent Done Done false
15574 4353 Acked Read 8 Sent Done Done false
15575 4354 Acked Read 6 Sent Done Done false
15576 4355 Acked Read 2 Sent Done Done false
15577 4356 Acked Write 3 Sent Done Done false
15578 4357 Acked Write 5 Sent Done Done false
15579 4358 Acked Write 3 Sent Done Done false
15580 4359 Acked Read 1 Sent Done Done false
15581 4360 Acked Write 10 Sent Done Done false
15582 4361 Acked Read 1 Sent Done Done false
15583 4362 Acked Read 2 Sent Done Done false
15584 4363 Acked Read 1 Sent Done Done false
15585 4364 Acked Write 6 Sent Done Done false
15586 4365 Acked Read 8 Sent Done Done false
15587 4366 Acked Read 5 Sent Done Done false
15588 4367 Acked Write 3 Sent Done Done false
15589 4368 Acked Write 6 Sent Done Done false
15590 4369 Acked Flush 0 Sent Done Done false
15591 4370 Acked Write 9 Sent Done Done false
15592 4371 Acked Read 1 Sent Done Done false
15593 4372 Acked Write 3 Sent Done Done false
15594 4373 Acked Read 6 Sent Done Done false
15595 4374 Acked Read 5 Sent Done Done false
15596 4375 Acked Write 1 Sent Done Done false
15597 4376 Acked Write 3 Sent Done Done false
15598 4377 Acked Read 4 Sent Done Done false
15599 4378 Acked Read 6 Sent Done Done false
15600 4379 Acked Write 10 Sent Done Done false
15601 4380 Acked Read 5 Sent Done Done false
15602 4381 Acked Write 10 Sent Done Done false
15603 4382 Acked Read 5 Sent Done Done false
15604 4383 Acked Read 7 Sent Done Done false
15605 4384 Acked Read 8 Sent Done Done false
15606 4385 Acked Write 7 Sent Done Done false
15607 4386 Acked Write 4 Sent Done Done false
15608 4387 Acked Flush 0 Sent Done Done false
15609 4388 Acked Read 9 Sent Done Done false
15610 4389 Acked Write 1 Sent Done Done false
15611 4390 Acked Write 7 Sent Done Done false
15612 4391 Acked Write 7 Sent Done Done false
15613 4392 Acked Read 1 Sent Done Done false
15614 4393 Acked Write 4 Sent Done Done false
15615 4394 Acked Read 9 Sent Done Done false
15616 4395 Acked Read 10 Sent Done Done false
15617 4396 Acked Read 8 Sent Done Done false
15618 4397 Acked Write 8 Sent Done Done false
15619 4398 Acked Read 10 Sent Done Done false
15620 4399 Acked Write 3 Sent Done Done false
15621 4400 Acked Write 9 Sent Done Done false
15622 4401 Acked Write 10 Sent Done Done false
15623 4402 Acked Read 2 Sent Done Done false
15624 4403 Acked Write 2 Sent Done Done false
15625 4404 Acked Flush 0 Sent Done Done false
15626 4405 Acked Write 7 Sent Done Done false
15627 4406 Acked Read 5 Sent Done Done false
15628 4407 Acked Read 5 Sent Done Done false
15629 4408 Acked Read 10 Sent Done Done false
15630 4409 Acked Read 7 Sent Done Done false
15631 4410 Acked Read 9 Sent Done Done false
15632 4411 Acked Write 10 Sent Done Done false
15633 4412 Acked Write 9 Sent Done Done false
15634 4413 Acked Read 3 Sent Done Done false
15635 4414 Acked Read 7 Sent Done Done false
15636 4415 Acked Write 1 Sent Done Done false
15637 4416 Acked Write 9 Sent Done Done false
15638 4417 Acked Write 3 Sent Done Done false
15639 4418 Acked Read 3 Sent Done Done false
15640 4419 Acked Write 4 Sent Done Done false
15641 4420 Acked Read 3 Sent Done Done false
15642 4421 Acked Read 4 Sent Done Done false
15643 4422 Acked Write 9 Sent Done Done false
15644 4423 Acked Read 5 Sent Done Done false
15645 4424 Acked Write 6 Sent Done Done false
15646 4425 Acked Read 10 Sent Done Done false
15647 4426 Acked Write 5 Sent Done Done false
15648 4427 Acked Read 5 Sent Done Done false
15649 4428 Acked Read 8 Sent Done Done false
15650 4429 Acked Write 2 Sent Done Done false
15651 4430 Acked Write 8 Sent Done Done false
15652 4431 Acked Write 6 Sent Done Done false
15653 4432 Acked Read 9 Sent Done Done false
15654 4433 Acked Write 4 Sent Done Done false
15655 4434 Acked Read 1 Sent Done Done false
15656 4435 Acked Read 7 Sent Done Done false
15657 4436 Acked Write 6 Sent Done Done false
15658 4437 Acked Read 1 Sent Done Done false
15659 4438 Acked Flush 0 Sent Done Done false
15660 4439 Acked Read 7 Sent Done Done false
15661 4440 Acked Write 9 Sent Done Done false
15662 4441 Acked Write 8 Sent Done Done false
15663 4442 Acked Read 4 Sent Done Done false
15664 4443 Acked Write 10 Sent Done Done false
15665 4444 Acked Write 2 Sent Done Done false
15666 4445 Acked Write 10 Sent Done Done false
15667 4446 Acked Read 2 Sent Done Done false
15668 4447 Acked Flush 0 Sent Done Done false
15669 4448 Acked Read 9 Sent Done Done false
15670 4449 Acked Read 4 Sent Done Done false
15671 4450 Acked Read 8 Sent Done Done false
15672 4451 Acked Read 3 Sent Done Done false
15673 4452 Acked Read 6 Sent Done Done false
15674 4453 Acked Write 2 Sent Done Done false
15675 4454 Acked Read 9 Sent Done Done false
15676 4455 Acked Read 1 Sent Done Done false
15677 4456 Acked Read 3 Sent Done Done false
15678 4457 Acked Read 7 Sent Done Done false
15679 4458 Acked Read 4 Sent Done Done false
15680 4459 Acked Read 7 Sent Done Done false
15681 4460 Acked Flush 0 Sent Done Done false
15682 4461 Acked Read 8 Sent Done Done false
15683 4462 Acked Write 4 Sent Done Done false
15684 4463 Acked Flush 0 Sent Done Done false
15685 4464 Acked Write 10 Sent Done Done false
15686 4465 Acked Write 5 Sent Done Done false
15687 4466 Acked Read 2 Sent Done Done false
15688 4467 Acked Write 9 Sent Done Done false
15689 4468 Acked Write 9 Sent Done Done false
15690 4469 Acked Write 7 Sent Done Done false
15691 4470 Acked Read 3 Sent Done Done false
15692 4471 Acked Write 10 Sent Done Done false
15693 4472 Acked Read 5 Sent Done Done false
15694 4473 Acked Read 7 Sent Done Done false
15695 4474 Acked Write 2 Sent Done Done false
15696 4475 Acked Read 10 Sent Done Done false
15697 4476 Acked Read 1 Sent Done Done false
15698 4477 Acked Read 6 Sent Done Done false
15699 4478 Acked Write 8 Sent Done Done false
15700 4479 Acked Write 7 Sent Done Done false
15701 4480 Acked Read 8 Sent Done Done false
15702 4481 Acked Read 2 Sent Done Done false
15703 4482 Acked Read 1 Sent Done Done false
15704 4483 Acked Read 4 Sent Done Done false
15705 4484 Acked Flush 0 Sent Done Done false
15706 4485 Acked Read 7 Sent Done Done false
15707 4486 Acked Read 10 Sent Done Done false
15708 4487 Acked Write 5 Sent Done Done false
15709 4488 Acked Read 6 Sent Done Done false
15710 4489 Acked Write 7 Sent Done Done false
15711 4490 Acked Read 10 Sent Done Done false
15712 4491 Acked Read 3 Sent Done Done false
15713 4492 Acked Write 10 Sent Done Done false
15714 4493 Acked Write 1 Sent Done Done false
15715 4494 Acked Write 10 Sent Done Done false
15716 4495 Acked Read 1 Sent Done Done false
15717 4496 Acked Write 8 Sent Done Done false
15718 4497 Acked Write 2 Sent Done Done false
15719 4498 Acked Write 7 Sent Done Done false
15720 4499 Acked Write 6 Sent Done Done false
15721 4500 Acked Read 9 Sent Done Done false
15722 4501 Acked Write 4 Sent Done Done false
15723 4502 Acked Write 1 Sent Done Done false
15724 4503 Acked Write 5 Sent Done Done false
15725 4504 Acked Read 4 Sent Done Done false
15726 4505 Acked Write 5 Sent Done Done false
15727 4506 Acked Read 1 Sent Done Done false
15728 4507 Acked Write 7 Sent Done Done false
15729 4508 Acked Flush 0 Sent Done Done false
15730 4509 Acked Read 9 Sent Done Done false
15731 4510 Acked Read 1 Sent Done Done false
15732 4511 Acked Write 7 Sent Done Done false
15733 4512 Acked Read 1 Sent Done Done false
15734 4513 Acked Read 9 Sent Done Done false
15735 4514 Acked Flush 0 Sent Done Done false
15736 4515 Acked Write 9 Sent Done Done false
15737 4516 Acked Read 5 Sent Done Done false
15738 4517 Acked Write 7 Sent Done Done false
15739 4518 Acked Read 2 Sent Done Done false
15740 4519 Acked Write 1 Sent Done Done false
15741 4520 Acked Flush 0 Sent Done Done false
15742 4521 Acked Write 6 Sent Done Done false
15743 4522 Acked Write 9 Sent Done Done false
15744 4523 Acked Read 2 Sent Done Done false
15745 4524 Acked Read 8 Sent Done Done false
15746 4525 Acked Write 10 Sent Done Done false
15747 4526 Acked Write 8 Sent Done Done false
15748 4527 Acked Read 1 Sent Done Done false
15749 4528 Acked Write 9 Sent Done Done false
15750 4529 Acked Write 1 Sent Done Done false
15751 4530 Acked Read 10 Sent Done Done false
15752 4531 Acked Write 10 Sent Done Done false
15753 4532 Acked Write 5 Sent Done Done false
15754 4533 Acked Write 2 Sent Done Done false
15755 4534 Acked Write 8 Sent Done Done false
15756 4535 Acked Read 6 Sent Done Done false
15757 4536 Acked Write 3 Sent Done Done false
15758 4537 Acked Read 4 Sent Done Done false
15759 4538 Acked Write 2 Sent Done Done false
15760 4539 Acked Write 2 Sent Done Done false
15761 4540 Acked Write 7 Sent Done Done false
15762 4541 Acked Read 8 Sent Done Done false
15763 4542 Acked Read 5 Sent Done Done false
15764 4543 Acked Read 5 Sent Done Done false
15765 4544 Acked Read 1 Sent Done Done false
15766 4545 Acked Write 9 Sent Done Done false
15767 4546 Acked Read 6 Sent Done Done false
15768 4547 Acked Read 6 Sent Done Done false
15769 4548 Acked Write 9 Sent Done Done false
15770 4549 Acked Write 2 Sent Done Done false
15771 4550 Acked Read 5 Sent Done Done false
15772 4551 Acked Read 8 Sent Done Done false
15773 4552 Acked Read 5 Sent Done Done false
15774 4553 Acked Read 5 Sent Done Done false
15775 4554 Acked Read 2 Sent Done Done false
15776 4555 Acked Read 10 Sent Done Done false
15777 4556 Acked Flush 0 Sent Done Done false
15778 4557 Acked Read 2 Sent Done Done false
15779 4558 Acked Read 3 Sent Done Done false
15780 4559 Acked Write 7 Sent Done Done false
15781 4560 Acked Flush 0 Sent Done Done false
15782 4561 Acked Write 5 Sent Done Done false
15783 4562 Acked Read 6 Sent Done Done false
15784 4563 Acked Read 4 Sent Done Done false
15785 4564 Acked Read 3 Sent Done Done false
15786 4565 Acked Write 6 Sent Done Done false
15787 4566 Acked Write 5 Sent Done Done false
15788 4567 Acked Read 7 Sent Done Done false
15789 4568 Acked Write 1 Sent Done Done false
15790 4569 Acked Write 10 Sent Done Done false
15791 4570 Acked Read 10 Sent Done Done false
15792 4571 Acked Flush 0 Sent Done Done false
15793 4572 Acked Read 10 Sent Done Done false
15794 4573 Acked Read 7 Sent Done Done false
15795 4574 Acked Write 6 Sent Done Done false
15796 4575 Acked Read 9 Sent Done Done false
15797 4576 Acked Read 10 Sent Done Done false
15798 4577 Acked Read 4 Sent Done Done false
15799 4578 Acked Read 5 Sent Done Done false
15800 4579 Acked Flush 0 Sent Done Done false
15801 4580 Acked Flush 0 Sent Done Done false
15802 4581 Acked Write 4 Sent Done Done false
15803 4582 Acked Write 10 Sent Done Done false
15804 4583 Acked Flush 0 Sent Done Done false
15805 4584 Acked Write 2 Sent Done Done false
15806 4585 Acked Write 9 Sent Done Done false
15807 4586 Acked Read 6 Sent Done Done false
15808 4587 Acked Read 8 Sent Done Done false
15809 4588 Acked Read 10 Sent Done Done false
15810 4589 Acked Write 10 Sent Done Done false
15811 4590 Acked Read 10 Sent Done Done false
15812 4591 Acked Write 6 Sent Done Done false
15813 4592 Acked Write 6 Sent Done Done false
15814 4593 Acked Flush 0 Sent Done Done false
15815 4594 Acked Read 3 Sent Done Done false
15816 4595 Acked Write 5 Sent Done Done false
15817 4596 Acked Read 5 Sent Done Done false
15818 4597 Acked Write 1 Sent Done Done false
15819 4598 Acked Flush 0 Sent Done Done false
15820 4599 Acked Write 1 Sent Done Done false
15821 4600 Acked Write 6 Sent Done Done false
15822 4601 Acked Write 7 Sent Done Done false
15823 4602 Acked Write 1 Sent Done Done false
15824 4603 Acked Read 6 Sent Done Done false
15825 4604 Acked Flush 0 Sent Done Done false
15826 4605 Acked Flush 0 Sent Done Done false
15827 4606 Acked Read 9 Sent Done Done false
15828 4607 Acked Write 7 Sent Done Done false
15829 4608 Acked Read 2 Sent Done Done false
15830 4609 Acked Write 1 Sent Done Done false
15831 4610 Acked Write 8 Sent Done Done false
15832 4611 Acked Read 9 Sent Done Done false
15833 4612 Acked Write 10 Sent Done Done false
15834 4613 Acked Write 7 Sent Done Done false
15835 4614 Acked Read 9 Sent Done Done false
15836 4615 Acked Write 2 Sent Done Done false
15837 4616 Acked Read 10 Sent Done Done false
15838 4617 Acked Read 9 Sent Done Done false
15839 4618 Acked Read 9 Sent Done Done false
15840 4619 Acked Flush 0 Sent Done Done false
15841 4620 Acked Read 10 Sent Done Done false
15842 4621 Acked Read 8 Sent Done Done false
15843 STATES DS:0 DS:1 DS:2 TOTAL
15844 Sent 301 0 0 301
15845 Done 0 301 301 602
15846 Skipped 0 0 0 0
15847 Error 0 0 0 0
15848 Last Flush: Some(JobId(4320)) Some(JobId(4619)) Some(JobId(4619))
15849 Downstairs last five completed: 4320 4319 4318 4317 4316
15850 Upstairs last five completed: 4621 4620 4619 4618 4617
15851 CLIENT: Up:0 ds:301 act:2
158522025-03-17T22:22:29.901ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
158532025-03-17T22:22:30.880ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
158542025-03-17T22:22:30.880ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:43022) = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
158552025-03-17T22:22:30.881ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
158562025-03-17T22:22:30.882ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
158572025-03-17T22:22:30.882ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
158582025-03-17T22:22:30.882ZINFOcrucible: send last flush ID to this DS: Some(JobId(4320)) = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
158592025-03-17T22:22:30.882ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
158602025-03-17T22:22:30.882ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(4320)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
158612025-03-17T22:22:30.882ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(4320)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
15862 ----------------------------------------------------------------
15863 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
15864 Downstairs last five completed: 4622 4621 4620 4619 4618
15865 Upstairs last five completed: 4622 4621 4620 4619 4618
15866 CLIENT: Up:0 ds:0 act:3
15867 Replay: All jobs finished
15868 Main task has work to do, go find it
15869 got dsc Stop(2)
15870 stop 2
15871 Mar 17 22:22:33.139 INFO request completed, latency_us: 429[8830] Got stop action so:false kr:false
15872 , response_code: 204, uri: /stop/cid/2, method: POST, req_id: f29f0be9-5f66-429c-955f-760a17317311, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
158732025-03-17T22:22:33.140ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
158742025-03-17T22:22:33.140ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
158752025-03-17T22:22:33.140ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
15876 Mar 17 22:22:33.140 INFO request completed, latency_us: 180, response_code: 200, uri: /state/cid/2, method: GET, req_id: ef2c49ec-904f-48ab-83cb-642b5a024787, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
158772025-03-17T22:22:33.140ZINFOcrucible: client task is exiting = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
15878 [8830] Exited with: ExitStatus(unix_wait_status(9))
15879 [8830][2] reports Exit
15880 Mar 17 22:22:37.142 INFO request completed, latency_us: 208, response_code: 200, uri: /state/cid/2, method: GET, req_id: c5d2b294-4041-44e1-9fdd-01c8d33ff8e8, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
15881 001/300 Write block 4273 len 36864 data: 4 3 3 3 3 2 3 3 4
15882 002/300 Flush
15883 003/300 Read block 1027 len 28672
15884 004/300 Write block 2096 len 32768 data: 2 2 3 4 3 3 3 3
15885 005/300 Read block 977 len 20480
15886 006/300 Flush
15887 007/300 Read block 2500 len 20480
15888 008/300 Write block 4453 len 12288 data: 4 5 4
15889 009/300 Read block 4127 len 8192
15890 010/300 Write block 265 len 20480 data: 4 5 4 4 5
15891 011/300 Read block 4261 len 32768
15892 012/300 Read block 4480 len 8192
15893 013/300 Read block 1543 len 32768
15894 014/300 Write block 2559 len 4096 data: 3
15895 015/300 Read block 1691 len 16384
15896 016/300 Write block 2675 len 24576 data: 3 2 2 2 2 2
15897 017/300 Read block 1657 len 36864
15898 018/300 Read block 4400 len 32768
15899 019/300 Read block 3216 len 28672
15900 020/300 Write block 1617 len 16384 data: 2 2 2 2
15901 021/300 Flush
15902 022/300 Read block 1845 len 16384
15903 023/300 Write block 1183 len 40960 data: 3 3 4 4 4 3 3 2 2 2
15904 024/300 Write block 1462 len 12288 data: 2 2 2
15905 025/300 Read block 2162 len 40960
15906 026/300 Read block 1040 len 32768
15907 027/300 Write block 2584 len 32768 data: 8 8 7 4 4 4 2 2
15908 028/300 Write block 4896 len 36864 data: 4 4 4 4 4 4 3 4 4
15909 029/300 Read block 2021 len 16384
15910 030/300 Write block 685 len 16384 data: 3 4 4 4
15911 031/300 Flush
15912 032/300 Read block 2938 len 20480
15913 033/300 Read block 1437 len 4096
15914 034/300 Read block 4750 len 24576
15915 035/300 Write block 907 len 24576 data: 4 5 5 6 6 5
15916 036/300 Write block 1867 len 40960 data: 6 5 6 6 7 8 5 4 6 6
15917 037/300 Read block 84 len 36864
15918 038/300 Write block 740 len 24576 data: 7 7 7 7 7 5
15919 039/300 Read block 2766 len 4096
15920 040/300 Read block 3533 len 20480
15921 041/300 Flush
15922 042/300 Write block 3021 len 16384 data: 5 5 5 5
15923 043/300 Write block 2548 len 32768 data: 4 4 4 3 3 4 4 4
15924 044/300 Read block 2591 len 24576
15925 045/300 Read block 3546 len 36864
15926 046/300 Read block 171 len 28672
15927 047/300 Read block 1790 len 24576
15928 048/300 Write block 4979 len 16384 data: 3 3 3 3
15929 049/300 Write block 888 len 24576 data: 4 4 4 3 3 3
15930 050/300 Read block 2756 len 40960
15931 051/300 Read block 296 len 16384
15932 052/300 Write block 1038 len 36864 data: 3 3 2 2 2 2 2 3 5
15933 053/300 Flush
15934 054/300 Read block 2470 len 8192
15935 055/300 Write block 2969 len 16384 data: 3 3 3 3
15936 056/300 Read block 1812 len 32768
15937 057/300 Read block 393 len 16384
15938 058/300 Read block 3763 len 24576
15939 059/300 Write block 1669 len 8192 data: 4 5
15940 060/300 Write block 1654 len 28672 data: 4 4 4 3 3 3 3
15941 061/300 Read block 2904 len 4096
15942 062/300 Read block 4346 len 16384
15943 063/300 Read block 513 len 20480
15944 064/300 Read block 1532 len 32768
15945 065/300 Read block 4556 len 4096
15946 066/300 Write block 1594 len 8192 data: 3 4
15947 067/300 Read block 1549 len 40960
15948 068/300 Read block 2434 len 12288
15949 069/300 Write block 4911 len 4096 data: 4
15950 070/300 Read block 2667 len 16384
15951 071/300 Flush
15952 072/300 Read block 13 len 24576
15953 073/300 Write block 2375 len 8192 data: 4 4
15954 074/300 Write block 2515 len 28672 data: 2 2 2 2 3 3 4
15955 075/300 Flush
15956 076/300 Read block 4876 len 12288
15957 077/300 Read block 4841 len 36864
15958 078/300 Write block 4437 len 8192 data: 3 2
15959 079/300 Read block 4573 len 20480
15960 080/300 Read block 3938 len 8192
15961 081/300 Write block 45 len 32768 data: 5 4 4 3 3 3 3 3
15962 082/300 Read block 831 len 12288
15963 083/300 Read block 865 len 40960
15964 084/300 Write block 3782 len 16384 data: 4 5 6 6
15965 085/300 Read block 2729 len 20480
15966 086/300 Flush
15967 087/300 Write block 2512 len 32768 data: 2 2 2 3 3 3 3 4
15968 088/300 Read block 557 len 8192
15969 089/300 Write block 3382 len 12288 data: 2 2 3
15970 090/300 Write block 3630 len 40960 data: 2 2 2 2 2 3 3 3 3 3
15971 091/300 Write block 1706 len 20480 data: 4 4 4 6 9
15972 092/300 Read block 2649 len 32768
15973 093/300 Read block 2998 len 24576
15974 094/300 Read block 47 len 20480
15975 095/300 Read block 2075 len 28672
15976 096/300 Flush
15977 097/300 Write block 3022 len 40960 data: 6 6 6 4 4 3 2 3 2 2
15978 098/300 Write block 3975 len 20480 data: 4 5 4 5 5
15979 099/300 Write block 3076 len 32768 data: 3 3 4 5 6 7 7 4
15980 100/300 Read block 176 len 20480
15981 101/300 Write block 3283 len 16384 data: 4 5 5 4
15982 102/300 Read block 1246 len 16384
15983 103/300 Read block 4857 len 40960
15984 104/300 Read block 4884 len 24576
15985 105/300 Read block 694 len 32768
15986 106/300 Read block 3336 len 4096
15987 107/300 Write block 220 len 32768 data: 2 2 2 2 3 3 3 3
15988 108/300 Write block 2615 len 4096 data: 3
15989 109/300 Read block 1341 len 36864
15990 110/300 Write block 3151 len 12288 data: 3 3 5
15991 111/300 Write block 972 len 24576 data: 3 3 3 4 5 4
15992 112/300 Write block 3348 len 40960 data: 6 6 6 3 2 2 2 2 2 2
15993 113/300 Write block 4411 len 20480 data: 4 4 5 5 4
15994 114/300 Read block 1178 len 20480
15995 115/300 Write block 4289 len 12288 data: 2 2 2
15996 116/300 Write block 609 len 40960 data: 3 3 2 3 3 3 3 3 3 3
15997 117/300 Flush
15998 118/300 Read block 2802 len 8192
15999 119/300 Read block 3563 len 16384
16000 120/300 Read block 4319 len 24576
16001 121/300 Flush
16002 122/300 Write block 4960 len 16384 data: 2 4 4 4
16003 123/300 Read block 1318 len 4096
16004 124/300 Write block 1782 len 20480 data: 4 4 4 3 2
16005 125/300 Read block 1781 len 12288
16006 126/300 Read block 245 len 32768
16007 127/300 Write block 705 len 32768 data: 4 2 2 3 2 3 3 3
16008 128/300 Read block 366 len 16384
16009 129/300 Read block 1944 len 28672
16010 130/300 Read block 3803 len 24576
16011 131/300 Read block 451 len 8192
16012 132/300 Read block 1217 len 8192
16013 133/300 Write block 3201 len 20480 data: 6 5 5 5 4
16014 134/300 Write block 3278 len 12288 data: 3 3 4
16015 135/300 Write block 2736 len 8192 data: 4 5
16016 136/300 Read block 4811 len 20480
16017 137/300 Read block 4473 len 20480
16018 138/300 Read block 642 len 40960
16019 139/300 Read block 179 len 4096
16020 140/300 Read block 2752 len 16384
16021 141/300 Read block 1652 len 32768
16022 142/300 Read block 3567 len 16384
16023 143/300 Write block 2499 len 4096 data: 2
16024 144/300 Write block 769 len 20480 data: 4 3 3 4 4
16025 145/300 Read block 3979 len 32768
16026 146/300 Write block 4853 len 40960 data: 3 4 4 3 4 5 4 5 5 5
16027 147/300 Write block 4017 len 8192 data: 5 4
16028 148/300 Write block 4883 len 8192 data: 2 2
16029 149/300 Write block 4360 len 20480 data: 2 2 2 2 2
16030 150/300 Write block 2106 len 24576 data: 3 2 3 4 4 4
16031 151/300 Write block 2355 len 20480 data: 2 2 2 3 4
16032 152/300 Read block 2471 len 4096
16033 153/300 Read block 4236 len 20480
16034 154/300 Read block 2539 len 8192
16035 155/300 Read block 4099 len 40960
16036 156/300 Read block 3145 len 8192
16037 157/300 Read block 290 len 24576
16038 158/300 Write block 3680 len 32768 data: 2 3 3 4 4 4 4 4
16039 159/300 Write block 4808 len 20480 data: 3 3 3 4 3
16040 160/300 Read block 1125 len 40960
16041 161/300 Read block 3412 len 36864
16042 162/300 Read block 1019 len 40960
16043 163/300 Read block 1695 len 12288
16044 164/300 Read block 2611 len 4096
16045 165/300 Write block 4157 len 4096 data: 5
16046 166/300 Read block 3015 len 32768
16047 167/300 Read block 2550 len 36864
16048 168/300 Read block 1216 len 4096
16049 169/300 Read block 3136 len 36864
16050 170/300 Read block 4690 len 28672
16051 171/300 Flush
16052 172/300 Read block 468 len 36864
16053 173/300 Flush
16054 174/300 Read block 1663 len 12288
16055 175/300 Read block 3510 len 16384
16056 176/300 Read block 407 len 24576
16057 177/300 Write block 4329 len 28672 data: 4 4 4 5 4 4 4
16058 178/300 Flush
16059 179/300 Flush
16060 180/300 Read block 724 len 32768
16061 181/300 Flush
16062 182/300 Read block 1629 len 8192
16063 183/300 Read block 3751 len 12288
16064 184/300 Read block 687 len 8192
16065 185/300 Write block 2258 len 20480 data: 4 4 4 5 5
16066 186/300 Read block 1157 len 24576
16067 187/300 Write block 3536 len 12288 data: 6 6 7
16068 188/300 Read block 2382 len 16384
16069 189/300 Write block 3637 len 16384 data: 4 4 4 4
16070 190/300 Read block 3109 len 12288
16071 191/300 Write block 3544 len 12288 data: 5 6 6
16072 192/300 Flush
16073 193/300 Read block 4928 len 16384
16074 194/300 Read block 769 len 40960
16075 195/300 Read block 2580 len 32768
16076 196/300 Write block 1422 len 24576 data: 2 2 3 4 2 3
16077 197/300 Read block 1221 len 40960
16078 198/300 Read block 4 len 12288
16079 199/300 Read block 525 len 4096
16080 200/300 Read block 2658 len 24576
16081 201/300 Write block 2192 len 20480 data: 5 5 5 3 3
16082 202/300 Read block 2786 len 32768
16083 203/300 Write block 2931 len 40960 data: 3 4 2 2 3 3 3 3 4 3
16084 204/300 Write block 1403 len 20480 data: 3 2 2 2 2
16085 205/300 Read block 183 len 16384
16086 206/300 Write block 3883 len 8192 data: 3 3
16087 207/300 Read block 3793 len 4096
16088 208/300 Read block 3129 len 12288
16089 209/300 Read block 435 len 28672
16090 210/300 Read block 2404 len 28672
16091 211/300 Write block 2667 len 8192 data: 3 4
16092 212/300 Write block 3920 len 24576 data: 2 2 2 2 3 4
16093 213/300 Write block 2420 len 4096 data: 6
16094 214/300 Read block 3489 len 8192
16095 215/300 Read block 3082 len 4096
16096 216/300 Read block 4986 len 12288
16097 217/300 Write block 4301 len 16384 data: 3 2 2 2
16098 218/300 Write block 2446 len 8192 data: 7 7
16099 219/300 Flush
16100 220/300 Read block 4725 len 4096
16101 221/300 Read block 2059 len 28672
16102 222/300 Read block 1106 len 32768
16103 223/300 Read block 4981 len 24576
16104 224/300 Read block 3912 len 20480
16105 225/300 Write block 4490 len 24576 data: 3 2 2 2 3 3
16106 226/300 Read block 3003 len 32768
16107 227/300 Write block 1667 len 32768 data: 5 4 5 6 5 2 3 3
16108 228/300 Read block 296 len 12288
16109 229/300 Read block 1967 len 32768
16110 230/300 Write block 313 len 36864 data: 3 4 4 4 4 6 4 4 4
16111 231/300 Read block 824 len 8192
16112 232/300 Write block 2109 len 16384 data: 5 5 5 4
16113 233/300 Write block 291 len 32768 data: 4 4 4 4 5 5 5 4
16114 234/300 Read block 3542 len 8192
16115 235/300 Write block 3818 len 32768 data: 2 3 3 3 3 3 3 3
16116 236/300 Read block 2284 len 32768
16117 237/300 Read block 297 len 32768
16118 238/300 Write block 4880 len 8192 data: 2 2
16119 239/300 Flush
16120 240/300 Write block 2822 len 8192 data: 4 4
16121 241/300 Read block 866 len 8192
16122 242/300 Write block 4933 len 36864 data: 5 3 4 4 4 3 4 5 6
16123 243/300 Write block 367 len 40960 data: 6 6 6 6 7 7 5 4 5 5
16124 244/300 Read block 3378 len 36864
16125 245/300 Write block 2277 len 32768 data: 4 4 4 4 5 3 3 3
16126 246/300 Read block 1636 len 12288
16127 247/300 Write block 1234 len 40960 data: 4 3 3 3 2 3 3 3 3 3
16128 248/300 Read block 3902 len 24576
16129 249/300 Write block 465 len 32768 data: 5 4 5 5 4 4 3 4
16130 250/300 Read block 4079 len 24576
16131 251/300 Read block 4712 len 32768
16132 252/300 Write block 2141 len 16384 data: 6 5 4 3
16133 253/300 Write block 1031 len 8192 data: 3 3
16134 254/300 Read block 726 len 36864
16135 255/300 Read block 655 len 24576
16136 256/300 Write block 4902 len 24576 data: 4 5 5 4 4 4
16137 257/300 Read block 4012 len 12288
16138 258/300 Write block 2394 len 28672 data: 5 3 3 4 4 4 4
16139 259/300 Read block 3043 len 16384
16140 260/300 Read block 3676 len 16384
16141 261/300 Read block 290 len 20480
16142 262/300 Read block 2925 len 4096
16143 263/300 Flush
16144 264/300 Read block 1859 len 40960
16145 265/300 Read block 822 len 12288
16146 266/300 Flush
16147 267/300 Write block 688 len 8192 data: 5 4
16148 268/300 Read block 2137 len 4096
16149 269/300 Read block 872 len 16384
16150 270/300 Flush
16151 271/300 Read block 4508 len 28672
16152 272/300 Write block 321 len 24576 data: 5 3 3 3 4 4
16153 273/300 Write block 2798 len 4096 data: 2
16154 274/300 Read block 1386 len 32768
16155 275/300 Read block 3197 len 36864
16156 276/300 Read block 4479 len 32768
16157 277/300 Read block 2715 len 4096
16158 278/300 Read block 1025 len 20480
16159 279/300 Flush
16160 280/300 Read block 3698 len 16384
16161 281/300 Read block 2395 len 28672
16162 282/300 Flush
16163 283/300 Read block 3804 len 20480
16164 284/300 Read block 1003 len 12288
16165 285/300 Write block 2896 len 36864 data: 2 2 4 5 7 7 6 5 5
16166 286/300 Flush
16167 287/300 Write block 341 len 32768 data: 3 3 3 2 2 4 4 3
16168 288/300 Read block 1962 len 12288
16169 289/300 Write block 3345 len 20480 data: 6 5 6 7 7
16170 290/300 Read block 2899 len 36864
16171 291/300 Read block 1347 len 36864
16172 292/300 Read block 713 len 32768
16173 293/300 Write block 3919 len 4096 data: 2
16174 294/300 Write block 2420 len 4096 data: 7
16175 295/300 Write block 3852 len 4096 data: 5
16176 296/300 Read block 103 len 24576
16177 297/300 Read block 2126 len 40960
16178 298/300 Write block 266 len 4096 data: 6
16179 299/300 Write block 3294 len 36864 data: 2 2 2 2 2 3 3 3 4
16180 300/300 Write block 4238 len 20480 data: 3 3 3 4 5
16181 301/300 Write block 682 len 32768 data: 2 3 3 4 5 5 6 5
16182 Main task has work to do, go find it
16183 got dsc Start(2)
16184 start 2
16185 Mar 17 22:22:37.393 INFO request completed, latency_us: 258, response_code: 204, uri: /start/cid/2, method: POST, req_id: d49cb37e-e6bb-4a5d-bad4-b8d705e4e93f, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
16186 Replay: started 2, returned:Ok(())
16187 ----------------------------------------------------------------
16188 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16189 States: Active Active Offline
16190 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16191 4623 Acked Write 9 Done Done Sent false
16192 4624 Acked Flush 0 Done Done Sent false
16193 4625 Acked Read 7 Done Done Sent false
16194 4626 Acked Write 8 Done Done Sent false
16195 4627 Acked Read 5 Done Done Sent false
16196 4628 Acked Flush 0 Done Done Sent false
16197 4629 Acked Read 5 Done Done Sent false
16198 4630 Acked Write 3 Done Done Sent false
16199 4631 Acked Read 2 Done Done Sent false
16200 4632 Acked Write 5 Done Done Sent false
16201 4633 Acked Read 8 Done Done Sent false
16202 4634 Acked Read 2 Done Done Sent false
16203 4635 Acked Read 8 Done Done Sent false
16204 4636 Acked Write 1 Done Done Sent false
16205 4637 Acked Read 4 Done Done Sent false
16206 4638 Acked Write 6 Done Done Sent false
16207 4639 Acked Read 9 Done Done Sent false
16208 4640 Acked Read 8 Done Done Sent false
16209 4641 Acked Read 7 Done Done Sent false
16210 4642 Acked Write 4 Done Done Sent false
16211 4643 Acked Flush 0 Done Done Sent false
16212 4644 Acked Read 4 Done Done Sent false
16213 4645 Acked Write 10 Done Done Sent false
16214 4646 Acked Write 3 Done Done Sent false
16215 4647 Acked Read 10 Done Done Sent false
16216 4648 Acked Read 8 Done Done Sent false
16217 4649 Acked Write 8 Done Done Sent false
16218 4650 Acked Write 9 Done Done Sent false
16219 4651 Acked Read 4 Done Done Sent false
16220 4652 Acked Write 4 Done Done Sent false
16221 4653 Acked Flush 0 Done Done Sent false
16222 4654 Acked Read 5 Done Done Sent false
16223 4655 Acked Read 1 Done Done Sent false
16224 4656 Acked Read 6 Done Done Sent false
16225 4657 Acked Write 6 Done Done Sent false
16226 4658 Acked Write 10 Done Done Sent false
16227 4659 Acked Read 9 Done Done Sent false
16228 4660 Acked Write 6 Done Done Sent false
16229 4661 Acked Read 1 Done Done Sent false
16230 4662 Acked Read 5 Done Done Sent false
16231 4663 Acked Flush 0 Done Done Sent false
16232 4664 Acked Write 4 Done Done Sent false
16233 4665 Acked Write 8 Done Done Sent false
16234 4666 Acked Read 6 Done Done Sent false
16235 4667 Acked Read 9 Done Done Sent false
16236 4668 Acked Read 7 Done Done Sent false
16237 4669 Acked Read 6 Done Done Sent false
16238 4670 Acked Write 4 Done Done Sent false
16239 4671 Acked Write 6 Done Done Sent false
16240 4672 Acked Read 10 Done Done Sent false
16241 4673 Acked Read 4 Done Done Sent false
16242 4674 Acked Write 9 Done Done Sent false
16243 4675 Acked Flush 0 Done Done Sent false
16244 4676 Acked Read 2 Done Done Sent false
16245 4677 Acked Write 4 Done Done Sent false
16246 4678 Acked Read 8 Done Done Sent false
16247 4679 Acked Read 4 Done Done Sent false
16248 4680 Acked Read 6 Done Done Sent false
16249 4681 Acked Write 2 Done Done Sent false
16250 4682 Acked Write 7 Done Done Sent false
16251 4683 Acked Read 1 Done Done Sent false
16252 4684 Acked Read 4 Done Done Sent false
16253 4685 Acked Read 5 Done Done Sent false
16254 4686 Acked Read 8 Done Done Sent false
16255 4687 Acked Read 1 Done Done Sent false
16256 4688 Acked Write 2 Done Done Sent false
16257 4689 Acked Read 10 Done Done Sent false
16258 4690 Acked Read 3 Done Done Sent false
16259 4691 Acked Write 1 Done Done Sent false
16260 4692 Acked Read 4 Done Done Sent false
16261 4693 Acked Flush 0 Done Done Sent false
16262 4694 Acked Read 6 Done Done Sent false
16263 4695 Acked Write 2 Done Done Sent false
16264 4696 Acked Write 7 Done Done Sent false
16265 4697 Acked Flush 0 Done Done Sent false
16266 4698 Acked Read 3 Done Done Sent false
16267 4699 Acked Read 9 Done Done Sent false
16268 4700 Acked Write 2 Done Done Sent false
16269 4701 Acked Read 5 Done Done Sent false
16270 4702 Acked Read 2 Done Done Sent false
16271 4703 Acked Write 8 Done Done Sent false
16272 4704 Acked Read 3 Done Done Sent false
16273 4705 Acked Read 10 Done Done Sent false
16274 4706 Acked Write 4 Done Done Sent false
16275 4707 Acked Read 5 Done Done Sent false
16276 4708 Acked Flush 0 Done Done Sent false
16277 4709 Acked Write 8 Done Done Sent false
16278 4710 Acked Read 2 Done Done Sent false
16279 4711 Acked Write 3 Done Done Sent false
16280 4712 Acked Write 10 Done Done Sent false
16281 4713 Acked Write 5 Done Done Sent false
16282 4714 Acked Read 8 Done Done Sent false
16283 4715 Acked Read 6 Done Done Sent false
16284 4716 Acked Read 5 Done Done Sent false
16285 4717 Acked Read 7 Done Done Sent false
16286 4718 Acked Flush 0 Done Done Sent false
16287 4719 Acked Write 10 Done Done Sent false
16288 4720 Acked Write 5 Done Done Sent false
16289 4721 Acked Write 8 Done Done Sent false
16290 4722 Acked Read 5 Done Done Sent false
16291 4723 Acked Write 4 Done Done Sent false
16292 4724 Acked Read 4 Done Done Sent false
16293 4725 Acked Read 10 Done Done Sent false
16294 4726 Acked Read 6 Done Done Sent false
16295 4727 Acked Read 8 Done Done Sent false
16296 4728 Acked Read 1 Done Done Sent false
16297 4729 Acked Write 8 Done Done Sent false
16298 4730 Acked Write 1 Done Done Sent false
16299 4731 Acked Read 9 Done Done Sent false
16300 4732 Acked Write 3 Done Done Sent false
16301 4733 Acked Write 6 Done Done Sent false
16302 4734 Acked Write 10 Done Done Sent false
16303 4735 Acked Write 5 Done Done Sent false
16304 4736 Acked Read 5 Done Done Sent false
16305 4737 Acked Write 3 Done Done Sent false
16306 4738 Acked Write 10 Done Done Sent false
16307 4739 Acked Flush 0 Done Done Sent false
16308 4740 Acked Read 2 Done Done Sent false
16309 4741 Acked Read 4 Done Done Sent false
16310 4742 Acked Read 6 Done Done Sent false
16311 4743 Acked Flush 0 Done Done Sent false
16312 4744 Acked Write 4 Done Done Sent false
16313 4745 Acked Read 1 Done Done Sent false
16314 4746 Acked Write 5 Done Done Sent false
16315 4747 Acked Read 3 Done Done Sent false
16316 4748 Acked Read 8 Done Done Sent false
16317 4749 Acked Write 8 Done Done Sent false
16318 4750 Acked Read 4 Done Done Sent false
16319 4751 Acked Read 7 Done Done Sent false
16320 4752 Acked Read 6 Done Done Sent false
16321 4753 Acked Read 2 Done Done Sent false
16322 4754 Acked Read 2 Done Done Sent false
16323 4755 Acked Write 5 Done Done Sent false
16324 4756 Acked Write 3 Done Done Sent false
16325 4757 Acked Write 2 Done Done Sent false
16326 4758 Acked Read 5 Done Done Sent false
16327 4759 Acked Read 5 Done Done Sent false
16328 4760 Acked Read 10 Done Done Sent false
16329 4761 Acked Read 1 Done Done Sent false
16330 4762 Acked Read 4 Done Done Sent false
16331 4763 Acked Read 8 Done Done Sent false
16332 4764 Acked Read 4 Done Done Sent false
16333 4765 Acked Write 1 Done Done Sent false
16334 4766 Acked Write 5 Done Done Sent false
16335 4767 Acked Read 8 Done Done Sent false
16336 4768 Acked Write 10 Done Done Sent false
16337 4769 Acked Write 2 Done Done Sent false
16338 4770 Acked Write 2 Done Done Sent false
16339 4771 Acked Write 5 Done Done Sent false
16340 4772 Acked Write 6 Done Done Sent false
16341 4773 Acked Write 5 Done Done Sent false
16342 4774 Acked Read 1 Done Done Sent false
16343 4775 Acked Read 5 Done Done Sent false
16344 4776 Acked Read 2 Done Done Sent false
16345 4777 Acked Read 10 Done Done Sent false
16346 4778 Acked Read 2 Done Done Sent false
16347 4779 Acked Read 6 Done Done Sent false
16348 4780 Acked Write 8 Done Done Sent false
16349 4781 Acked Write 5 Done Done Sent false
16350 4782 Acked Read 10 Done Done Sent false
16351 4783 Acked Read 9 Done Done Sent false
16352 4784 Acked Read 10 Done Done Sent false
16353 4785 Acked Read 3 Done Done Sent false
16354 4786 Acked Read 1 Done Done Sent false
16355 4787 Acked Write 1 Done Done Sent false
16356 4788 Acked Read 8 Done Done Sent false
16357 4789 Acked Read 9 Done Done Sent false
16358 4790 Acked Read 1 Done Done Sent false
16359 4791 Acked Read 9 Done Done Sent false
16360 4792 Acked Read 7 Done Done Sent false
16361 4793 Acked Flush 0 Done Done Sent false
16362 4794 Acked Read 9 Done Done Sent false
16363 4795 Acked Flush 0 Done Done Sent false
16364 4796 Acked Read 3 Done Done Sent false
16365 4797 Acked Read 4 Done Done Sent false
16366 4798 Acked Read 6 Done Done Sent false
16367 4799 Acked Write 7 Done Done Sent false
16368 4800 Acked Flush 0 Done Done Sent false
16369 4801 Acked Flush 0 Done Done Sent false
16370 4802 Acked Read 8 Done Done Sent false
16371 4803 Acked Flush 0 Done Done Sent false
16372 4804 Acked Read 2 Done Done Sent false
16373 4805 Acked Read 3 Done Done Sent false
16374 4806 Acked Read 2 Done Done Sent false
16375 4807 Acked Write 5 Done Done Sent false
16376 4808 Acked Read 6 Done Done Sent false
16377 4809 Acked Write 3 Done Done Sent false
16378 4810 Acked Read 4 Done Done Sent false
16379 4811 Acked Write 4 Done Done Sent false
16380 4812 Acked Read 3 Done Done Sent false
16381 4813 Acked Write 3 Done Done Sent false
16382 4814 Acked Flush 0 Done Done Sent false
16383 4815 Acked Read 4 Done Done Sent false
16384 4816 Acked Read 10 Done Done Sent false
16385 4817 Acked Read 8 Done Done Sent false
16386 4818 Acked Write 6 Done Done Sent false
16387 4819 Acked Read 10 Done Done Sent false
16388 4820 Acked Read 3 Done Done Sent false
16389 4821 Acked Read 1 Done Done Sent false
16390 4822 Acked Read 6 Done Done Sent false
16391 4823 Acked Write 5 Done Done Sent false
16392 4824 Acked Read 8 Done Done Sent false
16393 4825 Acked Write 10 Done Done Sent false
16394 4826 Acked Write 5 Done Done Sent false
16395 4827 Acked Read 4 Done Done Sent false
16396 4828 Acked Write 2 Done Done Sent false
16397 4829 Acked Read 1 Done Done Sent false
16398 4830 Acked Read 3 Done Done Sent false
16399 4831 Acked Read 7 Done Done Sent false
16400 4832 Acked Read 7 Done Done Sent false
16401 4833 Acked Write 2 Done Done Sent false
16402 4834 Acked Write 6 Done Done Sent false
16403 4835 Acked Write 1 Done Done Sent false
16404 4836 Acked Read 2 Done Done Sent false
16405 4837 Acked Read 1 Done Done Sent false
16406 4838 Acked Read 3 Done Done Sent false
16407 4839 Acked Write 4 Done Done Sent false
16408 4840 Acked Write 2 Done Done Sent false
16409 4841 Acked Flush 0 Done Done Sent false
16410 4842 Acked Read 1 Done Done Sent false
16411 4843 Acked Read 7 Done Done Sent false
16412 4844 Acked Read 8 Done Done Sent false
16413 4845 Acked Read 6 Done Done Sent false
16414 4846 Acked Read 5 Done Done Sent false
16415 4847 Acked Write 6 Done Done Sent false
16416 4848 Acked Read 8 Done Done Sent false
16417 4849 Acked Write 8 Done Done Sent false
16418 4850 Acked Read 3 Done Done Sent false
16419 4851 Acked Read 8 Done Done Sent false
16420 4852 Acked Write 9 Done Done Sent false
16421 4853 Acked Read 2 Done Done Sent false
16422 4854 Acked Write 4 Done Done Sent false
16423 4855 Acked Write 8 Done Done Sent false
16424 4856 Acked Read 2 Done Done Sent false
16425 4857 Acked Write 8 Done Done Sent false
16426 4858 Acked Read 8 Done Done Sent false
16427 4859 Acked Read 8 Done Done Sent false
16428 4860 Acked Write 2 Done Done Sent false
16429 4861 Acked Flush 0 Done Done Sent false
16430 4862 Acked Write 2 Done Done Sent false
16431 4863 Acked Read 2 Done Done Sent false
16432 4864 Acked Write 9 Done Done Sent false
16433 4865 Acked Write 10 Done Done Sent false
16434 4866 Acked Read 9 Done Done Sent false
16435 4867 Acked Write 8 Done Done Sent false
16436 4868 Acked Read 3 Done Done Sent false
16437 4869 Acked Write 10 Done Done Sent false
16438 4870 Acked Read 6 Done Done Sent false
16439 4871 Acked Write 8 Done Done Sent false
16440 4872 Acked Read 6 Done Done Sent false
16441 4873 Acked Read 8 Done Done Sent false
16442 4874 Acked Write 4 Done Done Sent false
16443 4875 Acked Write 2 Done Done Sent false
16444 4876 Acked Read 9 Done Done Sent false
16445 4877 Acked Read 6 Done Done Sent false
16446 4878 Acked Write 6 Done Done Sent false
16447 4879 Acked Read 3 Done Done Sent false
16448 4880 Acked Write 7 Done Done Sent false
16449 4881 Acked Read 4 Done Done Sent false
16450 4882 Acked Read 4 Done Done Sent false
16451 4883 Acked Read 5 Done Done Sent false
16452 4884 Acked Read 1 Done Done Sent false
16453 4885 Acked Flush 0 Done Done Sent false
16454 4886 Acked Read 10 Done Done Sent false
16455 4887 Acked Read 3 Done Done Sent false
16456 4888 Acked Flush 0 Done Done Sent false
16457 4889 Acked Write 2 Done Done Sent false
16458 4890 Acked Read 1 Done Done Sent false
16459 4891 Acked Read 4 Done Done Sent false
16460 4892 Acked Flush 0 Done Done Sent false
16461 4893 Acked Read 7 Done Done Sent false
16462 4894 Acked Write 6 Done Done Sent false
16463 4895 Acked Write 1 Done Done Sent false
16464 4896 Acked Read 8 Done Done Sent false
16465 4897 Acked Read 9 Done Done Sent false
16466 4898 Acked Read 8 Done Done Sent false
16467 4899 Acked Read 1 Done Done Sent false
16468 4900 Acked Read 5 Done Done Sent false
16469 4901 Acked Flush 0 Done Done Sent false
16470 4902 Acked Read 4 Done Done Sent false
16471 4903 Acked Read 7 Done Done Sent false
16472 4904 Acked Flush 0 Done Done Sent false
16473 4905 Acked Read 5 Done Done Sent false
16474 4906 Acked Read 3 Done Done Sent false
16475 4907 Acked Write 9 Done Done Sent false
16476 4908 Acked Flush 0 Done Done Sent false
16477 4909 Acked Write 8 Done Done Sent false
16478 4910 Acked Read 3 Done Done Sent false
16479 4911 Acked Write 5 Done Done Sent false
16480 4912 Acked Read 9 Done Done Sent false
16481 4913 Acked Read 9 Done Done Sent false
16482 4914 Acked Read 8 Done Done Sent false
16483 4915 Acked Write 1 Done Done Sent false
16484 4916 Acked Write 1 Done Done Sent false
16485 4917 Acked Write 1 Done Done Sent false
16486 4918 Acked Read 6 Done Done Sent false
16487 4919 Acked Read 10 Done Done Sent false
16488 4920 Acked Write 1 Done Done Sent false
16489 4921 Acked Write 9 Done Done Sent false
16490 4922 Acked Write 5 Done Done Sent false
16491 4923 Acked Write 8 Done Done Sent false
16492 STATES DS:0 DS:1 DS:2 TOTAL
16493 Sent 0 0 301 301
16494 Done 301 301 0 602
16495 Skipped 0 0 0 0
16496 Error 0 0 0 0
16497 Last Flush: Some(JobId(4908)) Some(JobId(4908)) Some(JobId(4622))
16498 Downstairs last five completed: 4622 4621 4620 4619 4618
16499 Upstairs last five completed: 4923 4922 4921 4920 4919
16500 CLIENT: Up:0 ds:301 act:2
165012025-03-17T22:22:37.914ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
165022025-03-17T22:22:38.916ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
165032025-03-17T22:22:39.918ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
16504 [8830] got start action
16505 [8830] I am going to restart
16506 Starting downstairs at port 8830
16507 Make output file at "/tmp/dsc/downstairs-8830.txt"
16508 [8830][2] reports Starting
16509 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1224)
16510 [8830][2] reports Running
165112025-03-17T22:22:41.384ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
16512 ----------------------------------------------------------------
16513 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16514 States: Active Active Offline
16515 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16516 4623 Acked Write 9 Done Done Sent false
16517 4624 Acked Flush 0 Done Done Sent false
16518 4625 Acked Read 7 Done Done Sent false
16519 4626 Acked Write 8 Done Done Sent false
16520 4627 Acked Read 5 Done Done Sent false
16521 4628 Acked Flush 0 Done Done Sent false
16522 4629 Acked Read 5 Done Done Sent false
16523 4630 Acked Write 3 Done Done Sent false
16524 4631 Acked Read 2 Done Done Sent false
16525 4632 Acked Write 5 Done Done Sent false
16526 4633 Acked Read 8 Done Done Sent false
16527 4634 Acked Read 2 Done Done Sent false
16528 4635 Acked Read 8 Done Done Sent false
16529 4636 Acked Write 1 Done Done Sent false
16530 4637 Acked Read 4 Done Done Sent false
16531 4638 Acked Write 6 Done Done Sent false
16532 4639 Acked Read 9 Done Done Sent false
16533 4640 Acked Read 8 Done Done Sent false
16534 4641 Acked Read 7 Done Done Sent false
16535 4642 Acked Write 4 Done Done Sent false
16536 4643 Acked Flush 0 Done Done Sent false
16537 4644 Acked Read 4 Done Done Sent false
16538 4645 Acked Write 10 Done Done Sent false
16539 4646 Acked Write 3 Done Done Sent false
16540 4647 Acked Read 10 Done Done Sent false
16541 4648 Acked Read 8 Done Done Sent false
16542 4649 Acked Write 8 Done Done Sent false
16543 4650 Acked Write 9 Done Done Sent false
16544 4651 Acked Read 4 Done Done Sent false
16545 4652 Acked Write 4 Done Done Sent false
16546 4653 Acked Flush 0 Done Done Sent false
16547 4654 Acked Read 5 Done Done Sent false
16548 4655 Acked Read 1 Done Done Sent false
16549 4656 Acked Read 6 Done Done Sent false
16550 4657 Acked Write 6 Done Done Sent false
16551 4658 Acked Write 10 Done Done Sent false
16552 4659 Acked Read 9 Done Done Sent false
16553 4660 Acked Write 6 Done Done Sent false
16554 4661 Acked Read 1 Done Done Sent false
16555 4662 Acked Read 5 Done Done Sent false
16556 4663 Acked Flush 0 Done Done Sent false
16557 4664 Acked Write 4 Done Done Sent false
16558 4665 Acked Write 8 Done Done Sent false
16559 4666 Acked Read 6 Done Done Sent false
16560 4667 Acked Read 9 Done Done Sent false
16561 4668 Acked Read 7 Done Done Sent false
16562 4669 Acked Read 6 Done Done Sent false
16563 4670 Acked Write 4 Done Done Sent false
16564 4671 Acked Write 6 Done Done Sent false
16565 4672 Acked Read 10 Done Done Sent false
16566 4673 Acked Read 4 Done Done Sent false
16567 4674 Acked Write 9 Done Done Sent false
16568 4675 Acked Flush 0 Done Done Sent false
16569 4676 Acked Read 2 Done Done Sent false
16570 4677 Acked Write 4 Done Done Sent false
16571 4678 Acked Read 8 Done Done Sent false
16572 4679 Acked Read 4 Done Done Sent false
16573 4680 Acked Read 6 Done Done Sent false
16574 4681 Acked Write 2 Done Done Sent false
16575 4682 Acked Write 7 Done Done Sent false
16576 4683 Acked Read 1 Done Done Sent false
16577 4684 Acked Read 4 Done Done Sent false
16578 4685 Acked Read 5 Done Done Sent false
16579 4686 Acked Read 8 Done Done Sent false
16580 4687 Acked Read 1 Done Done Sent false
16581 4688 Acked Write 2 Done Done Sent false
16582 4689 Acked Read 10 Done Done Sent false
16583 4690 Acked Read 3 Done Done Sent false
16584 4691 Acked Write 1 Done Done Sent false
16585 4692 Acked Read 4 Done Done Sent false
16586 4693 Acked Flush 0 Done Done Sent false
16587 4694 Acked Read 6 Done Done Sent false
16588 4695 Acked Write 2 Done Done Sent false
16589 4696 Acked Write 7 Done Done Sent false
16590 4697 Acked Flush 0 Done Done Sent false
16591 4698 Acked Read 3 Done Done Sent false
16592 4699 Acked Read 9 Done Done Sent false
16593 4700 Acked Write 2 Done Done Sent false
16594 4701 Acked Read 5 Done Done Sent false
16595 4702 Acked Read 2 Done Done Sent false
16596 4703 Acked Write 8 Done Done Sent false
16597 4704 Acked Read 3 Done Done Sent false
16598 4705 Acked Read 10 Done Done Sent false
16599 4706 Acked Write 4 Done Done Sent false
16600 4707 Acked Read 5 Done Done Sent false
16601 4708 Acked Flush 0 Done Done Sent false
16602 4709 Acked Write 8 Done Done Sent false
16603 4710 Acked Read 2 Done Done Sent false
16604 4711 Acked Write 3 Done Done Sent false
16605 4712 Acked Write 10 Done Done Sent false
16606 4713 Acked Write 5 Done Done Sent false
16607 4714 Acked Read 8 Done Done Sent false
16608 4715 Acked Read 6 Done Done Sent false
16609 4716 Acked Read 5 Done Done Sent false
16610 4717 Acked Read 7 Done Done Sent false
16611 4718 Acked Flush 0 Done Done Sent false
16612 4719 Acked Write 10 Done Done Sent false
16613 4720 Acked Write 5 Done Done Sent false
16614 4721 Acked Write 8 Done Done Sent false
16615 4722 Acked Read 5 Done Done Sent false
16616 4723 Acked Write 4 Done Done Sent false
16617 4724 Acked Read 4 Done Done Sent false
16618 4725 Acked Read 10 Done Done Sent false
16619 4726 Acked Read 6 Done Done Sent false
16620 4727 Acked Read 8 Done Done Sent false
16621 4728 Acked Read 1 Done Done Sent false
16622 4729 Acked Write 8 Done Done Sent false
16623 4730 Acked Write 1 Done Done Sent false
16624 4731 Acked Read 9 Done Done Sent false
16625 4732 Acked Write 3 Done Done Sent false
16626 4733 Acked Write 6 Done Done Sent false
16627 4734 Acked Write 10 Done Done Sent false
16628 4735 Acked Write 5 Done Done Sent false
16629 4736 Acked Read 5 Done Done Sent false
16630 4737 Acked Write 3 Done Done Sent false
16631 4738 Acked Write 10 Done Done Sent false
16632 4739 Acked Flush 0 Done Done Sent false
16633 4740 Acked Read 2 Done Done Sent false
16634 4741 Acked Read 4 Done Done Sent false
16635 4742 Acked Read 6 Done Done Sent false
16636 4743 Acked Flush 0 Done Done Sent false
16637 4744 Acked Write 4 Done Done Sent false
16638 4745 Acked Read 1 Done Done Sent false
16639 4746 Acked Write 5 Done Done Sent false
16640 4747 Acked Read 3 Done Done Sent false
16641 4748 Acked Read 8 Done Done Sent false
16642 4749 Acked Write 8 Done Done Sent false
16643 4750 Acked Read 4 Done Done Sent false
16644 4751 Acked Read 7 Done Done Sent false
16645 4752 Acked Read 6 Done Done Sent false
16646 4753 Acked Read 2 Done Done Sent false
16647 4754 Acked Read 2 Done Done Sent false
16648 4755 Acked Write 5 Done Done Sent false
16649 4756 Acked Write 3 Done Done Sent false
16650 4757 Acked Write 2 Done Done Sent false
16651 4758 Acked Read 5 Done Done Sent false
16652 4759 Acked Read 5 Done Done Sent false
16653 4760 Acked Read 10 Done Done Sent false
16654 4761 Acked Read 1 Done Done Sent false
16655 4762 Acked Read 4 Done Done Sent false
16656 4763 Acked Read 8 Done Done Sent false
16657 4764 Acked Read 4 Done Done Sent false
16658 4765 Acked Write 1 Done Done Sent false
16659 4766 Acked Write 5 Done Done Sent false
16660 4767 Acked Read 8 Done Done Sent false
16661 4768 Acked Write 10 Done Done Sent false
16662 4769 Acked Write 2 Done Done Sent false
16663 4770 Acked Write 2 Done Done Sent false
16664 4771 Acked Write 5 Done Done Sent false
16665 4772 Acked Write 6 Done Done Sent false
16666 4773 Acked Write 5 Done Done Sent false
16667 4774 Acked Read 1 Done Done Sent false
16668 4775 Acked Read 5 Done Done Sent false
16669 4776 Acked Read 2 Done Done Sent false
16670 4777 Acked Read 10 Done Done Sent false
16671 4778 Acked Read 2 Done Done Sent false
16672 4779 Acked Read 6 Done Done Sent false
16673 4780 Acked Write 8 Done Done Sent false
16674 4781 Acked Write 5 Done Done Sent false
16675 4782 Acked Read 10 Done Done Sent false
16676 4783 Acked Read 9 Done Done Sent false
16677 4784 Acked Read 10 Done Done Sent false
16678 4785 Acked Read 3 Done Done Sent false
16679 4786 Acked Read 1 Done Done Sent false
16680 4787 Acked Write 1 Done Done Sent false
16681 4788 Acked Read 8 Done Done Sent false
16682 4789 Acked Read 9 Done Done Sent false
16683 4790 Acked Read 1 Done Done Sent false
16684 4791 Acked Read 9 Done Done Sent false
16685 4792 Acked Read 7 Done Done Sent false
16686 4793 Acked Flush 0 Done Done Sent false
16687 4794 Acked Read 9 Done Done Sent false
16688 4795 Acked Flush 0 Done Done Sent false
16689 4796 Acked Read 3 Done Done Sent false
16690 4797 Acked Read 4 Done Done Sent false
16691 4798 Acked Read 6 Done Done Sent false
16692 4799 Acked Write 7 Done Done Sent false
16693 4800 Acked Flush 0 Done Done Sent false
16694 4801 Acked Flush 0 Done Done Sent false
16695 4802 Acked Read 8 Done Done Sent false
16696 4803 Acked Flush 0 Done Done Sent false
16697 4804 Acked Read 2 Done Done Sent false
16698 4805 Acked Read 3 Done Done Sent false
16699 4806 Acked Read 2 Done Done Sent false
16700 4807 Acked Write 5 Done Done Sent false
16701 4808 Acked Read 6 Done Done Sent false
16702 4809 Acked Write 3 Done Done Sent false
16703 4810 Acked Read 4 Done Done Sent false
16704 4811 Acked Write 4 Done Done Sent false
16705 4812 Acked Read 3 Done Done Sent false
16706 4813 Acked Write 3 Done Done Sent false
16707 4814 Acked Flush 0 Done Done Sent false
16708 4815 Acked Read 4 Done Done Sent false
16709 4816 Acked Read 10 Done Done Sent false
16710 4817 Acked Read 8 Done Done Sent false
16711 4818 Acked Write 6 Done Done Sent false
16712 4819 Acked Read 10 Done Done Sent false
16713 4820 Acked Read 3 Done Done Sent false
16714 4821 Acked Read 1 Done Done Sent false
16715 4822 Acked Read 6 Done Done Sent false
16716 4823 Acked Write 5 Done Done Sent false
16717 4824 Acked Read 8 Done Done Sent false
16718 4825 Acked Write 10 Done Done Sent false
16719 4826 Acked Write 5 Done Done Sent false
16720 4827 Acked Read 4 Done Done Sent false
16721 4828 Acked Write 2 Done Done Sent false
16722 4829 Acked Read 1 Done Done Sent false
16723 4830 Acked Read 3 Done Done Sent false
16724 4831 Acked Read 7 Done Done Sent false
16725 4832 Acked Read 7 Done Done Sent false
16726 4833 Acked Write 2 Done Done Sent false
16727 4834 Acked Write 6 Done Done Sent false
16728 4835 Acked Write 1 Done Done Sent false
16729 4836 Acked Read 2 Done Done Sent false
16730 4837 Acked Read 1 Done Done Sent false
16731 4838 Acked Read 3 Done Done Sent false
16732 4839 Acked Write 4 Done Done Sent false
16733 4840 Acked Write 2 Done Done Sent false
16734 4841 Acked Flush 0 Done Done Sent false
16735 4842 Acked Read 1 Done Done Sent false
16736 4843 Acked Read 7 Done Done Sent false
16737 4844 Acked Read 8 Done Done Sent false
16738 4845 Acked Read 6 Done Done Sent false
16739 4846 Acked Read 5 Done Done Sent false
16740 4847 Acked Write 6 Done Done Sent false
16741 4848 Acked Read 8 Done Done Sent false
16742 4849 Acked Write 8 Done Done Sent false
16743 4850 Acked Read 3 Done Done Sent false
16744 4851 Acked Read 8 Done Done Sent false
16745 4852 Acked Write 9 Done Done Sent false
16746 4853 Acked Read 2 Done Done Sent false
16747 4854 Acked Write 4 Done Done Sent false
16748 4855 Acked Write 8 Done Done Sent false
16749 4856 Acked Read 2 Done Done Sent false
16750 4857 Acked Write 8 Done Done Sent false
16751 4858 Acked Read 8 Done Done Sent false
16752 4859 Acked Read 8 Done Done Sent false
16753 4860 Acked Write 2 Done Done Sent false
16754 4861 Acked Flush 0 Done Done Sent false
16755 4862 Acked Write 2 Done Done Sent false
16756 4863 Acked Read 2 Done Done Sent false
16757 4864 Acked Write 9 Done Done Sent false
16758 4865 Acked Write 10 Done Done Sent false
16759 4866 Acked Read 9 Done Done Sent false
16760 4867 Acked Write 8 Done Done Sent false
16761 4868 Acked Read 3 Done Done Sent false
16762 4869 Acked Write 10 Done Done Sent false
16763 4870 Acked Read 6 Done Done Sent false
16764 4871 Acked Write 8 Done Done Sent false
16765 4872 Acked Read 6 Done Done Sent false
16766 4873 Acked Read 8 Done Done Sent false
16767 4874 Acked Write 4 Done Done Sent false
16768 4875 Acked Write 2 Done Done Sent false
16769 4876 Acked Read 9 Done Done Sent false
16770 4877 Acked Read 6 Done Done Sent false
16771 4878 Acked Write 6 Done Done Sent false
16772 4879 Acked Read 3 Done Done Sent false
16773 4880 Acked Write 7 Done Done Sent false
16774 4881 Acked Read 4 Done Done Sent false
16775 4882 Acked Read 4 Done Done Sent false
16776 4883 Acked Read 5 Done Done Sent false
16777 4884 Acked Read 1 Done Done Sent false
16778 4885 Acked Flush 0 Done Done Sent false
16779 4886 Acked Read 10 Done Done Sent false
16780 4887 Acked Read 3 Done Done Sent false
16781 4888 Acked Flush 0 Done Done Sent false
16782 4889 Acked Write 2 Done Done Sent false
16783 4890 Acked Read 1 Done Done Sent false
16784 4891 Acked Read 4 Done Done Sent false
16785 4892 Acked Flush 0 Done Done Sent false
16786 4893 Acked Read 7 Done Done Sent false
16787 4894 Acked Write 6 Done Done Sent false
16788 4895 Acked Write 1 Done Done Sent false
16789 4896 Acked Read 8 Done Done Sent false
16790 4897 Acked Read 9 Done Done Sent false
16791 4898 Acked Read 8 Done Done Sent false
16792 4899 Acked Read 1 Done Done Sent false
16793 4900 Acked Read 5 Done Done Sent false
16794 4901 Acked Flush 0 Done Done Sent false
16795 4902 Acked Read 4 Done Done Sent false
16796 4903 Acked Read 7 Done Done Sent false
16797 4904 Acked Flush 0 Done Done Sent false
16798 4905 Acked Read 5 Done Done Sent false
16799 4906 Acked Read 3 Done Done Sent false
16800 4907 Acked Write 9 Done Done Sent false
16801 4908 Acked Flush 0 Done Done Sent false
16802 4909 Acked Write 8 Done Done Sent false
16803 4910 Acked Read 3 Done Done Sent false
16804 4911 Acked Write 5 Done Done Sent false
16805 4912 Acked Read 9 Done Done Sent false
16806 4913 Acked Read 9 Done Done Sent false
16807 4914 Acked Read 8 Done Done Sent false
16808 4915 Acked Write 1 Done Done Sent false
16809 4916 Acked Write 1 Done Done Sent false
16810 4917 Acked Write 1 Done Done Sent false
16811 4918 Acked Read 6 Done Done Sent false
16812 4919 Acked Read 10 Done Done Sent false
16813 4920 Acked Write 1 Done Done Sent false
16814 4921 Acked Write 9 Done Done Sent false
16815 4922 Acked Write 5 Done Done Sent false
16816 4923 Acked Write 8 Done Done Sent false
16817 STATES DS:0 DS:1 DS:2 TOTAL
16818 Sent 0 0 301 301
16819 Done 301 301 0 602
16820 Skipped 0 0 0 0
16821 Error 0 0 0 0
16822 Last Flush: Some(JobId(4908)) Some(JobId(4908)) Some(JobId(4622))
16823 Downstairs last five completed: 4622 4621 4620 4619 4618
16824 Upstairs last five completed: 4923 4922 4921 4920 4919
16825 CLIENT: Up:0 ds:301 act:2
168262025-03-17T22:22:41.922ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
168272025-03-17T22:22:42.923ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
168282025-03-17T22:22:43.141ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
168292025-03-17T22:22:43.141ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:44265) = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
168302025-03-17T22:22:43.143ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
168312025-03-17T22:22:43.143ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
168322025-03-17T22:22:43.143ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
168332025-03-17T22:22:43.143ZINFOcrucible: send last flush ID to this DS: Some(JobId(4622)) = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
168342025-03-17T22:22:43.143ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
168352025-03-17T22:22:43.143ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(4622)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
168362025-03-17T22:22:43.144ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(4622)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
16837 ----------------------------------------------------------------
16838 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
16839 Downstairs last five completed: 4924 4923 4922 4921 4920
16840 Upstairs last five completed: 4924 4923 4922 4921 4920
16841 CLIENT: Up:0 ds:0 act:3
16842 Replay: All jobs finished
16843 Main task has work to do, go find it
16844 got dsc Stop(1)
16845 stop 1
16846 Mar 17 22:22:45.398 INFO request completed, latency_us: 349, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 1c81e8ce-ddf3-41dc-b986-6a4a8e734398, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
16847 [8820] Got stop action so:false kr:false
16848 Mar 17 22:22:45.399 INFO request completed, latency_us: 131, response_code: 200, uri: /state/cid/1, method: GET, req_id: 5679c7eb-160f-4e1f-979c-cd86004bac0c, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
168492025-03-17T22:22:45.399ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
168502025-03-17T22:22:45.399ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
168512025-03-17T22:22:45.399ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
168522025-03-17T22:22:45.400ZINFOcrucible: client task is exiting = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
16853 [8820] Exited with: ExitStatus(unix_wait_status(9))
16854 [8820][1] reports Exit
16855 Mar 17 22:22:49.401 INFO request completed, latency_us: 215, response_code: 200, uri: /state/cid/1, method: GET, req_id: 56d8893d-b6ea-4602-82b4-e2a61e783ef0, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
16856 001/300 Write block 2519 len 28672 data: 5 4 5 4 5 4 3
16857 002/300 Read block 4258 len 36864
16858 003/300 Read block 3658 len 4096
16859 004/300 Read block 4966 len 4096
16860 005/300 Write block 4210 len 36864 data: 6 5 5 5 5 4 5 5 6
16861 006/300 Write block 4556 len 8192 data: 6 6
16862 007/300 Write block 3325 len 20480 data: 4 5 6 4 3
16863 008/300 Write block 4201 len 40960 data: 3 3 2 3 3 3 3 5 5 7
16864 009/300 Read block 1888 len 20480
16865 010/300 Write block 68 len 4096 data: 3
16866 011/300 Read block 871 len 40960
16867 012/300 Flush
16868 013/300 Read block 3340 len 36864
16869 014/300 Write block 1225 len 12288 data: 2 2 2
16870 015/300 Read block 3711 len 8192
16871 016/300 Write block 4775 len 36864 data: 3 3 3 3 3 3 3 4 4
16872 017/300 Read block 928 len 4096
16873 018/300 Write block 2197 len 8192 data: 4 3
16874 019/300 Flush
16875 020/300 Write block 3575 len 24576 data: 4 4 4 6 6 5
16876 021/300 Write block 1041 len 32768 data: 3 3 3 3 4 6 5 6
16877 022/300 Read block 4004 len 28672
16878 023/300 Write block 1609 len 36864 data: 4 4 2 2 2 3 3 3 3
16879 024/300 Write block 2840 len 36864 data: 2 2 3 3 3 3 3 3 2
16880 025/300 Write block 168 len 16384 data: 3 3 2 2
16881 026/300 Write block 2783 len 32768 data: 3 3 3 3 2 3 3 3
16882 027/300 Read block 3124 len 12288
16883 028/300 Read block 526 len 20480
16884 029/300 Write block 1110 len 40960 data: 6 5 4 4 3 3 3 3 3 3
16885 030/300 Read block 1229 len 36864
16886 031/300 Read block 2940 len 24576
16887 032/300 Write block 4167 len 12288 data: 8 7 8
16888 033/300 Write block 726 len 36864 data: 3 3 4 4 4 2 2 2 2
16889 034/300 Write block 4252 len 36864 data: 3 2 4 4 5 6 5 6 5
16890 035/300 Read block 2899 len 24576
16891 036/300 Write block 3016 len 24576 data: 4 6 5 5 5 6
16892 037/300 Write block 4309 len 28672 data: 2 3 4 5 5 5 5
16893 038/300 Read block 3128 len 8192
16894 039/300 Write block 2522 len 12288 data: 5 6 5
16895 040/300 Write block 4355 len 32768 data: 3 3 3 3 2 3 3 3
16896 041/300 Write block 814 len 24576 data: 4 3 2 2 2 3
16897 042/300 Read block 3324 len 32768
16898 043/300 Read block 402 len 8192
16899 044/300 Read block 2840 len 4096
16900 045/300 Flush
16901 046/300 Read block 2182 len 4096
16902 047/300 Write block 1471 len 40960 data: 5 6 6 6 6 5 4 5 5 6
16903 048/300 Write block 3120 len 32768 data: 5 4 3 3 3 4 5 7
16904 049/300 Write block 3682 len 40960 data: 4 5 5 5 5 5 3 3 3 3
16905 050/300 Read block 2769 len 4096
16906 051/300 Read block 3608 len 36864
16907 052/300 Write block 2311 len 20480 data: 5 5 7 8 7
16908 053/300 Write block 85 len 20480 data: 8 8 6 6 4
16909 054/300 Write block 923 len 16384 data: 5 5 3 2
16910 055/300 Write block 2418 len 28672 data: 5 6 8 5 5 4 4
16911 056/300 Write block 2990 len 24576 data: 2 2 3 3 3 2
16912 057/300 Read block 177 len 16384
16913 058/300 Write block 1587 len 12288 data: 6 6 7
16914 059/300 Write block 4903 len 36864 data: 6 6 5 5 5 3 3 4 5
16915 060/300 Read block 1197 len 24576
16916 061/300 Read block 1497 len 4096
16917 062/300 Read block 3393 len 12288
16918 063/300 Read block 4131 len 28672
16919 064/300 Read block 1192 len 12288
16920 065/300 Flush
16921 066/300 Flush
16922 067/300 Flush
16923 068/300 Write block 3729 len 36864 data: 3 4 4 4 4 3 2 2 2
16924 069/300 Read block 3555 len 16384
16925 070/300 Read block 2178 len 20480
16926 071/300 Read block 2002 len 4096
16927 072/300 Read block 863 len 24576
16928 073/300 Write block 377 len 8192 data: 4 4
16929 074/300 Read block 3480 len 12288
16930 075/300 Read block 2637 len 36864
16931 076/300 Read block 3415 len 20480
16932 077/300 Read block 4586 len 20480
16933 078/300 Read block 533 len 8192
16934 079/300 Read block 2605 len 40960
16935 080/300 Write block 4561 len 4096 data: 4
16936 081/300 Read block 2853 len 16384
16937 082/300 Read block 2237 len 24576
16938 083/300 Write block 309 len 12288 data: 2 2 3
16939 084/300 Read block 1548 len 24576
16940 085/300 Read block 4755 len 36864
16941 086/300 Write block 4544 len 32768 data: 4 3 3 4 5 5 6 7
16942 087/300 Read block 3835 len 36864
16943 088/300 Read block 3350 len 40960
16944 089/300 Write block 2481 len 8192 data: 2 2
16945 090/300 Read block 3373 len 24576
16946 091/300 Write block 918 len 40960 data: 3 3 3 3 4 6 6 4 3 3
16947 092/300 Read block 4535 len 36864
16948 093/300 Write block 590 len 40960 data: 3 4 5 5 5 5 4 3 3 2
16949 094/300 Write block 2836 len 20480 data: 3 3 3 2 3
16950 095/300 Write block 2606 len 36864 data: 3 4 4 3 4 4 3 3 3
16951 096/300 Write block 2168 len 24576 data: 3 2 2 2 2 2
16952 097/300 Flush
16953 098/300 Read block 3368 len 24576
16954 099/300 Write block 4494 len 16384 data: 4 4 4 4
16955 100/300 Read block 2437 len 16384
16956 101/300 Read block 4523 len 28672
16957 102/300 Flush
16958 103/300 Read block 4160 len 32768
16959 104/300 Flush
16960 105/300 Write block 3492 len 4096 data: 4
16961 106/300 Flush
16962 107/300 Write block 817 len 16384 data: 3 3 4 3
16963 108/300 Write block 2973 len 28672 data: 3 2 2 2 2 2 4
16964 109/300 Write block 36 len 40960 data: 4 5 5 3 3 3 3 3 4 6
16965 110/300 Flush
16966 111/300 Flush
16967 112/300 Read block 4107 len 16384
16968 113/300 Write block 243 len 16384 data: 2 2 3 3
16969 114/300 Write block 2695 len 24576 data: 2 3 3 3 3 3
16970 115/300 Flush
16971 116/300 Read block 4538 len 24576
16972 117/300 Flush
16973 118/300 Read block 44 len 36864
16974 119/300 Read block 4505 len 20480
16975 120/300 Write block 2742 len 4096 data: 5
16976 121/300 Read block 2470 len 36864
16977 122/300 Write block 1819 len 28672 data: 2 2 2 3 3 3 3
16978 123/300 Read block 1181 len 36864
16979 124/300 Write block 4853 len 24576 data: 4 5 5 4 5 6
16980 125/300 Write block 3525 len 20480 data: 6 4 4 5 7
16981 126/300 Write block 2430 len 12288 data: 4 4 4
16982 127/300 Write block 2240 len 16384 data: 5 4 4 4
16983 128/300 Read block 4009 len 20480
16984 129/300 Flush
16985 130/300 Read block 233 len 16384
16986 131/300 Write block 2476 len 12288 data: 4 4 3
16987 132/300 Read block 1595 len 16384
16988 133/300 Read block 3077 len 16384
16989 134/300 Write block 1826 len 12288 data: 3 3 3
16990 135/300 Read block 1620 len 4096
16991 136/300 Flush
16992 137/300 Write block 501 len 4096 data: 3
16993 138/300 Write block 3380 len 28672 data: 2 2 3 3 4 3 2
16994 139/300 Write block 4836 len 40960 data: 5 4 4 4 3 3 4 6 5 5
16995 140/300 Write block 1065 len 28672 data: 2 2 2 2 3 3 3
16996 141/300 Write block 1114 len 40960 data: 4 4 4 4 4 4 2 2 2 3
16997 142/300 Write block 1672 len 32768 data: 3 4 4 3 3 3 3 2
16998 143/300 Write block 740 len 28672 data: 8 8 8 8 8 6 5
16999 144/300 Read block 199 len 16384
17000 145/300 Read block 2480 len 24576
17001 146/300 Write block 4714 len 16384 data: 6 6 6 6
17002 147/300 Write block 2605 len 20480 data: 5 4 5 5 4
17003 148/300 Write block 1837 len 12288 data: 5 4 6
17004 149/300 Read block 397 len 36864
17005 150/300 Write block 3041 len 28672 data: 6 6 5 5 6 4 5
17006 151/300 Read block 3357 len 28672
17007 152/300 Read block 391 len 36864
17008 153/300 Write block 2228 len 16384 data: 5 5 5 5
17009 154/300 Read block 1402 len 40960
17010 155/300 Read block 891 len 20480
17011 156/300 Read block 1757 len 28672
17012 157/300 Write block 1266 len 16384 data: 3 3 2 3
17013 158/300 Write block 3426 len 32768 data: 3 3 4 4 4 4 3 3
17014 159/300 Write block 3945 len 12288 data: 4 4 4
17015 160/300 Read block 1976 len 40960
17016 161/300 Write block 4230 len 36864 data: 6 7 5 4 4 3 3 2 4
17017 162/300 Read block 2049 len 24576
17018 163/300 Write block 846 len 28672 data: 4 4 3 4 4 3 3
17019 164/300 Read block 3445 len 32768
17020 165/300 Read block 4049 len 36864
17021 166/300 Read block 3076 len 24576
17022 167/300 Write block 4033 len 12288 data: 2 2 3
17023 168/300 Write block 4159 len 28672 data: 3 4 5 5 6 6 6
17024 169/300 Flush
17025 170/300 Read block 2146 len 36864
17026 171/300 Write block 3874 len 12288 data: 4 4 4
17027 172/300 Read block 2292 len 24576
17028 173/300 Read block 1651 len 40960
17029 174/300 Write block 1298 len 20480 data: 3 3 3 2 2
17030 175/300 Write block 1458 len 8192 data: 4 3
17031 176/300 Read block 3741 len 12288
17032 177/300 Read block 484 len 20480
17033 178/300 Write block 3097 len 20480 data: 4 4 4 4 3
17034 179/300 Read block 2141 len 16384
17035 180/300 Read block 61 len 28672
17036 181/300 Write block 592 len 24576 data: 6 6 6 6 5 4
17037 182/300 Write block 2602 len 28672 data: 6 5 5 6 5 6 6
17038 183/300 Read block 2077 len 36864
17039 184/300 Write block 974 len 4096 data: 4
17040 185/300 Write block 1621 len 8192 data: 2 2
17041 186/300 Read block 1264 len 4096
17042 187/300 Read block 3341 len 4096
17043 188/300 Read block 1321 len 16384
17044 189/300 Write block 2894 len 20480 data: 2 2 3 3 5
17045 190/300 Write block 729 len 40960 data: 5 5 3 3 3 3 2 2 2 4
17046 191/300 Read block 3177 len 4096
17047 192/300 Read block 3570 len 40960
17048 193/300 Flush
17049 194/300 Write block 525 len 16384 data: 3 3 3 2
17050 195/300 Read block 1511 len 20480
17051 196/300 Read block 899 len 8192
17052 197/300 Write block 2164 len 32768 data: 2 3 3 3 4 3 3 3
17053 198/300 Read block 4814 len 4096
17054 199/300 Read block 1051 len 32768
17055 200/300 Read block 146 len 28672
17056 201/300 Read block 3425 len 16384
17057 202/300 Read block 455 len 40960
17058 203/300 Read block 4695 len 20480
17059 204/300 Write block 2962 len 40960 data: 4 4 4 4 3 3 3 4 4 4
17060 205/300 Read block 1208 len 8192
17061 206/300 Read block 1761 len 4096
17062 207/300 Read block 2169 len 36864
17063 208/300 Read block 3536 len 24576
17064 209/300 Read block 3716 len 24576
17065 210/300 Read block 4702 len 4096
17066 211/300 Write block 2822 len 28672 data: 5 5 4 4 4 4 4
17067 212/300 Write block 4376 len 4096 data: 5
17068 213/300 Write block 241 len 24576 data: 3 3 3 3 4 4
17069 214/300 Write block 2704 len 8192 data: 3 4
17070 215/300 Read block 2902 len 36864
17071 216/300 Write block 1439 len 8192 data: 3 2
17072 217/300 Write block 2394 len 32768 data: 6 4 4 5 5 5 5 5
17073 218/300 Read block 4922 len 28672
17074 219/300 Write block 3994 len 16384 data: 4 3 3 3
17075 220/300 Read block 4706 len 40960
17076 221/300 Read block 3147 len 12288
17077 222/300 Write block 852 len 36864 data: 4 3 3 3 4 5 4 4 4
17078 223/300 Write block 1721 len 4096 data: 4
17079 224/300 Write block 4556 len 40960 data: 7 7 5 5 4 5 5 5 4 2
17080 225/300 Write block 4631 len 12288 data: 4 3 3
17081 226/300 Write block 3773 len 16384 data: 5 6 6 7
17082 227/300 Read block 986 len 32768
17083 228/300 Read block 3114 len 12288
17084 229/300 Read block 4613 len 40960
17085 230/300 Read block 3411 len 36864
17086 231/300 Read block 4371 len 40960
17087 232/300 Write block 4128 len 16384 data: 2 2 2 3
17088 233/300 Read block 1421 len 12288
17089 234/300 Read block 4501 len 32768
17090 235/300 Write block 22 len 20480 data: 3 2 2 3 3
17091 236/300 Read block 1009 len 32768
17092 237/300 Write block 847 len 16384 data: 5 4 5 5
17093 238/300 Write block 3035 len 12288 data: 4 4 5
17094 239/300 Read block 1541 len 40960
17095 240/300 Read block 4132 len 20480
17096 241/300 Read block 4892 len 28672
17097 242/300 Read block 4501 len 16384
17098 243/300 Read block 2216 len 4096
17099 244/300 Read block 2338 len 16384
17100 245/300 Write block 81 len 4096 data: 3
17101 246/300 Read block 240 len 40960
17102 247/300 Write block 2603 len 32768 data: 6 6 7 6 7 7 5 5
17103 248/300 Read block 1667 len 36864
17104 249/300 Read block 3661 len 24576
17105 250/300 Write block 3659 len 24576 data: 3 2 3 4 4 4
17106 251/300 Read block 897 len 8192
17107 252/300 Write block 4032 len 16384 data: 3 3 3 4
17108 253/300 Write block 4996 len 16384 data: 3 3 3 4
17109 254/300 Read block 3557 len 8192
17110 255/300 Read block 2826 len 36864
17111 256/300 Write block 1084 len 8192 data: 3 4
17112 257/300 Read block 374 len 20480
17113 258/300 Write block 1584 len 4096 data: 5
17114 259/300 Write block 4176 len 12288 data: 5 5 6
17115 260/300 Write block 310 len 12288 data: 3 4 3
17116 261/300 Write block 4343 len 24576 data: 3 3 3 3 3 2
17117 262/300 Read block 445 len 12288
17118 263/300 Write block 2170 len 4096 data: 4
17119 264/300 Read block 733 len 36864
17120 265/300 Read block 4602 len 28672
17121 266/300 Write block 2946 len 36864 data: 4 5 5 5 5 5 4 4 3
17122 267/300 Write block 2657 len 28672 data: 3 4 4 2 2 2 2
17123 268/300 Read block 4319 len 8192
17124 269/300 Flush
17125 270/300 Read block 4957 len 4096
17126 271/300 Read block 1538 len 16384
17127 272/300 Read block 1957 len 40960
17128 273/300 Write block 1067 len 12288 data: 3 3 4
17129 274/300 Write block 2011 len 24576 data: 4 5 5 5 5 5
17130 275/300 Read block 4508 len 8192
17131 276/300 Write block 529 len 8192 data: 2 3
17132 277/300 Read block 221 len 40960
17133 278/300 Read block 20 len 32768
17134 279/300 Read block 4121 len 8192
17135 280/300 Read block 1765 len 28672
17136 281/300 Read block 4093 len 32768
17137 282/300 Read block 3957 len 40960
17138 283/300 Write block 1561 len 12288 data: 5 5 6
17139 284/300 Read block 2874 len 16384
17140 285/300 Read block 598 len 28672
17141 286/300 Read block 4913 len 12288
17142 287/300 Read block 3561 len 32768
17143 288/300 Read block 3721 len 4096
17144 289/300 Write block 2878 len 24576 data: 4 4 4 5 4 4
17145 290/300 Read block 3092 len 40960
17146 291/300 Read block 4058 len 28672
17147 292/300 Read block 1725 len 12288
17148 293/300 Read block 4221 len 28672
17149 294/300 Read block 685 len 28672
17150 295/300 Write block 2668 len 40960 data: 5 4 2 2 2 2 2 4 3 3
17151 296/300 Read block 1615 len 16384
17152 297/300 Write block 4735 len 12288 data: 4 4 5
17153 298/300 Read block 1960 len 28672
17154 299/300 Write block 2552 len 36864 data: 4 5 5 5 3 4 4 4 3
17155 300/300 Read block 1594 len 32768
17156 301/300 Write block 2820 len 8192 data: 4 4
17157 Main task has work to do, go find it
17158 got dsc Start(1)
17159 start 1
17160 Mar 17 22:22:49.647 INFO request completed, latency_us: 218, response_code: 204, uri: /start/cid/1, method: POST, req_id: 678dba37-9623-4b03-ab1d-0a6e16954869, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
17161 Replay: started 1, returned:Ok(())
17162 ----------------------------------------------------------------
17163 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17164 States: Active Offline Active
17165 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17166 4925 Acked Write 7 Done Sent Done false
17167 4926 Acked Read 9 Done Sent Done false
17168 4927 Acked Read 1 Done Sent Done false
17169 4928 Acked Read 1 Done Sent Done false
17170 4929 Acked Write 9 Done Sent Done false
17171 4930 Acked Write 2 Done Sent Done false
17172 4931 Acked Write 5 Done Sent Done false
17173 4932 Acked Write 10 Done Sent Done false
17174 4933 Acked Read 5 Done Sent Done false
17175 4934 Acked Write 1 Done Sent Done false
17176 4935 Acked Read 10 Done Sent Done false
17177 4936 Acked Flush 0 Done Sent Done false
17178 4937 Acked Read 9 Done Sent Done false
17179 4938 Acked Write 3 Done Sent Done false
17180 4939 Acked Read 2 Done Sent Done false
17181 4940 Acked Write 9 Done Sent Done false
17182 4941 Acked Read 1 Done Sent Done false
17183 4942 Acked Write 2 Done Sent Done false
17184 4943 Acked Flush 0 Done Sent Done false
17185 4944 Acked Write 6 Done Sent Done false
17186 4945 Acked Write 8 Done Sent Done false
17187 4946 Acked Read 7 Done Sent Done false
17188 4947 Acked Write 9 Done Sent Done false
17189 4948 Acked Write 9 Done Sent Done false
17190 4949 Acked Write 4 Done Sent Done false
17191 4950 Acked Write 8 Done Sent Done false
17192 4951 Acked Read 3 Done Sent Done false
17193 4952 Acked Read 5 Done Sent Done false
17194 4953 Acked Write 10 Done Sent Done false
17195 4954 Acked Read 9 Done Sent Done false
17196 4955 Acked Read 6 Done Sent Done false
17197 4956 Acked Write 3 Done Sent Done false
17198 4957 Acked Write 9 Done Sent Done false
17199 4958 Acked Write 9 Done Sent Done false
17200 4959 Acked Read 6 Done Sent Done false
17201 4960 Acked Write 6 Done Sent Done false
17202 4961 Acked Write 7 Done Sent Done false
17203 4962 Acked Read 2 Done Sent Done false
17204 4963 Acked Write 3 Done Sent Done false
17205 4964 Acked Write 8 Done Sent Done false
17206 4965 Acked Write 6 Done Sent Done false
17207 4966 Acked Read 8 Done Sent Done false
17208 4967 Acked Read 2 Done Sent Done false
17209 4968 Acked Read 1 Done Sent Done false
17210 4969 Acked Flush 0 Done Sent Done false
17211 4970 Acked Read 1 Done Sent Done false
17212 4971 Acked Write 10 Done Sent Done false
17213 4972 Acked Write 8 Done Sent Done false
17214 4973 Acked Write 10 Done Sent Done false
17215 4974 Acked Read 1 Done Sent Done false
17216 4975 Acked Read 9 Done Sent Done false
17217 4976 Acked Write 5 Done Sent Done false
17218 4977 Acked Write 5 Done Sent Done false
17219 4978 Acked Write 4 Done Sent Done false
17220 4979 Acked Write 7 Done Sent Done false
17221 4980 Acked Write 6 Done Sent Done false
17222 4981 Acked Read 4 Done Sent Done false
17223 4982 Acked Write 3 Done Sent Done false
17224 4983 Acked Write 9 Done Sent Done false
17225 4984 Acked Read 6 Done Sent Done false
17226 4985 Acked Read 1 Done Sent Done false
17227 4986 Acked Read 3 Done Sent Done false
17228 4987 Acked Read 7 Done Sent Done false
17229 4988 Acked Read 3 Done Sent Done false
17230 4989 Acked Flush 0 Done Sent Done false
17231 4990 Acked Flush 0 Done Sent Done false
17232 4991 Acked Flush 0 Done Sent Done false
17233 4992 Acked Write 9 Done Sent Done false
17234 4993 Acked Read 4 Done Sent Done false
17235 4994 Acked Read 5 Done Sent Done false
17236 4995 Acked Read 1 Done Sent Done false
17237 4996 Acked Read 6 Done Sent Done false
17238 4997 Acked Write 2 Done Sent Done false
17239 4998 Acked Read 3 Done Sent Done false
17240 4999 Acked Read 9 Done Sent Done false
17241 5000 Acked Read 5 Done Sent Done false
17242 5001 Acked Read 5 Done Sent Done false
17243 5002 Acked Read 2 Done Sent Done false
17244 5003 Acked Read 10 Done Sent Done false
17245 5004 Acked Write 1 Done Sent Done false
17246 5005 Acked Read 4 Done Sent Done false
17247 5006 Acked Read 6 Done Sent Done false
17248 5007 Acked Write 3 Done Sent Done false
17249 5008 Acked Read 6 Done Sent Done false
17250 5009 Acked Read 9 Done Sent Done false
17251 5010 Acked Write 8 Done Sent Done false
17252 5011 Acked Read 9 Done Sent Done false
17253 5012 Acked Read 10 Done Sent Done false
17254 5013 Acked Write 2 Done Sent Done false
17255 5014 Acked Read 6 Done Sent Done false
17256 5015 Acked Write 10 Done Sent Done false
17257 5016 Acked Read 9 Done Sent Done false
17258 5017 Acked Write 10 Done Sent Done false
17259 5018 Acked Write 5 Done Sent Done false
17260 5019 Acked Write 9 Done Sent Done false
17261 5020 Acked Write 6 Done Sent Done false
17262 5021 Acked Flush 0 Done Sent Done false
17263 5022 Acked Read 6 Done Sent Done false
17264 5023 Acked Write 4 Done Sent Done false
17265 5024 Acked Read 4 Done Sent Done false
17266 5025 Acked Read 7 Done Sent Done false
17267 5026 Acked Flush 0 Done Sent Done false
17268 5027 Acked Read 8 Done Sent Done false
17269 5028 Acked Flush 0 Done Sent Done false
17270 5029 Acked Write 1 Done Sent Done false
17271 5030 Acked Flush 0 Done Sent Done false
17272 5031 Acked Write 4 Done Sent Done false
17273 5032 Acked Write 7 Done Sent Done false
17274 5033 Acked Write 10 Done Sent Done false
17275 5034 Acked Flush 0 Done Sent Done false
17276 5035 Acked Flush 0 Done Sent Done false
17277 5036 Acked Read 4 Done Sent Done false
17278 5037 Acked Write 4 Done Sent Done false
17279 5038 Acked Write 6 Done Sent Done false
17280 5039 Acked Flush 0 Done Sent Done false
17281 5040 Acked Read 6 Done Sent Done false
17282 5041 Acked Flush 0 Done Sent Done false
17283 5042 Acked Read 9 Done Sent Done false
17284 5043 Acked Read 5 Done Sent Done false
17285 5044 Acked Write 1 Done Sent Done false
17286 5045 Acked Read 9 Done Sent Done false
17287 5046 Acked Write 7 Done Sent Done false
17288 5047 Acked Read 9 Done Sent Done false
17289 5048 Acked Write 6 Done Sent Done false
17290 5049 Acked Write 5 Done Sent Done false
17291 5050 Acked Write 3 Done Sent Done false
17292 5051 Acked Write 4 Done Sent Done false
17293 5052 Acked Read 5 Done Sent Done false
17294 5053 Acked Flush 0 Done Sent Done false
17295 5054 Acked Read 4 Done Sent Done false
17296 5055 Acked Write 3 Done Sent Done false
17297 5056 Acked Read 4 Done Sent Done false
17298 5057 Acked Read 4 Done Sent Done false
17299 5058 Acked Write 3 Done Sent Done false
17300 5059 Acked Read 1 Done Sent Done false
17301 5060 Acked Flush 0 Done Sent Done false
17302 5061 Acked Write 1 Done Sent Done false
17303 5062 Acked Write 7 Done Sent Done false
17304 5063 Acked Write 10 Done Sent Done false
17305 5064 Acked Write 7 Done Sent Done false
17306 5065 Acked Write 10 Done Sent Done false
17307 5066 Acked Write 8 Done Sent Done false
17308 5067 Acked Write 7 Done Sent Done false
17309 5068 Acked Read 4 Done Sent Done false
17310 5069 Acked Read 6 Done Sent Done false
17311 5070 Acked Write 4 Done Sent Done false
17312 5071 Acked Write 5 Done Sent Done false
17313 5072 Acked Write 3 Done Sent Done false
17314 5073 Acked Read 9 Done Sent Done false
17315 5074 Acked Write 7 Done Sent Done false
17316 5075 Acked Read 7 Done Sent Done false
17317 5076 Acked Read 9 Done Sent Done false
17318 5077 Acked Write 4 Done Sent Done false
17319 5078 Acked Read 10 Done Sent Done false
17320 5079 Acked Read 5 Done Sent Done false
17321 5080 Acked Read 7 Done Sent Done false
17322 5081 Acked Write 4 Done Sent Done false
17323 5082 Acked Write 8 Done Sent Done false
17324 5083 Acked Write 3 Done Sent Done false
17325 5084 Acked Read 10 Done Sent Done false
17326 5085 Acked Write 9 Done Sent Done false
17327 5086 Acked Read 6 Done Sent Done false
17328 5087 Acked Write 7 Done Sent Done false
17329 5088 Acked Read 8 Done Sent Done false
17330 5089 Acked Read 9 Done Sent Done false
17331 5090 Acked Read 6 Done Sent Done false
17332 5091 Acked Write 3 Done Sent Done false
17333 5092 Acked Write 7 Done Sent Done false
17334 5093 Acked Flush 0 Done Sent Done false
17335 5094 Acked Read 9 Done Sent Done false
17336 5095 Acked Write 3 Done Sent Done false
17337 5096 Acked Read 6 Done Sent Done false
17338 5097 Acked Read 10 Done Sent Done false
17339 5098 Acked Write 5 Done Sent Done false
17340 5099 Acked Write 2 Done Sent Done false
17341 5100 Acked Read 3 Done Sent Done false
17342 5101 Acked Read 5 Done Sent Done false
17343 5102 Acked Write 5 Done Sent Done false
17344 5103 Acked Read 4 Done Sent Done false
17345 5104 Acked Read 7 Done Sent Done false
17346 5105 Acked Write 6 Done Sent Done false
17347 5106 Acked Write 7 Done Sent Done false
17348 5107 Acked Read 9 Done Sent Done false
17349 5108 Acked Write 1 Done Sent Done false
17350 5109 Acked Write 2 Done Sent Done false
17351 5110 Acked Read 1 Done Sent Done false
17352 5111 Acked Read 1 Done Sent Done false
17353 5112 Acked Read 4 Done Sent Done false
17354 5113 Acked Write 5 Done Sent Done false
17355 5114 Acked Write 10 Done Sent Done false
17356 5115 Acked Read 1 Done Sent Done false
17357 5116 Acked Read 10 Done Sent Done false
17358 5117 Acked Flush 0 Done Sent Done false
17359 5118 Acked Write 4 Done Sent Done false
17360 5119 Acked Read 5 Done Sent Done false
17361 5120 Acked Read 2 Done Sent Done false
17362 5121 Acked Write 8 Done Sent Done false
17363 5122 Acked Read 1 Done Sent Done false
17364 5123 Acked Read 8 Done Sent Done false
17365 5124 Acked Read 7 Done Sent Done false
17366 5125 Acked Read 4 Done Sent Done false
17367 5126 Acked Read 10 Done Sent Done false
17368 5127 Acked Read 5 Done Sent Done false
17369 5128 Acked Write 10 Done Sent Done false
17370 5129 Acked Read 2 Done Sent Done false
17371 5130 Acked Read 1 Done Sent Done false
17372 5131 Acked Read 9 Done Sent Done false
17373 5132 Acked Read 6 Done Sent Done false
17374 5133 Acked Read 6 Done Sent Done false
17375 5134 Acked Read 1 Done Sent Done false
17376 5135 Acked Write 7 Done Sent Done false
17377 5136 Acked Write 1 Done Sent Done false
17378 5137 Acked Write 6 Done Sent Done false
17379 5138 Acked Write 2 Done Sent Done false
17380 5139 Acked Read 9 Done Sent Done false
17381 5140 Acked Write 2 Done Sent Done false
17382 5141 Acked Write 8 Done Sent Done false
17383 5142 Acked Read 7 Done Sent Done false
17384 5143 Acked Write 4 Done Sent Done false
17385 5144 Acked Read 10 Done Sent Done false
17386 5145 Acked Read 3 Done Sent Done false
17387 5146 Acked Write 9 Done Sent Done false
17388 5147 Acked Write 1 Done Sent Done false
17389 5148 Acked Write 10 Done Sent Done false
17390 5149 Acked Write 3 Done Sent Done false
17391 5150 Acked Write 4 Done Sent Done false
17392 5151 Acked Read 8 Done Sent Done false
17393 5152 Acked Read 3 Done Sent Done false
17394 5153 Acked Read 10 Done Sent Done false
17395 5154 Acked Read 9 Done Sent Done false
17396 5155 Acked Read 10 Done Sent Done false
17397 5156 Acked Write 4 Done Sent Done false
17398 5157 Acked Read 3 Done Sent Done false
17399 5158 Acked Read 8 Done Sent Done false
17400 5159 Acked Write 5 Done Sent Done false
17401 5160 Acked Read 8 Done Sent Done false
17402 5161 Acked Write 4 Done Sent Done false
17403 5162 Acked Write 3 Done Sent Done false
17404 5163 Acked Read 10 Done Sent Done false
17405 5164 Acked Read 5 Done Sent Done false
17406 5165 Acked Read 7 Done Sent Done false
17407 5166 Acked Read 4 Done Sent Done false
17408 5167 Acked Read 1 Done Sent Done false
17409 5168 Acked Read 4 Done Sent Done false
17410 5169 Acked Write 1 Done Sent Done false
17411 5170 Acked Read 10 Done Sent Done false
17412 5171 Acked Write 8 Done Sent Done false
17413 5172 Acked Read 9 Done Sent Done false
17414 5173 Acked Read 6 Done Sent Done false
17415 5174 Acked Write 6 Done Sent Done false
17416 5175 Acked Read 2 Done Sent Done false
17417 5176 Acked Write 4 Done Sent Done false
17418 5177 Acked Write 4 Done Sent Done false
17419 5178 Acked Read 2 Done Sent Done false
17420 5179 Acked Read 9 Done Sent Done false
17421 5180 Acked Write 2 Done Sent Done false
17422 5181 Acked Read 5 Done Sent Done false
17423 5182 Acked Write 1 Done Sent Done false
17424 5183 Acked Write 3 Done Sent Done false
17425 5184 Acked Write 3 Done Sent Done false
17426 5185 Acked Write 6 Done Sent Done false
17427 5186 Acked Read 3 Done Sent Done false
17428 5187 Acked Write 1 Done Sent Done false
17429 5188 Acked Read 9 Done Sent Done false
17430 5189 Acked Read 7 Done Sent Done false
17431 5190 Acked Write 9 Done Sent Done false
17432 5191 Acked Write 7 Done Sent Done false
17433 5192 Acked Read 2 Done Sent Done false
17434 5193 Acked Flush 0 Done Sent Done false
17435 5194 Acked Read 1 Done Sent Done false
17436 5195 Acked Read 4 Done Sent Done false
17437 5196 Acked Read 10 Done Sent Done false
17438 5197 Acked Write 3 Done Sent Done false
17439 5198 Acked Write 6 Done Sent Done false
17440 5199 Acked Read 2 Done Sent Done false
17441 5200 Acked Write 2 Done Sent Done false
17442 5201 Acked Read 10 Done Sent Done false
17443 5202 Acked Read 8 Done Sent Done false
17444 5203 Acked Read 2 Done Sent Done false
17445 5204 Acked Read 7 Done Sent Done false
17446 5205 Acked Read 8 Done Sent Done false
17447 5206 Acked Read 10 Done Sent Done false
17448 5207 Acked Write 3 Done Sent Done false
17449 5208 Acked Read 4 Done Sent Done false
17450 5209 Acked Read 7 Done Sent Done false
17451 5210 Acked Read 3 Done Sent Done false
17452 5211 Acked Read 8 Done Sent Done false
17453 5212 Acked Read 1 Done Sent Done false
17454 5213 Acked Write 6 Done Sent Done false
17455 5214 Acked Read 10 Done Sent Done false
17456 5215 Acked Read 7 Done Sent Done false
17457 5216 Acked Read 3 Done Sent Done false
17458 5217 Acked Read 7 Done Sent Done false
17459 5218 Acked Read 7 Done Sent Done false
17460 5219 Acked Write 10 Done Sent Done false
17461 5220 Acked Read 4 Done Sent Done false
17462 5221 Acked Write 3 Done Sent Done false
17463 5222 Acked Read 7 Done Sent Done false
17464 5223 Acked Write 9 Done Sent Done false
17465 5224 Acked Read 8 Done Sent Done false
17466 5225 Acked Write 2 Done Sent Done false
17467 STATES DS:0 DS:1 DS:2 TOTAL
17468 Sent 0 301 0 301
17469 Done 301 0 301 602
17470 Skipped 0 0 0 0
17471 Error 0 0 0 0
17472 Last Flush: Some(JobId(5193)) Some(JobId(4924)) Some(JobId(5193))
17473 Downstairs last five completed: 4924 4923 4922 4921 4920
17474 Upstairs last five completed: 5225 5224 5223 5222 5221
17475 CLIENT: Up:0 ds:301 act:2
174762025-03-17T22:22:50.149ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
174772025-03-17T22:22:51.386ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
174782025-03-17T22:22:51.938ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
174792025-03-17T22:22:52.940ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
17480 ----------------------------------------------------------------
17481 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17482 States: Active Offline Active
17483 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17484 4925 Acked Write 7 Done Sent Done false
17485 4926 Acked Read 9 Done Sent Done false
17486 4927 Acked Read 1 Done Sent Done false
17487 4928 Acked Read 1 Done Sent Done false
17488 4929 Acked Write 9 Done Sent Done false
17489 4930 Acked Write 2 Done Sent Done false
17490 4931 Acked Write 5 Done Sent Done false
17491 4932 Acked Write 10 Done Sent Done false
17492 4933 Acked Read 5 Done Sent Done false
17493 4934 Acked Write 1 Done Sent Done false
17494 4935 Acked Read 10 Done Sent Done false
17495 4936 Acked Flush 0 Done Sent Done false
17496 4937 Acked Read 9 Done Sent Done false
17497 4938 Acked Write 3 Done Sent Done false
17498 4939 Acked Read 2 Done Sent Done false
17499 4940 Acked Write 9 Done Sent Done false
17500 4941 Acked Read 1 Done Sent Done false
17501 4942 Acked Write 2 Done Sent Done false
17502 4943 Acked Flush 0 Done Sent Done false
17503 4944 Acked Write 6 Done Sent Done false
17504 4945 Acked Write 8 Done Sent Done false
17505 4946 Acked Read 7 Done Sent Done false
17506 4947 Acked Write 9 Done Sent Done false
17507 4948 Acked Write 9 Done Sent Done false
17508 4949 Acked Write 4 Done Sent Done false
17509 4950 Acked Write 8 Done Sent Done false
17510 4951 Acked Read 3 Done Sent Done false
17511 4952 Acked Read 5 Done Sent Done false
17512 4953 Acked Write 10 Done Sent Done false
17513 4954 Acked Read 9 Done Sent Done false
17514 4955 Acked Read 6 Done Sent Done false
17515 4956 Acked Write 3 Done Sent Done false
17516 4957 Acked Write 9 Done Sent Done false
17517 4958 Acked Write 9 Done Sent Done false
17518 4959 Acked Read 6 Done Sent Done false
17519 4960 Acked Write 6 Done Sent Done false
17520 4961 Acked Write 7 Done Sent Done false
17521 4962 Acked Read 2 Done Sent Done false
17522 4963 Acked Write 3 Done Sent Done false
17523 4964 Acked Write 8 Done Sent Done false
17524 4965 Acked Write 6 Done Sent Done false
17525 4966 Acked Read 8 Done Sent Done false
17526 4967 Acked Read 2 Done Sent Done false
17527 4968 Acked Read 1 Done Sent Done false
17528 4969 Acked Flush 0 Done Sent Done false
17529 4970 Acked Read 1 Done Sent Done false
17530 4971 Acked Write 10 Done Sent Done false
17531 4972 Acked Write 8 Done Sent Done false
17532 4973 Acked Write 10 Done Sent Done false
17533 4974 Acked Read 1 Done Sent Done false
17534 4975 Acked Read 9 Done Sent Done false
17535 4976 Acked Write 5 Done Sent Done false
17536 4977 Acked Write 5 Done Sent Done false
17537 4978 Acked Write 4 Done Sent Done false
17538 4979 Acked Write 7 Done Sent Done false
17539 4980 Acked Write 6 Done Sent Done false
17540 4981 Acked Read 4 Done Sent Done false
17541 4982 Acked Write 3 Done Sent Done false
17542 4983 Acked Write 9 Done Sent Done false
17543 4984 Acked Read 6 Done Sent Done false
17544 4985 Acked Read 1 Done Sent Done false
17545 4986 Acked Read 3 Done Sent Done false
17546 4987 Acked Read 7 Done Sent Done false
17547 4988 Acked Read 3 Done Sent Done false
17548 4989 Acked Flush 0 Done Sent Done false
17549 4990 Acked Flush 0 Done Sent Done false
17550 4991 Acked Flush 0 Done Sent Done false
17551 4992 Acked Write 9 Done Sent Done false
17552 4993 Acked Read 4 Done Sent Done false
17553 4994 Acked Read 5 Done Sent Done false
17554 4995 Acked Read 1 Done Sent Done false
17555 4996 Acked Read 6 Done Sent Done false
17556 4997 Acked Write 2 Done Sent Done false
17557 4998 Acked Read 3 Done Sent Done false
17558 4999 Acked Read 9 Done Sent Done false
17559 5000 Acked Read 5 Done Sent Done false
17560 5001 Acked Read 5 Done Sent Done false
17561 5002 Acked Read 2 Done Sent Done false
17562 5003 Acked Read 10 Done Sent Done false
17563 5004 Acked Write 1 Done Sent Done false
17564 5005 Acked Read 4 Done Sent Done false
17565 5006 Acked Read 6 Done Sent Done false
17566 5007 Acked Write 3 Done Sent Done false
17567 5008 Acked Read 6 Done Sent Done false
17568 5009 Acked Read 9 Done Sent Done false
17569 5010 Acked Write 8 Done Sent Done false
17570 5011 Acked Read 9 Done Sent Done false
17571 5012 Acked Read 10 Done Sent Done false
17572 5013 Acked Write 2 Done Sent Done false
17573 5014 Acked Read 6 Done Sent Done false
17574 5015 Acked Write 10 Done Sent Done false
17575 5016 Acked Read 9 Done Sent Done false
17576 5017 Acked Write 10 Done Sent Done false
17577 5018 Acked Write 5 Done Sent Done false
17578 5019 Acked Write 9 Done Sent Done false
17579 5020 Acked Write 6 Done Sent Done false
17580 5021 Acked Flush 0 Done Sent Done false
17581 5022 Acked Read 6 Done Sent Done false
17582 5023 Acked Write 4 Done Sent Done false
17583 5024 Acked Read 4 Done Sent Done false
17584 5025 Acked Read 7 Done Sent Done false
17585 5026 Acked Flush 0 Done Sent Done false
17586 5027 Acked Read 8 Done Sent Done false
17587 5028 Acked Flush 0 Done Sent Done false
17588 5029 Acked Write 1 Done Sent Done false
17589 5030 Acked Flush 0 Done Sent Done false
17590 5031 Acked Write 4 Done Sent Done false
17591 5032 Acked Write 7 Done Sent Done false
17592 5033 Acked Write 10 Done Sent Done false
17593 5034 Acked Flush 0 Done Sent Done false
17594 5035 Acked Flush 0 Done Sent Done false
17595 5036 Acked Read 4 Done Sent Done false
17596 5037 Acked Write 4 Done Sent Done false
17597 5038 Acked Write 6 Done Sent Done false
17598 5039 Acked Flush 0 Done Sent Done false
17599 5040 Acked Read 6 Done Sent Done false
17600 5041 Acked Flush 0 Done Sent Done false
17601 5042 Acked Read 9 Done Sent Done false
17602 5043 Acked Read 5 Done Sent Done false
17603 5044 Acked Write 1 Done Sent Done false
17604 5045 Acked Read 9 Done Sent Done false
17605 5046 Acked Write 7 Done Sent Done false
17606 5047 Acked Read 9 Done Sent Done false
17607 5048 Acked Write 6 Done Sent Done false
17608 5049 Acked Write 5 Done Sent Done false
17609 5050 Acked Write 3 Done Sent Done false
17610 5051 Acked Write 4 Done Sent Done false
17611 5052 Acked Read 5 Done Sent Done false
17612 5053 Acked Flush 0 Done Sent Done false
17613 5054 Acked Read 4 Done Sent Done false
17614 5055 Acked Write 3 Done Sent Done false
17615 5056 Acked Read 4 Done Sent Done false
17616 5057 Acked Read 4 Done Sent Done false
17617 5058 Acked Write 3 Done Sent Done false
17618 5059 Acked Read 1 Done Sent Done false
17619 5060 Acked Flush 0 Done Sent Done false
17620 5061 Acked Write 1 Done Sent Done false
17621 5062 Acked Write 7 Done Sent Done false
17622 5063 Acked Write 10 Done Sent Done false
17623 5064 Acked Write 7 Done Sent Done false
17624 5065 Acked Write 10 Done Sent Done false
17625 5066 Acked Write 8 Done Sent Done false
17626 5067 Acked Write 7 Done Sent Done false
17627 5068 Acked Read 4 Done Sent Done false
17628 5069 Acked Read 6 Done Sent Done false
17629 5070 Acked Write 4 Done Sent Done false
17630 5071 Acked Write 5 Done Sent Done false
17631 5072 Acked Write 3 Done Sent Done false
17632 5073 Acked Read 9 Done Sent Done false
17633 5074 Acked Write 7 Done Sent Done false
17634 5075 Acked Read 7 Done Sent Done false
17635 5076 Acked Read 9 Done Sent Done false
17636 5077 Acked Write 4 Done Sent Done false
17637 5078 Acked Read 10 Done Sent Done false
17638 5079 Acked Read 5 Done Sent Done false
17639 5080 Acked Read 7 Done Sent Done false
17640 5081 Acked Write 4 Done Sent Done false
17641 5082 Acked Write 8 Done Sent Done false
17642 5083 Acked Write 3 Done Sent Done false
17643 5084 Acked Read 10 Done Sent Done false
17644 5085 Acked Write 9 Done Sent Done false
17645 5086 Acked Read 6 Done Sent Done false
17646 5087 Acked Write 7 Done Sent Done false
17647 5088 Acked Read 8 Done Sent Done false
17648 5089 Acked Read 9 Done Sent Done false
17649 5090 Acked Read 6 Done Sent Done false
17650 5091 Acked Write 3 Done Sent Done false
17651 5092 Acked Write 7 Done Sent Done false
17652 5093 Acked Flush 0 Done Sent Done false
17653 5094 Acked Read 9 Done Sent Done false
17654 5095 Acked Write 3 Done Sent Done false
17655 5096 Acked Read 6 Done Sent Done false
17656 5097 Acked Read 10 Done Sent Done false
17657 5098 Acked Write 5 Done Sent Done false
17658 5099 Acked Write 2 Done Sent Done false
17659 5100 Acked Read 3 Done Sent Done false
17660 5101 Acked Read 5 Done Sent Done false
17661 5102 Acked Write 5 Done Sent Done false
17662 5103 Acked Read 4 Done Sent Done false
17663 5104 Acked Read 7 Done Sent Done false
17664 5105 Acked Write 6 Done Sent Done false
17665 5106 Acked Write 7 Done Sent Done false
17666 5107 Acked Read 9 Done Sent Done false
17667 5108 Acked Write 1 Done Sent Done false
17668 5109 Acked Write 2 Done Sent Done false
17669 5110 Acked Read 1 Done Sent Done false
17670 5111 Acked Read 1 Done Sent Done false
17671 5112 Acked Read 4 Done Sent Done false
17672 5113 Acked Write 5 Done Sent Done false
17673 5114 Acked Write 10 Done Sent Done false
17674 5115 Acked Read 1 Done Sent Done false
17675 5116 Acked Read 10 Done Sent Done false
17676 5117 Acked Flush 0 Done Sent Done false
17677 5118 Acked Write 4 Done Sent Done false
17678 5119 Acked Read 5 Done Sent Done false
17679 5120 Acked Read 2 Done Sent Done false
17680 5121 Acked Write 8 Done Sent Done false
17681 5122 Acked Read 1 Done Sent Done false
17682 5123 Acked Read 8 Done Sent Done false
17683 5124 Acked Read 7 Done Sent Done false
17684 5125 Acked Read 4 Done Sent Done false
17685 5126 Acked Read 10 Done Sent Done false
17686 5127 Acked Read 5 Done Sent Done false
17687 5128 Acked Write 10 Done Sent Done false
17688 5129 Acked Read 2 Done Sent Done false
17689 5130 Acked Read 1 Done Sent Done false
17690 5131 Acked Read 9 Done Sent Done false
17691 5132 Acked Read 6 Done Sent Done false
17692 5133 Acked Read 6 Done Sent Done false
17693 5134 Acked Read 1 Done Sent Done false
17694 5135 Acked Write 7 Done Sent Done false
17695 5136 Acked Write 1 Done Sent Done false
17696 5137 Acked Write 6 Done Sent Done false
17697 5138 Acked Write 2 Done Sent Done false
17698 5139 Acked Read 9 Done Sent Done false
17699 5140 Acked Write 2 Done Sent Done false
17700 5141 Acked Write 8 Done Sent Done false
17701 5142 Acked Read 7 Done Sent Done false
17702 5143 Acked Write 4 Done Sent Done false
17703 5144 Acked Read 10 Done Sent Done false
17704 5145 Acked Read 3 Done Sent Done false
17705 5146 Acked Write 9 Done Sent Done false
17706 5147 Acked Write 1 Done Sent Done false
17707 5148 Acked Write 10 Done Sent Done false
17708 5149 Acked Write 3 Done Sent Done false
17709 5150 Acked Write 4 Done Sent Done false
17710 5151 Acked Read 8 Done Sent Done false
17711 5152 Acked Read 3 Done Sent Done false
17712 5153 Acked Read 10 Done Sent Done false
17713 5154 Acked Read 9 Done Sent Done false
17714 5155 Acked Read 10 Done Sent Done false
17715 5156 Acked Write 4 Done Sent Done false
17716 5157 Acked Read 3 Done Sent Done false
17717 5158 Acked Read 8 Done Sent Done false
17718 5159 Acked Write 5 Done Sent Done false
17719 5160 Acked Read 8 Done Sent Done false
17720 5161 Acked Write 4 Done Sent Done false
17721 5162 Acked Write 3 Done Sent Done false
17722 5163 Acked Read 10 Done Sent Done false
17723 5164 Acked Read 5 Done Sent Done false
17724 5165 Acked Read 7 Done Sent Done false
17725 5166 Acked Read 4 Done Sent Done false
17726 5167 Acked Read 1 Done Sent Done false
17727 5168 Acked Read 4 Done Sent Done false
17728 5169 Acked Write 1 Done Sent Done false
17729 5170 Acked Read 10 Done Sent Done false
17730 5171 Acked Write 8 Done Sent Done false
17731 5172 Acked Read 9 Done Sent Done false
17732 5173 Acked Read 6 Done Sent Done false
17733 5174 Acked Write 6 Done Sent Done false
17734 5175 Acked Read 2 Done Sent Done false
17735 5176 Acked Write 4 Done Sent Done false
17736 5177 Acked Write 4 Done Sent Done false
17737 5178 Acked Read 2 Done Sent Done false
17738 5179 Acked Read 9 Done Sent Done false
17739 5180 Acked Write 2 Done Sent Done false
17740 5181 Acked Read 5 Done Sent Done false
17741 5182 Acked Write 1 Done Sent Done false
17742 5183 Acked Write 3 Done Sent Done false
17743 5184 Acked Write 3 Done Sent Done false
17744 5185 Acked Write 6 Done Sent Done false
17745 5186 Acked Read 3 Done Sent Done false
17746 5187 Acked Write 1 Done Sent Done false
17747 5188 Acked Read 9 Done Sent Done false
17748 5189 Acked Read 7 Done Sent Done false
17749 5190 Acked Write 9 Done Sent Done false
17750 5191 Acked Write 7 Done Sent Done false
17751 5192 Acked Read 2 Done Sent Done false
17752 5193 Acked Flush 0 Done Sent Done false
17753 5194 Acked Read 1 Done Sent Done false
17754 5195 Acked Read 4 Done Sent Done false
17755 5196 Acked Read 10 Done Sent Done false
17756 5197 Acked Write 3 Done Sent Done false
17757 5198 Acked Write 6 Done Sent Done false
17758 5199 Acked Read 2 Done Sent Done false
17759 5200 Acked Write 2 Done Sent Done false
17760 5201 Acked Read 10 Done Sent Done false
17761 5202 Acked Read 8 Done Sent Done false
17762 5203 Acked Read 2 Done Sent Done false
17763 5204 Acked Read 7 Done Sent Done false
17764 5205 Acked Read 8 Done Sent Done false
17765 5206 Acked Read 10 Done Sent Done false
17766 5207 Acked Write 3 Done Sent Done false
17767 5208 Acked Read 4 Done Sent Done false
17768 5209 Acked Read 7 Done Sent Done false
17769 5210 Acked Read 3 Done Sent Done false
17770 5211 Acked Read 8 Done Sent Done false
17771 5212 Acked Read 1 Done Sent Done false
17772 5213 Acked Write 6 Done Sent Done false
17773 5214 Acked Read 10 Done Sent Done false
17774 5215 Acked Read 7 Done Sent Done false
17775 5216 Acked Read 3 Done Sent Done false
17776 5217 Acked Read 7 Done Sent Done false
17777 5218 Acked Read 7 Done Sent Done false
17778 5219 Acked Write 10 Done Sent Done false
17779 5220 Acked Read 4 Done Sent Done false
17780 5221 Acked Write 3 Done Sent Done false
17781 5222 Acked Read 7 Done Sent Done false
17782 5223 Acked Write 9 Done Sent Done false
17783 5224 Acked Read 8 Done Sent Done false
17784 5225 Acked Write 2 Done Sent Done false
17785 STATES DS:0 DS:1 DS:2 TOTAL
17786 Sent 0 301 0 301
17787 Done 301 0 301 602
17788 Skipped 0 0 0 0
17789 Error 0 0 0 0
17790 Last Flush: Some(JobId(5193)) Some(JobId(4924)) Some(JobId(5193))
17791 Downstairs last five completed: 4924 4923 4922 4921 4920
17792 Upstairs last five completed: 5225 5224 5223 5222 5221
17793 CLIENT: Up:0 ds:301 act:2
177942025-03-17T22:22:54.151ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
177952025-03-17T22:22:54.942ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
177962025-03-17T22:22:55.401ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
177972025-03-17T22:22:55.401ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
177982025-03-17T22:22:55.401ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
177992025-03-17T22:22:55.401ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
178002025-03-17T22:22:55.401ZINFOcrucible: client task is exiting = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
178012025-03-17T22:22:56.387ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
178022025-03-17T22:22:56.945ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
17803 [8820] got start action
17804 [8820] I am going to restart
17805 Starting downstairs at port 8820
17806 Make output file at "/tmp/dsc/downstairs-8820.txt"
17807 [8820][1] reports Starting
17808 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1225)
17809 [8820][1] reports Running
17810 ----------------------------------------------------------------
17811 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17812 States: Active Offline Active
17813 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17814 4925 Acked Write 7 Done Sent Done false
17815 4926 Acked Read 9 Done Sent Done false
17816 4927 Acked Read 1 Done Sent Done false
17817 4928 Acked Read 1 Done Sent Done false
17818 4929 Acked Write 9 Done Sent Done false
17819 4930 Acked Write 2 Done Sent Done false
17820 4931 Acked Write 5 Done Sent Done false
17821 4932 Acked Write 10 Done Sent Done false
17822 4933 Acked Read 5 Done Sent Done false
17823 4934 Acked Write 1 Done Sent Done false
17824 4935 Acked Read 10 Done Sent Done false
17825 4936 Acked Flush 0 Done Sent Done false
17826 4937 Acked Read 9 Done Sent Done false
17827 4938 Acked Write 3 Done Sent Done false
17828 4939 Acked Read 2 Done Sent Done false
17829 4940 Acked Write 9 Done Sent Done false
17830 4941 Acked Read 1 Done Sent Done false
17831 4942 Acked Write 2 Done Sent Done false
17832 4943 Acked Flush 0 Done Sent Done false
17833 4944 Acked Write 6 Done Sent Done false
17834 4945 Acked Write 8 Done Sent Done false
17835 4946 Acked Read 7 Done Sent Done false
17836 4947 Acked Write 9 Done Sent Done false
17837 4948 Acked Write 9 Done Sent Done false
17838 4949 Acked Write 4 Done Sent Done false
17839 4950 Acked Write 8 Done Sent Done false
17840 4951 Acked Read 3 Done Sent Done false
17841 4952 Acked Read 5 Done Sent Done false
17842 4953 Acked Write 10 Done Sent Done false
17843 4954 Acked Read 9 Done Sent Done false
17844 4955 Acked Read 6 Done Sent Done false
17845 4956 Acked Write 3 Done Sent Done false
17846 4957 Acked Write 9 Done Sent Done false
17847 4958 Acked Write 9 Done Sent Done false
17848 4959 Acked Read 6 Done Sent Done false
17849 4960 Acked Write 6 Done Sent Done false
17850 4961 Acked Write 7 Done Sent Done false
17851 4962 Acked Read 2 Done Sent Done false
17852 4963 Acked Write 3 Done Sent Done false
17853 4964 Acked Write 8 Done Sent Done false
17854 4965 Acked Write 6 Done Sent Done false
17855 4966 Acked Read 8 Done Sent Done false
17856 4967 Acked Read 2 Done Sent Done false
17857 4968 Acked Read 1 Done Sent Done false
17858 4969 Acked Flush 0 Done Sent Done false
17859 4970 Acked Read 1 Done Sent Done false
17860 4971 Acked Write 10 Done Sent Done false
17861 4972 Acked Write 8 Done Sent Done false
17862 4973 Acked Write 10 Done Sent Done false
17863 4974 Acked Read 1 Done Sent Done false
17864 4975 Acked Read 9 Done Sent Done false
17865 4976 Acked Write 5 Done Sent Done false
17866 4977 Acked Write 5 Done Sent Done false
17867 4978 Acked Write 4 Done Sent Done false
17868 4979 Acked Write 7 Done Sent Done false
17869 4980 Acked Write 6 Done Sent Done false
17870 4981 Acked Read 4 Done Sent Done false
17871 4982 Acked Write 3 Done Sent Done false
17872 4983 Acked Write 9 Done Sent Done false
17873 4984 Acked Read 6 Done Sent Done false
17874 4985 Acked Read 1 Done Sent Done false
17875 4986 Acked Read 3 Done Sent Done false
17876 4987 Acked Read 7 Done Sent Done false
17877 4988 Acked Read 3 Done Sent Done false
17878 4989 Acked Flush 0 Done Sent Done false
17879 4990 Acked Flush 0 Done Sent Done false
17880 4991 Acked Flush 0 Done Sent Done false
17881 4992 Acked Write 9 Done Sent Done false
17882 4993 Acked Read 4 Done Sent Done false
17883 4994 Acked Read 5 Done Sent Done false
17884 4995 Acked Read 1 Done Sent Done false
17885 4996 Acked Read 6 Done Sent Done false
17886 4997 Acked Write 2 Done Sent Done false
17887 4998 Acked Read 3 Done Sent Done false
17888 4999 Acked Read 9 Done Sent Done false
17889 5000 Acked Read 5 Done Sent Done false
17890 5001 Acked Read 5 Done Sent Done false
17891 5002 Acked Read 2 Done Sent Done false
17892 5003 Acked Read 10 Done Sent Done false
17893 5004 Acked Write 1 Done Sent Done false
17894 5005 Acked Read 4 Done Sent Done false
17895 5006 Acked Read 6 Done Sent Done false
17896 5007 Acked Write 3 Done Sent Done false
17897 5008 Acked Read 6 Done Sent Done false
17898 5009 Acked Read 9 Done Sent Done false
17899 5010 Acked Write 8 Done Sent Done false
17900 5011 Acked Read 9 Done Sent Done false
17901 5012 Acked Read 10 Done Sent Done false
17902 5013 Acked Write 2 Done Sent Done false
17903 5014 Acked Read 6 Done Sent Done false
17904 5015 Acked Write 10 Done Sent Done false
17905 5016 Acked Read 9 Done Sent Done false
17906 5017 Acked Write 10 Done Sent Done false
17907 5018 Acked Write 5 Done Sent Done false
17908 5019 Acked Write 9 Done Sent Done false
17909 5020 Acked Write 6 Done Sent Done false
17910 5021 Acked Flush 0 Done Sent Done false
17911 5022 Acked Read 6 Done Sent Done false
17912 5023 Acked Write 4 Done Sent Done false
17913 5024 Acked Read 4 Done Sent Done false
17914 5025 Acked Read 7 Done Sent Done false
17915 5026 Acked Flush 0 Done Sent Done false
17916 5027 Acked Read 8 Done Sent Done false
17917 5028 Acked Flush 0 Done Sent Done false
17918 5029 Acked Write 1 Done Sent Done false
17919 5030 Acked Flush 0 Done Sent Done false
17920 5031 Acked Write 4 Done Sent Done false
17921 5032 Acked Write 7 Done Sent Done false
17922 5033 Acked Write 10 Done Sent Done false
17923 5034 Acked Flush 0 Done Sent Done false
17924 5035 Acked Flush 0 Done Sent Done false
17925 5036 Acked Read 4 Done Sent Done false
17926 5037 Acked Write 4 Done Sent Done false
17927 5038 Acked Write 6 Done Sent Done false
17928 5039 Acked Flush 0 Done Sent Done false
17929 5040 Acked Read 6 Done Sent Done false
17930 5041 Acked Flush 0 Done Sent Done false
17931 5042 Acked Read 9 Done Sent Done false
17932 5043 Acked Read 5 Done Sent Done false
17933 5044 Acked Write 1 Done Sent Done false
17934 5045 Acked Read 9 Done Sent Done false
17935 5046 Acked Write 7 Done Sent Done false
17936 5047 Acked Read 9 Done Sent Done false
17937 5048 Acked Write 6 Done Sent Done false
17938 5049 Acked Write 5 Done Sent Done false
17939 5050 Acked Write 3 Done Sent Done false
17940 5051 Acked Write 4 Done Sent Done false
17941 5052 Acked Read 5 Done Sent Done false
17942 5053 Acked Flush 0 Done Sent Done false
17943 5054 Acked Read 4 Done Sent Done false
17944 5055 Acked Write 3 Done Sent Done false
17945 5056 Acked Read 4 Done Sent Done false
17946 5057 Acked Read 4 Done Sent Done false
17947 5058 Acked Write 3 Done Sent Done false
17948 5059 Acked Read 1 Done Sent Done false
17949 5060 Acked Flush 0 Done Sent Done false
17950 5061 Acked Write 1 Done Sent Done false
17951 5062 Acked Write 7 Done Sent Done false
17952 5063 Acked Write 10 Done Sent Done false
17953 5064 Acked Write 7 Done Sent Done false
17954 5065 Acked Write 10 Done Sent Done false
17955 5066 Acked Write 8 Done Sent Done false
17956 5067 Acked Write 7 Done Sent Done false
17957 5068 Acked Read 4 Done Sent Done false
17958 5069 Acked Read 6 Done Sent Done false
17959 5070 Acked Write 4 Done Sent Done false
17960 5071 Acked Write 5 Done Sent Done false
17961 5072 Acked Write 3 Done Sent Done false
17962 5073 Acked Read 9 Done Sent Done false
17963 5074 Acked Write 7 Done Sent Done false
17964 5075 Acked Read 7 Done Sent Done false
17965 5076 Acked Read 9 Done Sent Done false
17966 5077 Acked Write 4 Done Sent Done false
17967 5078 Acked Read 10 Done Sent Done false
17968 5079 Acked Read 5 Done Sent Done false
17969 5080 Acked Read 7 Done Sent Done false
17970 5081 Acked Write 4 Done Sent Done false
17971 5082 Acked Write 8 Done Sent Done false
17972 5083 Acked Write 3 Done Sent Done false
17973 5084 Acked Read 10 Done Sent Done false
17974 5085 Acked Write 9 Done Sent Done false
17975 5086 Acked Read 6 Done Sent Done false
17976 5087 Acked Write 7 Done Sent Done false
17977 5088 Acked Read 8 Done Sent Done false
17978 5089 Acked Read 9 Done Sent Done false
17979 5090 Acked Read 6 Done Sent Done false
17980 5091 Acked Write 3 Done Sent Done false
17981 5092 Acked Write 7 Done Sent Done false
17982 5093 Acked Flush 0 Done Sent Done false
17983 5094 Acked Read 9 Done Sent Done false
17984 5095 Acked Write 3 Done Sent Done false
17985 5096 Acked Read 6 Done Sent Done false
17986 5097 Acked Read 10 Done Sent Done false
17987 5098 Acked Write 5 Done Sent Done false
17988 5099 Acked Write 2 Done Sent Done false
17989 5100 Acked Read 3 Done Sent Done false
17990 5101 Acked Read 5 Done Sent Done false
17991 5102 Acked Write 5 Done Sent Done false
17992 5103 Acked Read 4 Done Sent Done false
17993 5104 Acked Read 7 Done Sent Done false
17994 5105 Acked Write 6 Done Sent Done false
17995 5106 Acked Write 7 Done Sent Done false
17996 5107 Acked Read 9 Done Sent Done false
17997 5108 Acked Write 1 Done Sent Done false
17998 5109 Acked Write 2 Done Sent Done false
17999 5110 Acked Read 1 Done Sent Done false
18000 5111 Acked Read 1 Done Sent Done false
18001 5112 Acked Read 4 Done Sent Done false
18002 5113 Acked Write 5 Done Sent Done false
18003 5114 Acked Write 10 Done Sent Done false
18004 5115 Acked Read 1 Done Sent Done false
18005 5116 Acked Read 10 Done Sent Done false
18006 5117 Acked Flush 0 Done Sent Done false
18007 5118 Acked Write 4 Done Sent Done false
18008 5119 Acked Read 5 Done Sent Done false
18009 5120 Acked Read 2 Done Sent Done false
18010 5121 Acked Write 8 Done Sent Done false
18011 5122 Acked Read 1 Done Sent Done false
18012 5123 Acked Read 8 Done Sent Done false
18013 5124 Acked Read 7 Done Sent Done false
18014 5125 Acked Read 4 Done Sent Done false
18015 5126 Acked Read 10 Done Sent Done false
18016 5127 Acked Read 5 Done Sent Done false
18017 5128 Acked Write 10 Done Sent Done false
18018 5129 Acked Read 2 Done Sent Done false
18019 5130 Acked Read 1 Done Sent Done false
18020 5131 Acked Read 9 Done Sent Done false
18021 5132 Acked Read 6 Done Sent Done false
18022 5133 Acked Read 6 Done Sent Done false
18023 5134 Acked Read 1 Done Sent Done false
18024 5135 Acked Write 7 Done Sent Done false
18025 5136 Acked Write 1 Done Sent Done false
18026 5137 Acked Write 6 Done Sent Done false
18027 5138 Acked Write 2 Done Sent Done false
18028 5139 Acked Read 9 Done Sent Done false
18029 5140 Acked Write 2 Done Sent Done false
18030 5141 Acked Write 8 Done Sent Done false
18031 5142 Acked Read 7 Done Sent Done false
18032 5143 Acked Write 4 Done Sent Done false
18033 5144 Acked Read 10 Done Sent Done false
18034 5145 Acked Read 3 Done Sent Done false
18035 5146 Acked Write 9 Done Sent Done false
18036 5147 Acked Write 1 Done Sent Done false
18037 5148 Acked Write 10 Done Sent Done false
18038 5149 Acked Write 3 Done Sent Done false
18039 5150 Acked Write 4 Done Sent Done false
18040 5151 Acked Read 8 Done Sent Done false
18041 5152 Acked Read 3 Done Sent Done false
18042 5153 Acked Read 10 Done Sent Done false
18043 5154 Acked Read 9 Done Sent Done false
18044 5155 Acked Read 10 Done Sent Done false
18045 5156 Acked Write 4 Done Sent Done false
18046 5157 Acked Read 3 Done Sent Done false
18047 5158 Acked Read 8 Done Sent Done false
18048 5159 Acked Write 5 Done Sent Done false
18049 5160 Acked Read 8 Done Sent Done false
18050 5161 Acked Write 4 Done Sent Done false
18051 5162 Acked Write 3 Done Sent Done false
18052 5163 Acked Read 10 Done Sent Done false
18053 5164 Acked Read 5 Done Sent Done false
18054 5165 Acked Read 7 Done Sent Done false
18055 5166 Acked Read 4 Done Sent Done false
18056 5167 Acked Read 1 Done Sent Done false
18057 5168 Acked Read 4 Done Sent Done false
18058 5169 Acked Write 1 Done Sent Done false
18059 5170 Acked Read 10 Done Sent Done false
18060 5171 Acked Write 8 Done Sent Done false
18061 5172 Acked Read 9 Done Sent Done false
18062 5173 Acked Read 6 Done Sent Done false
18063 5174 Acked Write 6 Done Sent Done false
18064 5175 Acked Read 2 Done Sent Done false
18065 5176 Acked Write 4 Done Sent Done false
18066 5177 Acked Write 4 Done Sent Done false
18067 5178 Acked Read 2 Done Sent Done false
18068 5179 Acked Read 9 Done Sent Done false
18069 5180 Acked Write 2 Done Sent Done false
18070 5181 Acked Read 5 Done Sent Done false
18071 5182 Acked Write 1 Done Sent Done false
18072 5183 Acked Write 3 Done Sent Done false
18073 5184 Acked Write 3 Done Sent Done false
18074 5185 Acked Write 6 Done Sent Done false
18075 5186 Acked Read 3 Done Sent Done false
18076 5187 Acked Write 1 Done Sent Done false
18077 5188 Acked Read 9 Done Sent Done false
18078 5189 Acked Read 7 Done Sent Done false
18079 5190 Acked Write 9 Done Sent Done false
18080 5191 Acked Write 7 Done Sent Done false
18081 5192 Acked Read 2 Done Sent Done false
18082 5193 Acked Flush 0 Done Sent Done false
18083 5194 Acked Read 1 Done Sent Done false
18084 5195 Acked Read 4 Done Sent Done false
18085 5196 Acked Read 10 Done Sent Done false
18086 5197 Acked Write 3 Done Sent Done false
18087 5198 Acked Write 6 Done Sent Done false
18088 5199 Acked Read 2 Done Sent Done false
18089 5200 Acked Write 2 Done Sent Done false
18090 5201 Acked Read 10 Done Sent Done false
18091 5202 Acked Read 8 Done Sent Done false
18092 5203 Acked Read 2 Done Sent Done false
18093 5204 Acked Read 7 Done Sent Done false
18094 5205 Acked Read 8 Done Sent Done false
18095 5206 Acked Read 10 Done Sent Done false
18096 5207 Acked Write 3 Done Sent Done false
18097 5208 Acked Read 4 Done Sent Done false
18098 5209 Acked Read 7 Done Sent Done false
18099 5210 Acked Read 3 Done Sent Done false
18100 5211 Acked Read 8 Done Sent Done false
18101 5212 Acked Read 1 Done Sent Done false
18102 5213 Acked Write 6 Done Sent Done false
18103 5214 Acked Read 10 Done Sent Done false
18104 5215 Acked Read 7 Done Sent Done false
18105 5216 Acked Read 3 Done Sent Done false
18106 5217 Acked Read 7 Done Sent Done false
18107 5218 Acked Read 7 Done Sent Done false
18108 5219 Acked Write 10 Done Sent Done false
18109 5220 Acked Read 4 Done Sent Done false
18110 5221 Acked Write 3 Done Sent Done false
18111 5222 Acked Read 7 Done Sent Done false
18112 5223 Acked Write 9 Done Sent Done false
18113 5224 Acked Read 8 Done Sent Done false
18114 5225 Acked Write 2 Done Sent Done false
18115 STATES DS:0 DS:1 DS:2 TOTAL
18116 Sent 0 301 0 301
18117 Done 301 0 301 602
18118 Skipped 0 0 0 0
18119 Error 0 0 0 0
18120 Last Flush: Some(JobId(5193)) Some(JobId(4924)) Some(JobId(5193))
18121 Downstairs last five completed: 4924 4923 4922 4921 4920
18122 Upstairs last five completed: 5225 5224 5223 5222 5221
18123 CLIENT: Up:0 ds:301 act:2
181242025-03-17T22:22:58.950ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
181252025-03-17T22:22:59.951ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
181262025-03-17T22:23:01.388ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
18127 ----------------------------------------------------------------
18128 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18129 States: Active Offline Active
18130 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18131 4925 Acked Write 7 Done Sent Done false
18132 4926 Acked Read 9 Done Sent Done false
18133 4927 Acked Read 1 Done Sent Done false
18134 4928 Acked Read 1 Done Sent Done false
18135 4929 Acked Write 9 Done Sent Done false
18136 4930 Acked Write 2 Done Sent Done false
18137 4931 Acked Write 5 Done Sent Done false
18138 4932 Acked Write 10 Done Sent Done false
18139 4933 Acked Read 5 Done Sent Done false
18140 4934 Acked Write 1 Done Sent Done false
18141 4935 Acked Read 10 Done Sent Done false
18142 4936 Acked Flush 0 Done Sent Done false
18143 4937 Acked Read 9 Done Sent Done false
18144 4938 Acked Write 3 Done Sent Done false
18145 4939 Acked Read 2 Done Sent Done false
18146 4940 Acked Write 9 Done Sent Done false
18147 4941 Acked Read 1 Done Sent Done false
18148 4942 Acked Write 2 Done Sent Done false
18149 4943 Acked Flush 0 Done Sent Done false
18150 4944 Acked Write 6 Done Sent Done false
18151 4945 Acked Write 8 Done Sent Done false
18152 4946 Acked Read 7 Done Sent Done false
18153 4947 Acked Write 9 Done Sent Done false
18154 4948 Acked Write 9 Done Sent Done false
18155 4949 Acked Write 4 Done Sent Done false
18156 4950 Acked Write 8 Done Sent Done false
18157 4951 Acked Read 3 Done Sent Done false
18158 4952 Acked Read 5 Done Sent Done false
18159 4953 Acked Write 10 Done Sent Done false
18160 4954 Acked Read 9 Done Sent Done false
18161 4955 Acked Read 6 Done Sent Done false
18162 4956 Acked Write 3 Done Sent Done false
18163 4957 Acked Write 9 Done Sent Done false
18164 4958 Acked Write 9 Done Sent Done false
18165 4959 Acked Read 6 Done Sent Done false
18166 4960 Acked Write 6 Done Sent Done false
18167 4961 Acked Write 7 Done Sent Done false
18168 4962 Acked Read 2 Done Sent Done false
18169 4963 Acked Write 3 Done Sent Done false
18170 4964 Acked Write 8 Done Sent Done false
18171 4965 Acked Write 6 Done Sent Done false
18172 4966 Acked Read 8 Done Sent Done false
18173 4967 Acked Read 2 Done Sent Done false
18174 4968 Acked Read 1 Done Sent Done false
18175 4969 Acked Flush 0 Done Sent Done false
18176 4970 Acked Read 1 Done Sent Done false
18177 4971 Acked Write 10 Done Sent Done false
18178 4972 Acked Write 8 Done Sent Done false
18179 4973 Acked Write 10 Done Sent Done false
18180 4974 Acked Read 1 Done Sent Done false
18181 4975 Acked Read 9 Done Sent Done false
18182 4976 Acked Write 5 Done Sent Done false
18183 4977 Acked Write 5 Done Sent Done false
18184 4978 Acked Write 4 Done Sent Done false
18185 4979 Acked Write 7 Done Sent Done false
18186 4980 Acked Write 6 Done Sent Done false
18187 4981 Acked Read 4 Done Sent Done false
18188 4982 Acked Write 3 Done Sent Done false
18189 4983 Acked Write 9 Done Sent Done false
18190 4984 Acked Read 6 Done Sent Done false
18191 4985 Acked Read 1 Done Sent Done false
18192 4986 Acked Read 3 Done Sent Done false
18193 4987 Acked Read 7 Done Sent Done false
18194 4988 Acked Read 3 Done Sent Done false
18195 4989 Acked Flush 0 Done Sent Done false
18196 4990 Acked Flush 0 Done Sent Done false
18197 4991 Acked Flush 0 Done Sent Done false
18198 4992 Acked Write 9 Done Sent Done false
18199 4993 Acked Read 4 Done Sent Done false
18200 4994 Acked Read 5 Done Sent Done false
18201 4995 Acked Read 1 Done Sent Done false
18202 4996 Acked Read 6 Done Sent Done false
18203 4997 Acked Write 2 Done Sent Done false
18204 4998 Acked Read 3 Done Sent Done false
18205 4999 Acked Read 9 Done Sent Done false
18206 5000 Acked Read 5 Done Sent Done false
18207 5001 Acked Read 5 Done Sent Done false
18208 5002 Acked Read 2 Done Sent Done false
18209 5003 Acked Read 10 Done Sent Done false
18210 5004 Acked Write 1 Done Sent Done false
18211 5005 Acked Read 4 Done Sent Done false
18212 5006 Acked Read 6 Done Sent Done false
18213 5007 Acked Write 3 Done Sent Done false
18214 5008 Acked Read 6 Done Sent Done false
18215 5009 Acked Read 9 Done Sent Done false
18216 5010 Acked Write 8 Done Sent Done false
18217 5011 Acked Read 9 Done Sent Done false
18218 5012 Acked Read 10 Done Sent Done false
18219 5013 Acked Write 2 Done Sent Done false
18220 5014 Acked Read 6 Done Sent Done false
18221 5015 Acked Write 10 Done Sent Done false
18222 5016 Acked Read 9 Done Sent Done false
18223 5017 Acked Write 10 Done Sent Done false
18224 5018 Acked Write 5 Done Sent Done false
18225 5019 Acked Write 9 Done Sent Done false
18226 5020 Acked Write 6 Done Sent Done false
18227 5021 Acked Flush 0 Done Sent Done false
18228 5022 Acked Read 6 Done Sent Done false
18229 5023 Acked Write 4 Done Sent Done false
18230 5024 Acked Read 4 Done Sent Done false
18231 5025 Acked Read 7 Done Sent Done false
18232 5026 Acked Flush 0 Done Sent Done false
18233 5027 Acked Read 8 Done Sent Done false
18234 5028 Acked Flush 0 Done Sent Done false
18235 5029 Acked Write 1 Done Sent Done false
18236 5030 Acked Flush 0 Done Sent Done false
18237 5031 Acked Write 4 Done Sent Done false
18238 5032 Acked Write 7 Done Sent Done false
18239 5033 Acked Write 10 Done Sent Done false
18240 5034 Acked Flush 0 Done Sent Done false
18241 5035 Acked Flush 0 Done Sent Done false
18242 5036 Acked Read 4 Done Sent Done false
18243 5037 Acked Write 4 Done Sent Done false
18244 5038 Acked Write 6 Done Sent Done false
18245 5039 Acked Flush 0 Done Sent Done false
18246 5040 Acked Read 6 Done Sent Done false
18247 5041 Acked Flush 0 Done Sent Done false
18248 5042 Acked Read 9 Done Sent Done false
18249 5043 Acked Read 5 Done Sent Done false
18250 5044 Acked Write 1 Done Sent Done false
18251 5045 Acked Read 9 Done Sent Done false
18252 5046 Acked Write 7 Done Sent Done false
18253 5047 Acked Read 9 Done Sent Done false
18254 5048 Acked Write 6 Done Sent Done false
18255 5049 Acked Write 5 Done Sent Done false
18256 5050 Acked Write 3 Done Sent Done false
18257 5051 Acked Write 4 Done Sent Done false
18258 5052 Acked Read 5 Done Sent Done false
18259 5053 Acked Flush 0 Done Sent Done false
18260 5054 Acked Read 4 Done Sent Done false
18261 5055 Acked Write 3 Done Sent Done false
18262 5056 Acked Read 4 Done Sent Done false
18263 5057 Acked Read 4 Done Sent Done false
18264 5058 Acked Write 3 Done Sent Done false
18265 5059 Acked Read 1 Done Sent Done false
18266 5060 Acked Flush 0 Done Sent Done false
18267 5061 Acked Write 1 Done Sent Done false
18268 5062 Acked Write 7 Done Sent Done false
18269 5063 Acked Write 10 Done Sent Done false
18270 5064 Acked Write 7 Done Sent Done false
18271 5065 Acked Write 10 Done Sent Done false
18272 5066 Acked Write 8 Done Sent Done false
18273 5067 Acked Write 7 Done Sent Done false
18274 5068 Acked Read 4 Done Sent Done false
18275 5069 Acked Read 6 Done Sent Done false
18276 5070 Acked Write 4 Done Sent Done false
18277 5071 Acked Write 5 Done Sent Done false
18278 5072 Acked Write 3 Done Sent Done false
18279 5073 Acked Read 9 Done Sent Done false
18280 5074 Acked Write 7 Done Sent Done false
18281 5075 Acked Read 7 Done Sent Done false
18282 5076 Acked Read 9 Done Sent Done false
18283 5077 Acked Write 4 Done Sent Done false
18284 5078 Acked Read 10 Done Sent Done false
18285 5079 Acked Read 5 Done Sent Done false
18286 5080 Acked Read 7 Done Sent Done false
18287 5081 Acked Write 4 Done Sent Done false
18288 5082 Acked Write 8 Done Sent Done false
18289 5083 Acked Write 3 Done Sent Done false
18290 5084 Acked Read 10 Done Sent Done false
18291 5085 Acked Write 9 Done Sent Done false
18292 5086 Acked Read 6 Done Sent Done false
18293 5087 Acked Write 7 Done Sent Done false
18294 5088 Acked Read 8 Done Sent Done false
18295 5089 Acked Read 9 Done Sent Done false
18296 5090 Acked Read 6 Done Sent Done false
18297 5091 Acked Write 3 Done Sent Done false
18298 5092 Acked Write 7 Done Sent Done false
18299 5093 Acked Flush 0 Done Sent Done false
18300 5094 Acked Read 9 Done Sent Done false
18301 5095 Acked Write 3 Done Sent Done false
18302 5096 Acked Read 6 Done Sent Done false
18303 5097 Acked Read 10 Done Sent Done false
18304 5098 Acked Write 5 Done Sent Done false
18305 5099 Acked Write 2 Done Sent Done false
18306 5100 Acked Read 3 Done Sent Done false
18307 5101 Acked Read 5 Done Sent Done false
18308 5102 Acked Write 5 Done Sent Done false
18309 5103 Acked Read 4 Done Sent Done false
18310 5104 Acked Read 7 Done Sent Done false
18311 5105 Acked Write 6 Done Sent Done false
18312 5106 Acked Write 7 Done Sent Done false
18313 5107 Acked Read 9 Done Sent Done false
18314 5108 Acked Write 1 Done Sent Done false
18315 5109 Acked Write 2 Done Sent Done false
18316 5110 Acked Read 1 Done Sent Done false
18317 5111 Acked Read 1 Done Sent Done false
18318 5112 Acked Read 4 Done Sent Done false
18319 5113 Acked Write 5 Done Sent Done false
18320 5114 Acked Write 10 Done Sent Done false
18321 5115 Acked Read 1 Done Sent Done false
18322 5116 Acked Read 10 Done Sent Done false
18323 5117 Acked Flush 0 Done Sent Done false
18324 5118 Acked Write 4 Done Sent Done false
18325 5119 Acked Read 5 Done Sent Done false
18326 5120 Acked Read 2 Done Sent Done false
18327 5121 Acked Write 8 Done Sent Done false
18328 5122 Acked Read 1 Done Sent Done false
18329 5123 Acked Read 8 Done Sent Done false
18330 5124 Acked Read 7 Done Sent Done false
18331 5125 Acked Read 4 Done Sent Done false
18332 5126 Acked Read 10 Done Sent Done false
18333 5127 Acked Read 5 Done Sent Done false
18334 5128 Acked Write 10 Done Sent Done false
18335 5129 Acked Read 2 Done Sent Done false
18336 5130 Acked Read 1 Done Sent Done false
18337 5131 Acked Read 9 Done Sent Done false
18338 5132 Acked Read 6 Done Sent Done false
18339 5133 Acked Read 6 Done Sent Done false
18340 5134 Acked Read 1 Done Sent Done false
18341 5135 Acked Write 7 Done Sent Done false
18342 5136 Acked Write 1 Done Sent Done false
18343 5137 Acked Write 6 Done Sent Done false
18344 5138 Acked Write 2 Done Sent Done false
18345 5139 Acked Read 9 Done Sent Done false
18346 5140 Acked Write 2 Done Sent Done false
18347 5141 Acked Write 8 Done Sent Done false
18348 5142 Acked Read 7 Done Sent Done false
18349 5143 Acked Write 4 Done Sent Done false
18350 5144 Acked Read 10 Done Sent Done false
18351 5145 Acked Read 3 Done Sent Done false
18352 5146 Acked Write 9 Done Sent Done false
18353 5147 Acked Write 1 Done Sent Done false
18354 5148 Acked Write 10 Done Sent Done false
18355 5149 Acked Write 3 Done Sent Done false
18356 5150 Acked Write 4 Done Sent Done false
18357 5151 Acked Read 8 Done Sent Done false
18358 5152 Acked Read 3 Done Sent Done false
18359 5153 Acked Read 10 Done Sent Done false
18360 5154 Acked Read 9 Done Sent Done false
18361 5155 Acked Read 10 Done Sent Done false
18362 5156 Acked Write 4 Done Sent Done false
18363 5157 Acked Read 3 Done Sent Done false
18364 5158 Acked Read 8 Done Sent Done false
18365 5159 Acked Write 5 Done Sent Done false
18366 5160 Acked Read 8 Done Sent Done false
18367 5161 Acked Write 4 Done Sent Done false
18368 5162 Acked Write 3 Done Sent Done false
18369 5163 Acked Read 10 Done Sent Done false
18370 5164 Acked Read 5 Done Sent Done false
18371 5165 Acked Read 7 Done Sent Done false
18372 5166 Acked Read 4 Done Sent Done false
18373 5167 Acked Read 1 Done Sent Done false
18374 5168 Acked Read 4 Done Sent Done false
18375 5169 Acked Write 1 Done Sent Done false
18376 5170 Acked Read 10 Done Sent Done false
18377 5171 Acked Write 8 Done Sent Done false
18378 5172 Acked Read 9 Done Sent Done false
18379 5173 Acked Read 6 Done Sent Done false
18380 5174 Acked Write 6 Done Sent Done false
18381 5175 Acked Read 2 Done Sent Done false
18382 5176 Acked Write 4 Done Sent Done false
18383 5177 Acked Write 4 Done Sent Done false
18384 5178 Acked Read 2 Done Sent Done false
18385 5179 Acked Read 9 Done Sent Done false
18386 5180 Acked Write 2 Done Sent Done false
18387 5181 Acked Read 5 Done Sent Done false
18388 5182 Acked Write 1 Done Sent Done false
18389 5183 Acked Write 3 Done Sent Done false
18390 5184 Acked Write 3 Done Sent Done false
18391 5185 Acked Write 6 Done Sent Done false
18392 5186 Acked Read 3 Done Sent Done false
18393 5187 Acked Write 1 Done Sent Done false
18394 5188 Acked Read 9 Done Sent Done false
18395 5189 Acked Read 7 Done Sent Done false
18396 5190 Acked Write 9 Done Sent Done false
18397 5191 Acked Write 7 Done Sent Done false
18398 5192 Acked Read 2 Done Sent Done false
18399 5193 Acked Flush 0 Done Sent Done false
18400 5194 Acked Read 1 Done Sent Done false
18401 5195 Acked Read 4 Done Sent Done false
18402 5196 Acked Read 10 Done Sent Done false
18403 5197 Acked Write 3 Done Sent Done false
18404 5198 Acked Write 6 Done Sent Done false
18405 5199 Acked Read 2 Done Sent Done false
18406 5200 Acked Write 2 Done Sent Done false
18407 5201 Acked Read 10 Done Sent Done false
18408 5202 Acked Read 8 Done Sent Done false
18409 5203 Acked Read 2 Done Sent Done false
18410 5204 Acked Read 7 Done Sent Done false
18411 5205 Acked Read 8 Done Sent Done false
18412 5206 Acked Read 10 Done Sent Done false
18413 5207 Acked Write 3 Done Sent Done false
18414 5208 Acked Read 4 Done Sent Done false
18415 5209 Acked Read 7 Done Sent Done false
18416 5210 Acked Read 3 Done Sent Done false
18417 5211 Acked Read 8 Done Sent Done false
18418 5212 Acked Read 1 Done Sent Done false
18419 5213 Acked Write 6 Done Sent Done false
18420 5214 Acked Read 10 Done Sent Done false
18421 5215 Acked Read 7 Done Sent Done false
18422 5216 Acked Read 3 Done Sent Done false
18423 5217 Acked Read 7 Done Sent Done false
18424 5218 Acked Read 7 Done Sent Done false
18425 5219 Acked Write 10 Done Sent Done false
18426 5220 Acked Read 4 Done Sent Done false
18427 5221 Acked Write 3 Done Sent Done false
18428 5222 Acked Read 7 Done Sent Done false
18429 5223 Acked Write 9 Done Sent Done false
18430 5224 Acked Read 8 Done Sent Done false
18431 5225 Acked Write 2 Done Sent Done false
18432 STATES DS:0 DS:1 DS:2 TOTAL
18433 Sent 0 301 0 301
18434 Done 301 0 301 602
18435 Skipped 0 0 0 0
18436 Error 0 0 0 0
18437 Last Flush: Some(JobId(5193)) Some(JobId(4924)) Some(JobId(5193))
18438 Downstairs last five completed: 4924 4923 4922 4921 4920
18439 Upstairs last five completed: 5225 5224 5223 5222 5221
18440 CLIENT: Up:0 ds:301 act:2
184412025-03-17T22:23:02.156ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
184422025-03-17T22:23:02.957ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
184432025-03-17T22:23:03.958ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
184442025-03-17T22:23:04.960ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
184452025-03-17T22:23:05.402ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
184462025-03-17T22:23:05.402ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:51717) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
184472025-03-17T22:23:05.403ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
184482025-03-17T22:23:05.403ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
184492025-03-17T22:23:05.403ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
184502025-03-17T22:23:05.403ZINFOcrucible: send last flush ID to this DS: Some(JobId(4924)) = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
184512025-03-17T22:23:05.403ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
184522025-03-17T22:23:05.403ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(4924)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
184532025-03-17T22:23:05.404ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(4924)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
18454 ----------------------------------------------------------------
18455 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:32
18456 States: Active Active Active
18457 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18458 5194 Acked Read 1 Done Done Done true
18459 5195 Acked Read 4 Done Done Done true
18460 5196 Acked Read 10 Done Done Done true
18461 5197 Acked Write 3 Done Done Done true
18462 5198 Acked Write 6 Done Done Done true
18463 5199 Acked Read 2 Done Done Done true
18464 5200 Acked Write 2 Done Done Done true
18465 5201 Acked Read 10 Done Done Done true
18466 5202 Acked Read 8 Done Done Done true
18467 5203 Acked Read 2 Done Done Done true
18468 5204 Acked Read 7 Done Done Done true
18469 5205 Acked Read 8 Done Done Done true
18470 5206 Acked Read 10 Done Done Done true
18471 5207 Acked Write 3 Done Done Done true
18472 5208 Acked Read 4 Done Done Done true
18473 5209 Acked Read 7 Done Done Done true
18474 5210 Acked Read 3 Done Done Done true
18475 5211 Acked Read 8 Done Done Done true
18476 5212 Acked Read 1 Done Done Done true
18477 5213 Acked Write 6 Done Done Done true
18478 5214 Acked Read 10 Done Done Done true
18479 5215 Acked Read 7 Done Done Done true
18480 5216 Acked Read 3 Done Done Done true
18481 5217 Acked Read 7 Done Done Done true
18482 5218 Acked Read 7 Done Done Done true
18483 5219 Acked Write 10 Done Done Done true
18484 5220 Acked Read 4 Done Done Done true
18485 5221 Acked Write 3 Done Done Done true
18486 5222 Acked Read 7 Done Done Done true
18487 5223 Acked Write 9 Done Done Done true
18488 5224 Acked Read 8 Done Done Done true
18489 5225 Acked Write 2 Done Done Done true
18490 STATES DS:0 DS:1 DS:2 TOTAL
18491 Sent 0 0 0 0
18492 Done 32 32 32 96
18493 Skipped 0 0 0 0
18494 Error 0 0 0 0
18495 Last Flush: Some(JobId(5193)) Some(JobId(5193)) Some(JobId(5193))
18496 Downstairs last five completed: 5193 5192 5191 5190 5189
18497 Upstairs last five completed: 5225 5224 5223 5222 5221
18498 CLIENT: Up:0 ds:32 act:3
18499 ----------------------------------------------------------------
18500 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
18501 Downstairs last five completed: 5226 5225 5224 5223 5222
18502 Upstairs last five completed: 5226 5225 5224 5223 5222
18503 CLIENT: Up:0 ds:0 act:3
18504 Replay: All jobs finished
18505 Main task has work to do, go find it
18506 got dsc Stop(1)
18507 stop 1
18508 Mar 17 22:23:09.658 [8820] Got stop action so:false kr:false
18509 INFO request completed, latency_us: 298, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 716af85c-08ca-4166-93ef-4c54971abac1, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
18510 Mar 17 22:23:09.659 INFO request completed, latency_us: 133, response_code: 200, uri: /state/cid/1, method: GET{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:23:09.659489847Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
18511 , req_id: 15015336-5d3a-4108-945e-c1e113720dbe, remote_addr: 127.0.0.1:58317{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:23:09.659582487Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
18512 , local_addr: 127.0.0.1:9998{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:23:09.659612284Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
18513 
185142025-03-17T22:23:09.659ZINFOcrucible: client task is exiting = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
18515 [8820] Exited with: ExitStatus(unix_wait_status(9))
18516 [8820][1] reports Exit
18517 Mar 17 22:23:13.661 INFO request completed, latency_us: 190, response_code: 200, uri: /state/cid/1, method: GET, req_id: 15dd8d19-b1e9-4cac-b5c9-0be207c9243f, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
18518 001/300 Read block 4635 len 12288
18519 002/300 Read block 4321 len 24576
18520 003/300 Write block 2381 len 20480 data: 4 5 5 5 5
18521 004/300 Write block 397 len 8192 data: 3 4
18522 005/300 Write block 4713 len 12288 data: 5 7 7
18523 006/300 Write block 960 len 20480 data: 3 2 2 2 3
18524 007/300 Flush
18525 008/300 Write block 1335 len 8192 data: 4 4
18526 009/300 Read block 629 len 40960
18527 010/300 Read block 1358 len 40960
18528 011/300 Read block 2470 len 4096
18529 012/300 Write block 1069 len 16384 data: 5 4 4 4
18530 013/300 Write block 3717 len 8192 data: 4 4
18531 014/300 Read block 2074 len 4096
18532 015/300 Read block 346 len 40960
18533 016/300 Read block 3889 len 8192
18534 017/300 Write block 1582 len 8192 data: 3 5
18535 018/300 Read block 1275 len 24576
18536 019/300 Read block 3284 len 24576
18537 020/300 Read block 3419 len 36864
18538 021/300 Flush
18539 022/300 Write block 201 len 4096 data: 4
18540 023/300 Read block 4098 len 36864
18541 024/300 Read block 3465 len 40960
18542 025/300 Write block 4358 len 40960 data: 4 3 4 4 4 3 3 2 3 3
18543 026/300 Read block 2997 len 28672
18544 027/300 Read block 4455 len 36864
18545 028/300 Read block 577 len 8192
18546 029/300 Read block 4429 len 4096
18547 030/300 Read block 3937 len 4096
18548 031/300 Flush
18549 032/300 Write block 4023 len 8192 data: 3 3
18550 033/300 Read block 1307 len 36864
18551 034/300 Write block 3418 len 36864 data: 3 3 3 3 3 3 2 3 4
18552 035/300 Read block 2812 len 40960
18553 036/300 Write block 4580 len 20480 data: 3 3 3 4 4
18554 037/300 Read block 2207 len 24576
18555 038/300 Read block 1698 len 8192
18556 039/300 Write block 1425 len 20480 data: 5 3 4 3 2
18557 040/300 Write block 4116 len 24576 data: 3 4 4 4 3 3
18558 041/300 Read block 1131 len 36864
18559 042/300 Write block 3682 len 8192 data: 5 6
18560 043/300 Read block 2533 len 28672
18561 044/300 Flush
18562 045/300 Read block 37 len 16384
18563 046/300 Read block 1741 len 4096
18564 047/300 Read block 2267 len 36864
18565 048/300 Write block 4197 len 12288 data: 7 6 5
18566 049/300 Write block 2482 len 12288 data: 3 3 3
18567 050/300 Read block 3951 len 20480
18568 051/300 Write block 1347 len 8192 data: 4 4
18569 052/300 Write block 123 len 8192 data: 6 5
18570 053/300 Read block 1060 len 8192
18571 054/300 Write block 1401 len 8192 data: 3 3
18572 055/300 Read block 4699 len 36864
18573 056/300 Flush
18574 057/300 Read block 4747 len 28672
18575 058/300 Read block 4469 len 12288
18576 059/300 Read block 851 len 8192
18577 060/300 Read block 4107 len 8192
18578 061/300 Read block 4534 len 24576
18579 062/300 Write block 481 len 20480 data: 3 3 4 4 3
18580 063/300 Flush
18581 064/300 Write block 4116 len 8192 data: 4 5
18582 065/300 Write block 2258 len 24576 data: 5 5 5 6 6 4
18583 066/300 Flush
18584 067/300 Read block 244 len 4096
18585 068/300 Read block 961 len 32768
18586 069/300 Read block 3739 len 24576
18587 070/300 Read block 4443 len 36864
18588 071/300 Write block 214 len 4096 data: 3
18589 072/300 Write block 2420 len 16384 data: 9 6 6 5
18590 073/300 Write block 361 len 12288 data: 2 3 3
18591 074/300 Read block 549 len 32768
18592 075/300 Read block 4018 len 24576
18593 076/300 Flush
18594 077/300 Read block 621 len 20480
18595 078/300 Write block 1608 len 36864 data: 4 5 5 3 3 3 4 4 4
18596 079/300 Write block 967 len 12288 data: 3 3 3
18597 080/300 Read block 3914 len 36864
18598 081/300 Write block 1894 len 24576 data: 3 4 4 4 3 3
18599 082/300 Read block 4217 len 28672
18600 083/300 Write block 3530 len 32768 data: 7 8 6 6 6 6 7 7
18601 084/300 Read block 1330 len 4096
18602 085/300 Write block 4379 len 36864 data: 4 4 4 4 5 5 5 4 4
18603 086/300 Write block 1724 len 32768 data: 5 5 6 4 5 4 4 5
18604 087/300 Read block 3046 len 8192
18605 088/300 Read block 3625 len 4096
18606 089/300 Write block 2202 len 20480 data: 2 4 3 3 3
18607 090/300 Write block 3144 len 32768 data: 2 2 2 4 4 3 3 4
18608 091/300 Write block 4902 len 12288 data: 5 7 7
18609 092/300 Write block 4769 len 16384 data: 4 4 3 4
18610 093/300 Read block 2931 len 16384
18611 094/300 Write block 3648 len 28672 data: 3 3 3 3 4 6 6
18612 095/300 Write block 2844 len 8192 data: 4 4
18613 096/300 Read block 2335 len 8192
18614 097/300 Write block 2037 len 36864 data: 3 3 3 4 4 3 2 2 4
18615 098/300 Write block 3416 len 24576 data: 2 2 4 4 4 4
18616 099/300 Write block 4589 len 40960 data: 2 2 4 3 2 2 3 3 3 3
18617 100/300 Write block 2682 len 20480 data: 4 4 3 4 3
18618 101/300 Read block 3360 len 36864
18619 102/300 Write block 40 len 36864 data: 4 4 4 4 5 7 5 5 4
18620 103/300 Flush
18621 104/300 Flush
18622 105/300 Write block 1435 len 8192 data: 3 3
18623 106/300 Write block 1195 len 4096 data: 3
18624 107/300 Read block 2679 len 16384
18625 108/300 Read block 4466 len 28672
18626 109/300 Read block 816 len 20480
18627 110/300 Read block 543 len 20480
18628 111/300 Read block 4588 len 20480
18629 112/300 Read block 2835 len 24576
18630 113/300 Read block 3909 len 36864
18631 114/300 Flush
18632 115/300 Read block 3223 len 4096
18633 116/300 Read block 2488 len 32768
18634 117/300 Write block 914 len 28672 data: 2 3 3 3 4 4 4
18635 118/300 Write block 4662 len 40960 data: 5 5 4 3 4 4 3 4 4 4
18636 119/300 Write block 1855 len 36864 data: 2 3 3 3 3 3 3 3 3
18637 120/300 Read block 1795 len 28672
18638 121/300 Write block 2513 len 12288 data: 3 3 4
18639 122/300 Read block 1190 len 40960
18640 123/300 Flush
18641 124/300 Read block 3167 len 36864
18642 125/300 Read block 2682 len 40960
18643 126/300 Flush
18644 127/300 Read block 989 len 36864
18645 128/300 Read block 2411 len 16384
18646 129/300 Flush
18647 130/300 Flush
18648 131/300 Read block 3900 len 12288
18649 132/300 Write block 492 len 12288 data: 3 4 5
18650 133/300 Read block 1956 len 28672
18651 134/300 Flush
18652 135/300 Read block 3093 len 16384
18653 136/300 Write block 4182 len 24576 data: 6 5 4 2 2 3
18654 137/300 Read block 4557 len 8192
18655 138/300 Read block 2003 len 40960
18656 139/300 Write block 613 len 16384 data: 4 4 4 4
18657 140/300 Read block 2303 len 28672
18658 141/300 Read block 1492 len 16384
18659 142/300 Read block 2632 len 4096
18660 143/300 Read block 318 len 16384
18661 144/300 Flush
18662 145/300 Write block 918 len 24576 data: 5 5 5 4 5 7
18663 146/300 Read block 1698 len 28672
18664 147/300 Read block 1495 len 24576
18665 148/300 Read block 4842 len 16384
18666 149/300 Write block 1433 len 40960 data: 2 3 4 4 3 3 4 3 3 3
18667 150/300 Read block 2356 len 4096
18668 151/300 Write block 2207 len 12288 data: 3 3 3
18669 152/300 Read block 4229 len 24576
18670 153/300 Write block 335 len 40960 data: 2 3 2 2 2 2 4 4 4 3
18671 154/300 Read block 3040 len 20480
18672 155/300 Read block 2048 len 24576
18673 156/300 Flush
18674 157/300 Write block 4375 len 32768 data: 4 6 4 5 5 5 5 5
18675 158/300 Write block 3571 len 24576 data: 2 2 2 3 5 5
18676 159/300 Read block 816 len 8192
18677 160/300 Read block 0 len 20480
18678 161/300 Read block 4677 len 32768
18679 162/300 Flush
18680 163/300 Read block 1279 len 8192
18681 164/300 Read block 4561 len 24576
18682 165/300 Read block 1175 len 40960
18683 166/300 Write block 1233 len 40960 data: 4 5 4 4 4 3 4 4 4 4
18684 167/300 Read block 3827 len 20480
18685 168/300 Write block 477 len 4096 data: 4
18686 169/300 Write block 2169 len 36864 data: 4 5 4 3 3 2 2 2 2
18687 170/300 Write block 3474 len 32768 data: 4 4 3 3 2 3 3 3
18688 171/300 Write block 1901 len 24576 data: 3 3 4 3 4 5
18689 172/300 Write block 4085 len 12288 data: 5 2 2
18690 173/300 Write block 761 len 36864 data: 3 3 2 3 3 3 3 3 5
18691 174/300 Write block 4146 len 16384 data: 5 4 4 4
18692 175/300 Read block 2551 len 8192
18693 176/300 Flush
18694 177/300 Flush
18695 178/300 Read block 1535 len 8192
18696 179/300 Write block 2907 len 8192 data: 3 4
18697 180/300 Write block 5 len 4096 data: 5
18698 181/300 Write block 3597 len 36864 data: 3 3 3 4 4 4 2 2 3
18699 182/300 Write block 4726 len 36864 data: 2 2 3 3 3 3 3 4 4
18700 183/300 Read block 1640 len 12288
18701 184/300 Read block 1293 len 20480
18702 185/300 Read block 1975 len 4096
18703 186/300 Read block 3782 len 12288
18704 187/300 Write block 709 len 20480 data: 3 4 4 4 3
18705 188/300 Read block 1118 len 24576
18706 189/300 Read block 4246 len 4096
18707 190/300 Write block 823 len 24576 data: 4 3 2 2 2 2
18708 191/300 Flush
18709 192/300 Read block 3394 len 12288
18710 193/300 Read block 3883 len 20480
18711 194/300 Read block 3072 len 4096
18712 195/300 Read block 55 len 12288
18713 196/300 Read block 1142 len 32768
18714 197/300 Write block 4156 len 8192 data: 5 6
18715 198/300 Read block 4834 len 8192
18716 199/300 Write block 4586 len 28672 data: 2 2 2 3 3 5 4
18717 200/300 Read block 3967 len 16384
18718 201/300 Read block 2331 len 28672
18719 202/300 Read block 4983 len 32768
18720 203/300 Read block 1041 len 28672
18721 204/300 Flush
18722 205/300 Write block 657 len 36864 data: 3 3 4 5 5 4 3 3 2
18723 206/300 Write block 4181 len 20480 data: 6 7 6 5 3
18724 207/300 Write block 1922 len 8192 data: 6 6
18725 208/300 Read block 3136 len 12288
18726 209/300 Write block 335 len 4096 data: 3
18727 210/300 Write block 4345 len 40960 data: 4 4 4 3 2 3 3 3 3 3
18728 211/300 Write block 4645 len 12288 data: 5 6 6
18729 212/300 Read block 2247 len 12288
18730 213/300 Read block 4382 len 40960
18731 214/300 Write block 3360 len 12288 data: 3 3 3
18732 215/300 Read block 2804 len 36864
18733 216/300 Read block 327 len 4096
18734 217/300 Write block 2626 len 24576 data: 3 3 3 2 2 2
18735 218/300 Read block 4740 len 20480
18736 219/300 Read block 3485 len 28672
18737 220/300 Write block 518 len 24576 data: 2 3 4 4 4 3
18738 221/300 Write block 262 len 12288 data: 4 4 4
18739 222/300 Write block 861 len 4096 data: 3
18740 223/300 Write block 4847 len 16384 data: 4 3 4 5
18741 224/300 Write block 1937 len 4096 data: 3
18742 225/300 Write block 1442 len 8192 data: 4 2
18743 226/300 Write block 2228 len 20480 data: 6 6 6 6 5
18744 227/300 Write block 1626 len 40960 data: 3 2 2 2 2 2 3 4 4 6
18745 228/300 Read block 1391 len 40960
18746 229/300 Read block 1627 len 32768
18747 230/300 Read block 1864 len 4096
18748 231/300 Flush
18749 232/300 Read block 1251 len 8192
18750 233/300 Write block 2250 len 12288 data: 4 4 4
18751 234/300 Read block 1375 len 24576
18752 235/300 Read block 904 len 4096
18753 236/300 Read block 1501 len 4096
18754 237/300 Write block 2699 len 28672 data: 4 4 3 3 3 4 5
18755 238/300 Read block 2229 len 40960
18756 239/300 Read block 1962 len 4096
18757 240/300 Read block 4008 len 28672
18758 241/300 Read block 1211 len 8192
18759 242/300 Flush
18760 243/300 Read block 188 len 32768
18761 244/300 Read block 4916 len 32768
18762 245/300 Read block 944 len 28672
18763 246/300 Read block 2949 len 8192
18764 247/300 Read block 502 len 28672
18765 248/300 Write block 3040 len 4096 data: 6
18766 249/300 Read block 806 len 24576
18767 250/300 Read block 1803 len 12288
18768 251/300 Read block 4640 len 16384
18769 252/300 Flush
18770 253/300 Read block 4025 len 4096
18771 254/300 Read block 844 len 16384
18772 255/300 Read block 600 len 8192
18773 256/300 Write block 1475 len 16384 data: 7 6 5 6
18774 257/300 Read block 3868 len 12288
18775 258/300 Read block 1511 len 32768
18776 259/300 Read block 4157 len 28672
18777 260/300 Read block 899 len 40960
18778 261/300 Read block 2784 len 4096
18779 262/300 Read block 3186 len 4096
18780 263/300 Read block 2579 len 8192
18781 264/300 Read block 4602 len 28672
18782 265/300 Write block 1342 len 12288 data: 3 2 3
18783 266/300 Flush
18784 267/300 Read block 677 len 24576
18785 268/300 Read block 1675 len 28672
18786 269/300 Write block 1884 len 8192 data: 4 4
18787 270/300 Read block 1562 len 12288
18788 271/300 Read block 4498 len 24576
18789 272/300 Read block 2609 len 8192
18790 273/300 Read block 4020 len 32768
18791 274/300 Write block 574 len 8192 data: 5 5
18792 275/300 Flush
18793 276/300 Read block 3553 len 32768
18794 277/300 Write block 2620 len 32768 data: 3 3 3 3 3 3 4 4
18795 278/300 Read block 2693 len 24576
18796 279/300 Read block 2705 len 40960
18797 280/300 Read block 2911 len 8192
18798 281/300 Read block 1643 len 20480
18799 282/300 Read block 1271 len 16384
18800 283/300 Flush
18801 284/300 Write block 3232 len 40960 data: 4 3 3 2 2 2 2 2 2 2
18802 285/300 Read block 762 len 32768
18803 286/300 Write block 1131 len 24576 data: 3 4 5 3 3 4
18804 287/300 Read block 3901 len 24576
18805 288/300 Write block 1887 len 12288 data: 3 3 2
18806 289/300 Flush
18807 290/300 Write block 3372 len 28672 data: 4 4 4 4 4 3 3
18808 291/300 Write block 2025 len 24576 data: 3 3 3 3 2 2
18809 292/300 Write block 2695 len 8192 data: 3 4
18810 293/300 Read block 2358 len 4096
18811 294/300 Read block 927 len 20480
18812 295/300 Write block 662 len 24576 data: 5 4 4 3 2 2
18813 296/300 Flush
18814 297/300 Write block 1281 len 12288 data: 5 4 6
18815 298/300 Write block 3167 len 32768 data: 4 3 2 2 2 3 5 5
18816 299/300 Write block 838 len 12288 data: 3 3 3
18817 300/300 Write block 4288 len 36864 data: 3 3 3 3 2 2 2 2 3
18818 301/300 Read block 3251 len 24576
18819 Main task has work to do, go find it
18820 got dsc Start(1)
18821 start 1
18822 Mar 17 22:23:13.900 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/1, method: POST, req_id: 7e521626-a9e6-4d88-a44f-31ea6ca274b0, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
18823 Replay: started 1, returned:Ok(())
18824 ----------------------------------------------------------------
18825 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18826 States: Active Offline Active
18827 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18828 5227 Acked Read 3 Done Sent Done false
18829 5228 Acked Read 6 Done Sent Done false
18830 5229 Acked Write 5 Done Sent Done false
18831 5230 Acked Write 2 Done Sent Done false
18832 5231 Acked Write 3 Done Sent Done false
18833 5232 Acked Write 5 Done Sent Done false
18834 5233 Acked Flush 0 Done Sent Done false
18835 5234 Acked Write 2 Done Sent Done false
18836 5235 Acked Read 10 Done Sent Done false
18837 5236 Acked Read 10 Done Sent Done false
18838 5237 Acked Read 1 Done Sent Done false
18839 5238 Acked Write 4 Done Sent Done false
18840 5239 Acked Write 2 Done Sent Done false
18841 5240 Acked Read 1 Done Sent Done false
18842 5241 Acked Read 10 Done Sent Done false
18843 5242 Acked Read 2 Done Sent Done false
18844 5243 Acked Write 2 Done Sent Done false
18845 5244 Acked Read 6 Done Sent Done false
18846 5245 Acked Read 6 Done Sent Done false
18847 5246 Acked Read 9 Done Sent Done false
18848 5247 Acked Flush 0 Done Sent Done false
18849 5248 Acked Write 1 Done Sent Done false
18850 5249 Acked Read 9 Done Sent Done false
18851 5250 Acked Read 10 Done Sent Done false
18852 5251 Acked Write 10 Done Sent Done false
18853 5252 Acked Read 7 Done Sent Done false
18854 5253 Acked Read 9 Done Sent Done false
18855 5254 Acked Read 2 Done Sent Done false
18856 5255 Acked Read 1 Done Sent Done false
18857 5256 Acked Read 1 Done Sent Done false
18858 5257 Acked Flush 0 Done Sent Done false
18859 5258 Acked Write 2 Done Sent Done false
18860 5259 Acked Read 9 Done Sent Done false
18861 5260 Acked Write 9 Done Sent Done false
18862 5261 Acked Read 10 Done Sent Done false
18863 5262 Acked Write 5 Done Sent Done false
18864 5263 Acked Read 6 Done Sent Done false
18865 5264 Acked Read 2 Done Sent Done false
18866 5265 Acked Write 5 Done Sent Done false
18867 5266 Acked Write 6 Done Sent Done false
18868 5267 Acked Read 9 Done Sent Done false
18869 5268 Acked Write 2 Done Sent Done false
18870 5269 Acked Read 7 Done Sent Done false
18871 5270 Acked Flush 0 Done Sent Done false
18872 5271 Acked Read 4 Done Sent Done false
18873 5272 Acked Read 1 Done Sent Done false
18874 5273 Acked Read 9 Done Sent Done false
18875 5274 Acked Write 3 Done Sent Done false
18876 5275 Acked Write 3 Done Sent Done false
18877 5276 Acked Read 5 Done Sent Done false
18878 5277 Acked Write 2 Done Sent Done false
18879 5278 Acked Write 2 Done Sent Done false
18880 5279 Acked Read 2 Done Sent Done false
18881 5280 Acked Write 2 Done Sent Done false
18882 5281 Acked Read 9 Done Sent Done false
18883 5282 Acked Flush 0 Done Sent Done false
18884 5283 Acked Read 7 Done Sent Done false
18885 5284 Acked Read 3 Done Sent Done false
18886 5285 Acked Read 2 Done Sent Done false
18887 5286 Acked Read 2 Done Sent Done false
18888 5287 Acked Read 6 Done Sent Done false
18889 5288 Acked Write 5 Done Sent Done false
18890 5289 Acked Flush 0 Done Sent Done false
18891 5290 Acked Write 2 Done Sent Done false
18892 5291 Acked Write 6 Done Sent Done false
18893 5292 Acked Flush 0 Done Sent Done false
18894 5293 Acked Read 1 Done Sent Done false
18895 5294 Acked Read 8 Done Sent Done false
18896 5295 Acked Read 6 Done Sent Done false
18897 5296 Acked Read 9 Done Sent Done false
18898 5297 Acked Write 1 Done Sent Done false
18899 5298 Acked Write 4 Done Sent Done false
18900 5299 Acked Write 3 Done Sent Done false
18901 5300 Acked Read 8 Done Sent Done false
18902 5301 Acked Read 6 Done Sent Done false
18903 5302 Acked Flush 0 Done Sent Done false
18904 5303 Acked Read 5 Done Sent Done false
18905 5304 Acked Write 9 Done Sent Done false
18906 5305 Acked Write 3 Done Sent Done false
18907 5306 Acked Read 9 Done Sent Done false
18908 5307 Acked Write 6 Done Sent Done false
18909 5308 Acked Read 7 Done Sent Done false
18910 5309 Acked Write 8 Done Sent Done false
18911 5310 Acked Read 1 Done Sent Done false
18912 5311 Acked Write 9 Done Sent Done false
18913 5312 Acked Write 8 Done Sent Done false
18914 5313 Acked Read 2 Done Sent Done false
18915 5314 Acked Read 1 Done Sent Done false
18916 5315 Acked Write 5 Done Sent Done false
18917 5316 Acked Write 8 Done Sent Done false
18918 5317 Acked Write 3 Done Sent Done false
18919 5318 Acked Write 4 Done Sent Done false
18920 5319 Acked Read 4 Done Sent Done false
18921 5320 Acked Write 7 Done Sent Done false
18922 5321 Acked Write 2 Done Sent Done false
18923 5322 Acked Read 2 Done Sent Done false
18924 5323 Acked Write 9 Done Sent Done false
18925 5324 Acked Write 6 Done Sent Done false
18926 5325 Acked Write 10 Done Sent Done false
18927 5326 Acked Write 5 Done Sent Done false
18928 5327 Acked Read 9 Done Sent Done false
18929 5328 Acked Write 9 Done Sent Done false
18930 5329 Acked Flush 0 Done Sent Done false
18931 5330 Acked Flush 0 Done Sent Done false
18932 5331 Acked Write 2 Done Sent Done false
18933 5332 Acked Write 1 Done Sent Done false
18934 5333 Acked Read 4 Done Sent Done false
18935 5334 Acked Read 7 Done Sent Done false
18936 5335 Acked Read 5 Done Sent Done false
18937 5336 Acked Read 5 Done Sent Done false
18938 5337 Acked Read 5 Done Sent Done false
18939 5338 Acked Read 6 Done Sent Done false
18940 5339 Acked Read 9 Done Sent Done false
18941 5340 Acked Flush 0 Done Sent Done false
18942 5341 Acked Read 1 Done Sent Done false
18943 5342 Acked Read 8 Done Sent Done false
18944 5343 Acked Write 7 Done Sent Done false
18945 5344 Acked Write 10 Done Sent Done false
18946 5345 Acked Write 9 Done Sent Done false
18947 5346 Acked Read 7 Done Sent Done false
18948 5347 Acked Write 3 Done Sent Done false
18949 5348 Acked Read 10 Done Sent Done false
18950 5349 Acked Flush 0 Done Sent Done false
18951 5350 Acked Read 9 Done Sent Done false
18952 5351 Acked Read 10 Done Sent Done false
18953 5352 Acked Flush 0 Done Sent Done false
18954 5353 Acked Read 9 Done Sent Done false
18955 5354 Acked Read 4 Done Sent Done false
18956 5355 Acked Flush 0 Done Sent Done false
18957 5356 Acked Flush 0 Done Sent Done false
18958 5357 Acked Read 3 Done Sent Done false
18959 5358 Acked Write 3 Done Sent Done false
18960 5359 Acked Read 7 Done Sent Done false
18961 5360 Acked Flush 0 Done Sent Done false
18962 5361 Acked Read 4 Done Sent Done false
18963 5362 Acked Write 6 Done Sent Done false
18964 5363 Acked Read 2 Done Sent Done false
18965 5364 Acked Read 10 Done Sent Done false
18966 5365 Acked Write 4 Done Sent Done false
18967 5366 Acked Read 7 Done Sent Done false
18968 5367 Acked Read 4 Done Sent Done false
18969 5368 Acked Read 1 Done Sent Done false
18970 5369 Acked Read 4 Done Sent Done false
18971 5370 Acked Flush 0 Done Sent Done false
18972 5371 Acked Write 6 Done Sent Done false
18973 5372 Acked Read 7 Done Sent Done false
18974 5373 Acked Read 6 Done Sent Done false
18975 5374 Acked Read 4 Done Sent Done false
18976 5375 Acked Write 10 Done Sent Done false
18977 5376 Acked Read 1 Done Sent Done false
18978 5377 Acked Write 3 Done Sent Done false
18979 5378 Acked Read 6 Done Sent Done false
18980 5379 Acked Write 10 Done Sent Done false
18981 5380 Acked Read 5 Done Sent Done false
18982 5381 Acked Read 6 Done Sent Done false
18983 5382 Acked Flush 0 Done Sent Done false
18984 5383 Acked Write 8 Done Sent Done false
18985 5384 Acked Write 6 Done Sent Done false
18986 5385 Acked Read 2 Done Sent Done false
18987 5386 Acked Read 5 Done Sent Done false
18988 5387 Acked Read 8 Done Sent Done false
18989 5388 Acked Flush 0 Done Sent Done false
18990 5389 Acked Read 2 Done Sent Done false
18991 5390 Acked Read 6 Done Sent Done false
18992 5391 Acked Read 10 Done Sent Done false
18993 5392 Acked Write 10 Done Sent Done false
18994 5393 Acked Read 5 Done Sent Done false
18995 5394 Acked Write 1 Done Sent Done false
18996 5395 Acked Write 9 Done Sent Done false
18997 5396 Acked Write 8 Done Sent Done false
18998 5397 Acked Write 6 Done Sent Done false
18999 5398 Acked Write 3 Done Sent Done false
19000 5399 Acked Write 9 Done Sent Done false
19001 5400 Acked Write 4 Done Sent Done false
19002 5401 Acked Read 2 Done Sent Done false
19003 5402 Acked Flush 0 Done Sent Done false
19004 5403 Acked Flush 0 Done Sent Done false
19005 5404 Acked Read 2 Done Sent Done false
19006 5405 Acked Write 2 Done Sent Done false
19007 5406 Acked Write 1 Done Sent Done false
19008 5407 Acked Write 9 Done Sent Done false
19009 5408 Acked Write 9 Done Sent Done false
19010 5409 Acked Read 3 Done Sent Done false
19011 5410 Acked Read 5 Done Sent Done false
19012 5411 Acked Read 1 Done Sent Done false
19013 5412 Acked Read 3 Done Sent Done false
19014 5413 Acked Write 5 Done Sent Done false
19015 5414 Acked Read 6 Done Sent Done false
19016 5415 Acked Read 1 Done Sent Done false
19017 5416 Acked Write 6 Done Sent Done false
19018 5417 Acked Flush 0 Done Sent Done false
19019 5418 Acked Read 3 Done Sent Done false
19020 5419 Acked Read 5 Done Sent Done false
19021 5420 Acked Read 1 Done Sent Done false
19022 5421 Acked Read 3 Done Sent Done false
19023 5422 Acked Read 8 Done Sent Done false
19024 5423 Acked Write 2 Done Sent Done false
19025 5424 Acked Read 2 Done Sent Done false
19026 5425 Acked Write 7 Done Sent Done false
19027 5426 Acked Read 4 Done Sent Done false
19028 5427 Acked Read 7 Done Sent Done false
19029 5428 Acked Read 8 Done Sent Done false
19030 5429 Acked Read 7 Done Sent Done false
19031 5430 Acked Flush 0 Done Sent Done false
19032 5431 Acked Write 9 Done Sent Done false
19033 5432 Acked Write 5 Done Sent Done false
19034 5433 Acked Write 2 Done Sent Done false
19035 5434 Acked Read 3 Done Sent Done false
19036 5435 Acked Write 1 Done Sent Done false
19037 5436 Acked Write 10 Done Sent Done false
19038 5437 Acked Write 3 Done Sent Done false
19039 5438 Acked Read 3 Done Sent Done false
19040 5439 Acked Read 10 Done Sent Done false
19041 5440 Acked Write 3 Done Sent Done false
19042 5441 Acked Read 9 Done Sent Done false
19043 5442 Acked Read 1 Done Sent Done false
19044 5443 Acked Write 6 Done Sent Done false
19045 5444 Acked Read 5 Done Sent Done false
19046 5445 Acked Read 7 Done Sent Done false
19047 5446 Acked Write 6 Done Sent Done false
19048 5447 Acked Write 3 Done Sent Done false
19049 5448 Acked Write 1 Done Sent Done false
19050 5449 Acked Write 4 Done Sent Done false
19051 5450 Acked Write 1 Done Sent Done false
19052 5451 Acked Write 2 Done Sent Done false
19053 5452 Acked Write 5 Done Sent Done false
19054 5453 Acked Write 10 Done Sent Done false
19055 5454 Acked Read 10 Done Sent Done false
19056 5455 Acked Read 8 Done Sent Done false
19057 5456 Acked Read 1 Done Sent Done false
19058 5457 Acked Flush 0 Done Sent Done false
19059 5458 Acked Read 2 Done Sent Done false
19060 5459 Acked Write 3 Done Sent Done false
19061 5460 Acked Read 6 Done Sent Done false
19062 5461 Acked Read 1 Done Sent Done false
19063 5462 Acked Read 1 Done Sent Done false
19064 5463 Acked Write 7 Done Sent Done false
19065 5464 Acked Read 10 Done Sent Done false
19066 5465 Acked Read 1 Done Sent Done false
19067 5466 Acked Read 7 Done Sent Done false
19068 5467 Acked Read 2 Done Sent Done false
19069 5468 Acked Flush 0 Done Sent Done false
19070 5469 Acked Read 8 Done Sent Done false
19071 5470 Acked Read 8 Done Sent Done false
19072 5471 Acked Read 7 Done Sent Done false
19073 5472 Acked Read 2 Done Sent Done false
19074 5473 Acked Read 7 Done Sent Done false
19075 5474 Acked Write 1 Done Sent Done false
19076 5475 Acked Read 6 Done Sent Done false
19077 5476 Acked Read 3 Done Sent Done false
19078 5477 Acked Read 4 Done Sent Done false
19079 5478 Acked Flush 0 Done Sent Done false
19080 5479 Acked Read 1 Done Sent Done false
19081 5480 Acked Read 4 Done Sent Done false
19082 5481 Acked Read 2 Done Sent Done false
19083 5482 Acked Write 4 Done Sent Done false
19084 5483 Acked Read 3 Done Sent Done false
19085 5484 Acked Read 8 Done Sent Done false
19086 5485 Acked Read 7 Done Sent Done false
19087 5486 Acked Read 10 Done Sent Done false
19088 5487 Acked Read 1 Done Sent Done false
19089 5488 Acked Read 1 Done Sent Done false
19090 5489 Acked Read 2 Done Sent Done false
19091 5490 Acked Read 7 Done Sent Done false
19092 5491 Acked Write 3 Done Sent Done false
19093 5492 Acked Flush 0 Done Sent Done false
19094 5493 Acked Read 6 Done Sent Done false
19095 5494 Acked Read 7 Done Sent Done false
19096 5495 Acked Write 2 Done Sent Done false
19097 5496 Acked Read 3 Done Sent Done false
19098 5497 Acked Read 6 Done Sent Done false
19099 5498 Acked Read 2 Done Sent Done false
19100 5499 Acked Read 8 Done Sent Done false
19101 5500 Acked Write 2 Done Sent Done false
19102 5501 Acked Flush 0 Done Sent Done false
19103 5502 Acked Read 8 Done Sent Done false
19104 5503 Acked Write 8 Done Sent Done false
19105 5504 Acked Read 6 Done Sent Done false
19106 5505 Acked Read 10 Done Sent Done false
19107 5506 Acked Read 2 Done Sent Done false
19108 5507 Acked Read 5 Done Sent Done false
19109 5508 Acked Read 4 Done Sent Done false
19110 5509 Acked Flush 0 Done Sent Done false
19111 5510 Acked Write 10 Done Sent Done false
19112 5511 Acked Read 8 Done Sent Done false
19113 5512 Acked Write 6 Done Sent Done false
19114 5513 Acked Read 6 Done Sent Done false
19115 5514 Acked Write 3 Done Sent Done false
19116 5515 Acked Flush 0 Done Sent Done false
19117 5516 Acked Write 7 Done Sent Done false
19118 5517 Acked Write 6 Done Sent Done false
19119 5518 Acked Write 2 Done Sent Done false
19120 5519 Acked Read 1 Done Sent Done false
19121 5520 Acked Read 5 Done Sent Done false
19122 5521 Acked Write 6 Done Sent Done false
19123 5522 Acked Flush 0 Done Sent Done false
19124 5523 Acked Write 3 Done Sent Done false
19125 5524 Acked Write 8 Done Sent Done false
19126 5525 Acked Write 3 Done Sent Done false
19127 5526 Acked Write 9 Done Sent Done false
19128 5527 Acked Read 6 Done Sent Done false
19129 STATES DS:0 DS:1 DS:2 TOTAL
19130 Sent 0 301 0 301
19131 Done 301 0 301 602
19132 Skipped 0 0 0 0
19133 Error 0 0 0 0
19134 Last Flush: Some(JobId(5522)) Some(JobId(5226)) Some(JobId(5522))
19135 Downstairs last five completed: 5226 5225 5224 5223 5222
19136 Upstairs last five completed: 5527 5526 5525 5524 5523
19137 CLIENT: Up:0 ds:301 act:2
191382025-03-17T22:23:14.403ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
191392025-03-17T22:23:14.978ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
19140 [8820] got start action
19141 [8820] I am going to restart
19142 Starting downstairs at port 8820
19143 Make output file at "/tmp/dsc/downstairs-8820.txt"
19144 [8820][1] reports Starting
19145 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1226)
19146 [8820][1] reports Running
191472025-03-17T22:23:16.390ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
191482025-03-17T22:23:16.981ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
19149 ----------------------------------------------------------------
19150 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19151 States: Active Offline Active
19152 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19153 5227 Acked Read 3 Done Sent Done false
19154 5228 Acked Read 6 Done Sent Done false
19155 5229 Acked Write 5 Done Sent Done false
19156 5230 Acked Write 2 Done Sent Done false
19157 5231 Acked Write 3 Done Sent Done false
19158 5232 Acked Write 5 Done Sent Done false
19159 5233 Acked Flush 0 Done Sent Done false
19160 5234 Acked Write 2 Done Sent Done false
19161 5235 Acked Read 10 Done Sent Done false
19162 5236 Acked Read 10 Done Sent Done false
19163 5237 Acked Read 1 Done Sent Done false
19164 5238 Acked Write 4 Done Sent Done false
19165 5239 Acked Write 2 Done Sent Done false
19166 5240 Acked Read 1 Done Sent Done false
19167 5241 Acked Read 10 Done Sent Done false
19168 5242 Acked Read 2 Done Sent Done false
19169 5243 Acked Write 2 Done Sent Done false
19170 5244 Acked Read 6 Done Sent Done false
19171 5245 Acked Read 6 Done Sent Done false
19172 5246 Acked Read 9 Done Sent Done false
19173 5247 Acked Flush 0 Done Sent Done false
19174 5248 Acked Write 1 Done Sent Done false
19175 5249 Acked Read 9 Done Sent Done false
19176 5250 Acked Read 10 Done Sent Done false
19177 5251 Acked Write 10 Done Sent Done false
19178 5252 Acked Read 7 Done Sent Done false
19179 5253 Acked Read 9 Done Sent Done false
19180 5254 Acked Read 2 Done Sent Done false
19181 5255 Acked Read 1 Done Sent Done false
19182 5256 Acked Read 1 Done Sent Done false
19183 5257 Acked Flush 0 Done Sent Done false
19184 5258 Acked Write 2 Done Sent Done false
19185 5259 Acked Read 9 Done Sent Done false
19186 5260 Acked Write 9 Done Sent Done false
19187 5261 Acked Read 10 Done Sent Done false
19188 5262 Acked Write 5 Done Sent Done false
19189 5263 Acked Read 6 Done Sent Done false
19190 5264 Acked Read 2 Done Sent Done false
19191 5265 Acked Write 5 Done Sent Done false
19192 5266 Acked Write 6 Done Sent Done false
19193 5267 Acked Read 9 Done Sent Done false
19194 5268 Acked Write 2 Done Sent Done false
19195 5269 Acked Read 7 Done Sent Done false
19196 5270 Acked Flush 0 Done Sent Done false
19197 5271 Acked Read 4 Done Sent Done false
19198 5272 Acked Read 1 Done Sent Done false
19199 5273 Acked Read 9 Done Sent Done false
19200 5274 Acked Write 3 Done Sent Done false
19201 5275 Acked Write 3 Done Sent Done false
19202 5276 Acked Read 5 Done Sent Done false
19203 5277 Acked Write 2 Done Sent Done false
19204 5278 Acked Write 2 Done Sent Done false
19205 5279 Acked Read 2 Done Sent Done false
19206 5280 Acked Write 2 Done Sent Done false
19207 5281 Acked Read 9 Done Sent Done false
19208 5282 Acked Flush 0 Done Sent Done false
19209 5283 Acked Read 7 Done Sent Done false
19210 5284 Acked Read 3 Done Sent Done false
19211 5285 Acked Read 2 Done Sent Done false
19212 5286 Acked Read 2 Done Sent Done false
19213 5287 Acked Read 6 Done Sent Done false
19214 5288 Acked Write 5 Done Sent Done false
19215 5289 Acked Flush 0 Done Sent Done false
19216 5290 Acked Write 2 Done Sent Done false
19217 5291 Acked Write 6 Done Sent Done false
19218 5292 Acked Flush 0 Done Sent Done false
19219 5293 Acked Read 1 Done Sent Done false
19220 5294 Acked Read 8 Done Sent Done false
19221 5295 Acked Read 6 Done Sent Done false
19222 5296 Acked Read 9 Done Sent Done false
19223 5297 Acked Write 1 Done Sent Done false
19224 5298 Acked Write 4 Done Sent Done false
19225 5299 Acked Write 3 Done Sent Done false
19226 5300 Acked Read 8 Done Sent Done false
19227 5301 Acked Read 6 Done Sent Done false
19228 5302 Acked Flush 0 Done Sent Done false
19229 5303 Acked Read 5 Done Sent Done false
19230 5304 Acked Write 9 Done Sent Done false
19231 5305 Acked Write 3 Done Sent Done false
19232 5306 Acked Read 9 Done Sent Done false
19233 5307 Acked Write 6 Done Sent Done false
19234 5308 Acked Read 7 Done Sent Done false
19235 5309 Acked Write 8 Done Sent Done false
19236 5310 Acked Read 1 Done Sent Done false
19237 5311 Acked Write 9 Done Sent Done false
19238 5312 Acked Write 8 Done Sent Done false
19239 5313 Acked Read 2 Done Sent Done false
19240 5314 Acked Read 1 Done Sent Done false
19241 5315 Acked Write 5 Done Sent Done false
19242 5316 Acked Write 8 Done Sent Done false
19243 5317 Acked Write 3 Done Sent Done false
19244 5318 Acked Write 4 Done Sent Done false
19245 5319 Acked Read 4 Done Sent Done false
19246 5320 Acked Write 7 Done Sent Done false
19247 5321 Acked Write 2 Done Sent Done false
19248 5322 Acked Read 2 Done Sent Done false
19249 5323 Acked Write 9 Done Sent Done false
19250 5324 Acked Write 6 Done Sent Done false
19251 5325 Acked Write 10 Done Sent Done false
19252 5326 Acked Write 5 Done Sent Done false
19253 5327 Acked Read 9 Done Sent Done false
19254 5328 Acked Write 9 Done Sent Done false
19255 5329 Acked Flush 0 Done Sent Done false
19256 5330 Acked Flush 0 Done Sent Done false
19257 5331 Acked Write 2 Done Sent Done false
19258 5332 Acked Write 1 Done Sent Done false
19259 5333 Acked Read 4 Done Sent Done false
19260 5334 Acked Read 7 Done Sent Done false
19261 5335 Acked Read 5 Done Sent Done false
19262 5336 Acked Read 5 Done Sent Done false
19263 5337 Acked Read 5 Done Sent Done false
19264 5338 Acked Read 6 Done Sent Done false
19265 5339 Acked Read 9 Done Sent Done false
19266 5340 Acked Flush 0 Done Sent Done false
19267 5341 Acked Read 1 Done Sent Done false
19268 5342 Acked Read 8 Done Sent Done false
19269 5343 Acked Write 7 Done Sent Done false
19270 5344 Acked Write 10 Done Sent Done false
19271 5345 Acked Write 9 Done Sent Done false
19272 5346 Acked Read 7 Done Sent Done false
19273 5347 Acked Write 3 Done Sent Done false
19274 5348 Acked Read 10 Done Sent Done false
19275 5349 Acked Flush 0 Done Sent Done false
19276 5350 Acked Read 9 Done Sent Done false
19277 5351 Acked Read 10 Done Sent Done false
19278 5352 Acked Flush 0 Done Sent Done false
19279 5353 Acked Read 9 Done Sent Done false
19280 5354 Acked Read 4 Done Sent Done false
19281 5355 Acked Flush 0 Done Sent Done false
19282 5356 Acked Flush 0 Done Sent Done false
19283 5357 Acked Read 3 Done Sent Done false
19284 5358 Acked Write 3 Done Sent Done false
19285 5359 Acked Read 7 Done Sent Done false
19286 5360 Acked Flush 0 Done Sent Done false
19287 5361 Acked Read 4 Done Sent Done false
19288 5362 Acked Write 6 Done Sent Done false
19289 5363 Acked Read 2 Done Sent Done false
19290 5364 Acked Read 10 Done Sent Done false
19291 5365 Acked Write 4 Done Sent Done false
19292 5366 Acked Read 7 Done Sent Done false
19293 5367 Acked Read 4 Done Sent Done false
19294 5368 Acked Read 1 Done Sent Done false
19295 5369 Acked Read 4 Done Sent Done false
19296 5370 Acked Flush 0 Done Sent Done false
19297 5371 Acked Write 6 Done Sent Done false
19298 5372 Acked Read 7 Done Sent Done false
19299 5373 Acked Read 6 Done Sent Done false
19300 5374 Acked Read 4 Done Sent Done false
19301 5375 Acked Write 10 Done Sent Done false
19302 5376 Acked Read 1 Done Sent Done false
19303 5377 Acked Write 3 Done Sent Done false
19304 5378 Acked Read 6 Done Sent Done false
19305 5379 Acked Write 10 Done Sent Done false
19306 5380 Acked Read 5 Done Sent Done false
19307 5381 Acked Read 6 Done Sent Done false
19308 5382 Acked Flush 0 Done Sent Done false
19309 5383 Acked Write 8 Done Sent Done false
19310 5384 Acked Write 6 Done Sent Done false
19311 5385 Acked Read 2 Done Sent Done false
19312 5386 Acked Read 5 Done Sent Done false
19313 5387 Acked Read 8 Done Sent Done false
19314 5388 Acked Flush 0 Done Sent Done false
19315 5389 Acked Read 2 Done Sent Done false
19316 5390 Acked Read 6 Done Sent Done false
19317 5391 Acked Read 10 Done Sent Done false
19318 5392 Acked Write 10 Done Sent Done false
19319 5393 Acked Read 5 Done Sent Done false
19320 5394 Acked Write 1 Done Sent Done false
19321 5395 Acked Write 9 Done Sent Done false
19322 5396 Acked Write 8 Done Sent Done false
19323 5397 Acked Write 6 Done Sent Done false
19324 5398 Acked Write 3 Done Sent Done false
19325 5399 Acked Write 9 Done Sent Done false
19326 5400 Acked Write 4 Done Sent Done false
19327 5401 Acked Read 2 Done Sent Done false
19328 5402 Acked Flush 0 Done Sent Done false
19329 5403 Acked Flush 0 Done Sent Done false
19330 5404 Acked Read 2 Done Sent Done false
19331 5405 Acked Write 2 Done Sent Done false
19332 5406 Acked Write 1 Done Sent Done false
19333 5407 Acked Write 9 Done Sent Done false
19334 5408 Acked Write 9 Done Sent Done false
19335 5409 Acked Read 3 Done Sent Done false
19336 5410 Acked Read 5 Done Sent Done false
19337 5411 Acked Read 1 Done Sent Done false
19338 5412 Acked Read 3 Done Sent Done false
19339 5413 Acked Write 5 Done Sent Done false
19340 5414 Acked Read 6 Done Sent Done false
19341 5415 Acked Read 1 Done Sent Done false
19342 5416 Acked Write 6 Done Sent Done false
19343 5417 Acked Flush 0 Done Sent Done false
19344 5418 Acked Read 3 Done Sent Done false
19345 5419 Acked Read 5 Done Sent Done false
19346 5420 Acked Read 1 Done Sent Done false
19347 5421 Acked Read 3 Done Sent Done false
19348 5422 Acked Read 8 Done Sent Done false
19349 5423 Acked Write 2 Done Sent Done false
19350 5424 Acked Read 2 Done Sent Done false
19351 5425 Acked Write 7 Done Sent Done false
19352 5426 Acked Read 4 Done Sent Done false
19353 5427 Acked Read 7 Done Sent Done false
19354 5428 Acked Read 8 Done Sent Done false
19355 5429 Acked Read 7 Done Sent Done false
19356 5430 Acked Flush 0 Done Sent Done false
19357 5431 Acked Write 9 Done Sent Done false
19358 5432 Acked Write 5 Done Sent Done false
19359 5433 Acked Write 2 Done Sent Done false
19360 5434 Acked Read 3 Done Sent Done false
19361 5435 Acked Write 1 Done Sent Done false
19362 5436 Acked Write 10 Done Sent Done false
19363 5437 Acked Write 3 Done Sent Done false
19364 5438 Acked Read 3 Done Sent Done false
19365 5439 Acked Read 10 Done Sent Done false
19366 5440 Acked Write 3 Done Sent Done false
19367 5441 Acked Read 9 Done Sent Done false
19368 5442 Acked Read 1 Done Sent Done false
19369 5443 Acked Write 6 Done Sent Done false
19370 5444 Acked Read 5 Done Sent Done false
19371 5445 Acked Read 7 Done Sent Done false
19372 5446 Acked Write 6 Done Sent Done false
19373 5447 Acked Write 3 Done Sent Done false
19374 5448 Acked Write 1 Done Sent Done false
19375 5449 Acked Write 4 Done Sent Done false
19376 5450 Acked Write 1 Done Sent Done false
19377 5451 Acked Write 2 Done Sent Done false
19378 5452 Acked Write 5 Done Sent Done false
19379 5453 Acked Write 10 Done Sent Done false
19380 5454 Acked Read 10 Done Sent Done false
19381 5455 Acked Read 8 Done Sent Done false
19382 5456 Acked Read 1 Done Sent Done false
19383 5457 Acked Flush 0 Done Sent Done false
19384 5458 Acked Read 2 Done Sent Done false
19385 5459 Acked Write 3 Done Sent Done false
19386 5460 Acked Read 6 Done Sent Done false
19387 5461 Acked Read 1 Done Sent Done false
19388 5462 Acked Read 1 Done Sent Done false
19389 5463 Acked Write 7 Done Sent Done false
19390 5464 Acked Read 10 Done Sent Done false
19391 5465 Acked Read 1 Done Sent Done false
19392 5466 Acked Read 7 Done Sent Done false
19393 5467 Acked Read 2 Done Sent Done false
19394 5468 Acked Flush 0 Done Sent Done false
19395 5469 Acked Read 8 Done Sent Done false
19396 5470 Acked Read 8 Done Sent Done false
19397 5471 Acked Read 7 Done Sent Done false
19398 5472 Acked Read 2 Done Sent Done false
19399 5473 Acked Read 7 Done Sent Done false
19400 5474 Acked Write 1 Done Sent Done false
19401 5475 Acked Read 6 Done Sent Done false
19402 5476 Acked Read 3 Done Sent Done false
19403 5477 Acked Read 4 Done Sent Done false
19404 5478 Acked Flush 0 Done Sent Done false
19405 5479 Acked Read 1 Done Sent Done false
19406 5480 Acked Read 4 Done Sent Done false
19407 5481 Acked Read 2 Done Sent Done false
19408 5482 Acked Write 4 Done Sent Done false
19409 5483 Acked Read 3 Done Sent Done false
19410 5484 Acked Read 8 Done Sent Done false
19411 5485 Acked Read 7 Done Sent Done false
19412 5486 Acked Read 10 Done Sent Done false
19413 5487 Acked Read 1 Done Sent Done false
19414 5488 Acked Read 1 Done Sent Done false
19415 5489 Acked Read 2 Done Sent Done false
19416 5490 Acked Read 7 Done Sent Done false
19417 5491 Acked Write 3 Done Sent Done false
19418 5492 Acked Flush 0 Done Sent Done false
19419 5493 Acked Read 6 Done Sent Done false
19420 5494 Acked Read 7 Done Sent Done false
19421 5495 Acked Write 2 Done Sent Done false
19422 5496 Acked Read 3 Done Sent Done false
19423 5497 Acked Read 6 Done Sent Done false
19424 5498 Acked Read 2 Done Sent Done false
19425 5499 Acked Read 8 Done Sent Done false
19426 5500 Acked Write 2 Done Sent Done false
19427 5501 Acked Flush 0 Done Sent Done false
19428 5502 Acked Read 8 Done Sent Done false
19429 5503 Acked Write 8 Done Sent Done false
19430 5504 Acked Read 6 Done Sent Done false
19431 5505 Acked Read 10 Done Sent Done false
19432 5506 Acked Read 2 Done Sent Done false
19433 5507 Acked Read 5 Done Sent Done false
19434 5508 Acked Read 4 Done Sent Done false
19435 5509 Acked Flush 0 Done Sent Done false
19436 5510 Acked Write 10 Done Sent Done false
19437 5511 Acked Read 8 Done Sent Done false
19438 5512 Acked Write 6 Done Sent Done false
19439 5513 Acked Read 6 Done Sent Done false
19440 5514 Acked Write 3 Done Sent Done false
19441 5515 Acked Flush 0 Done Sent Done false
19442 5516 Acked Write 7 Done Sent Done false
19443 5517 Acked Write 6 Done Sent Done false
19444 5518 Acked Write 2 Done Sent Done false
19445 5519 Acked Read 1 Done Sent Done false
19446 5520 Acked Read 5 Done Sent Done false
19447 5521 Acked Write 6 Done Sent Done false
19448 5522 Acked Flush 0 Done Sent Done false
19449 5523 Acked Write 3 Done Sent Done false
19450 5524 Acked Write 8 Done Sent Done false
19451 5525 Acked Write 3 Done Sent Done false
19452 5526 Acked Write 9 Done Sent Done false
19453 5527 Acked Read 6 Done Sent Done false
19454 STATES DS:0 DS:1 DS:2 TOTAL
19455 Sent 0 301 0 301
19456 Done 301 0 301 602
19457 Skipped 0 0 0 0
19458 Error 0 0 0 0
19459 Last Flush: Some(JobId(5522)) Some(JobId(5226)) Some(JobId(5522))
19460 Downstairs last five completed: 5226 5225 5224 5223 5222
19461 Upstairs last five completed: 5527 5526 5525 5524 5523
19462 CLIENT: Up:0 ds:301 act:2
194632025-03-17T22:23:18.985ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
194642025-03-17T22:23:19.660ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
194652025-03-17T22:23:19.660ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:46609) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
194662025-03-17T22:23:19.661ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
194672025-03-17T22:23:19.662ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
194682025-03-17T22:23:19.662ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
194692025-03-17T22:23:19.662ZINFOcrucible: send last flush ID to this DS: Some(JobId(5226)) = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
194702025-03-17T22:23:19.662ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
194712025-03-17T22:23:19.662ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(5226)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
194722025-03-17T22:23:19.662ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(5226)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
19473 ----------------------------------------------------------------
19474 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
19475 Downstairs last five completed: 5528 5527 5526 5525 5524
19476 Upstairs last five completed: 5528 5527 5526 5525 5524
19477 CLIENT: Up:0 ds:0 act:3
19478 Replay: All jobs finished
19479 Main task has work to do, go find it
19480 got dsc Stop(1)
19481 stop 1
19482 Mar 17 22:23:21.905 INFO request completed[8820] Got stop action so:false kr:false
19483 , latency_us: 382, response_code: 204, uri: /stop/cid/1, method: POST, req_id: e0db4928-5b43-4061-9ff4-097667751a2c, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
194842025-03-17T22:23:21.906ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
194852025-03-17T22:23:21.906ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
194862025-03-17T22:23:21.906ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
19487 Mar 17 22:23:21.906 INFO request completed, latency_us: 149, response_code: 200, uri: /state/cid/1, method: GET, req_id: 5c18147c-13dc-4551-9408-87de998be294, remote_addr: 127{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:23:21.9067801Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
19488 .0.0.1:58317, local_addr: 127.0.0.1:9998
19489 [8820] Exited with: ExitStatus(unix_wait_status(9))
19490 [8820][1] reports Exit
19491 Mar 17 22:23:25.908 INFO request completed, latency_us: 206, response_code: 200, uri: /state/cid/1, method: GET, req_id: 1a6432bb-6a31-4b27-a6fd-239f3df2a638, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
19492 001/300 Write block 3466 len 12288 data: 3 4 5
19493 002/300 Read block 864 len 16384
19494 003/300 Write block 2556 len 24576 data: 4 5 5 5 4 3
19495 004/300 Read block 1163 len 16384
19496 005/300 Read block 2899 len 32768
19497 006/300 Read block 2364 len 12288
19498 007/300 Write block 797 len 4096 data: 6
19499 008/300 Read block 2591 len 4096
19500 009/300 Write block 4404 len 28672 data: 2 3 3 3 3 4 4
19501 010/300 Write block 956 len 36864 data: 4 4 3 3 4 3 3 3 4
19502 011/300 Write block 3732 len 40960 data: 5 5 4 3 3 3 3 3 3 4
19503 012/300 Write block 3821 len 28672 data: 4 4 4 4 4 3 4
19504 013/300 Write block 3334 len 24576 data: 3 2 3 3 3 3
19505 014/300 Write block 130 len 4096 data: 5
19506 015/300 Read block 371 len 24576
19507 016/300 Write block 1334 len 20480 data: 5 5 5 2 2
19508 017/300 Write block 4354 len 32768 data: 4 4 4 4 5 4 5 5
19509 018/300 Write block 4762 len 20480 data: 4 3 4 4 5
19510 019/300 Read block 4832 len 16384
19511 020/300 Read block 3838 len 8192
19512 021/300 Write block 2693 len 28672 data: 2 2 4 5 4 4 5
19513 022/300 Read block 1753 len 32768
19514 023/300 Write block 1053 len 12288 data: 7 7 7
19515 024/300 Write block 4251 len 8192 data: 3 4
19516 025/300 Read block 2822 len 36864
19517 026/300 Read block 1164 len 4096
19518 027/300 Read block 2096 len 4096
19519 028/300 Write block 1656 len 28672 data: 5 4 4 4 4 3 3
19520 029/300 Write block 888 len 20480 data: 5 5 5 4 4
19521 030/300 Read block 2510 len 12288
19522 031/300 Read block 2798 len 24576
19523 032/300 Write block 4497 len 36864 data: 5 4 3 4 4 4 5 3 3
19524 033/300 Flush
19525 034/300 Write block 291 len 20480 data: 5 5 5 5 6
19526 035/300 Write block 498 len 4096 data: 3
19527 036/300 Write block 3850 len 12288 data: 5 4 6
19528 037/300 Read block 3473 len 36864
19529 038/300 Flush
19530 039/300 Write block 1720 len 12288 data: 4 5 5
19531 040/300 Flush
19532 041/300 Read block 1919 len 4096
19533 042/300 Flush
19534 043/300 Read block 3543 len 8192
19535 044/300 Flush
19536 045/300 Write block 2062 len 40960 data: 2 3 4 5 7 7 7 6 6 5
19537 046/300 Write block 66 len 36864 data: 3 3 4 3 3 4 4 5 4
19538 047/300 Flush
19539 048/300 Read block 1122 len 40960
19540 049/300 Read block 4888 len 8192
19541 050/300 Read block 3667 len 28672
19542 051/300 Flush
19543 052/300 Read block 1891 len 12288
19544 053/300 Write block 626 len 4096 data: 3
19545 054/300 Write block 2244 len 24576 data: 3 2 2 2 3 4
19546 055/300 Read block 0 len 32768
19547 056/300 Read block 1597 len 8192
19548 057/300 Write block 3732 len 20480 data: 6 6 5 4 4
19549 058/300 Write block 490 len 36864 data: 3 3 4 5 6 5 3 3 4
19550 059/300 Write block 1500 len 8192 data: 3 3
19551 060/300 Read block 4881 len 12288
19552 061/300 Flush
19553 062/300 Write block 2180 len 24576 data: 3 3 2 2 2 2
19554 063/300 Read block 2496 len 28672
19555 064/300 Write block 3348 len 4096 data: 8
19556 065/300 Flush
19557 066/300 Read block 656 len 4096
19558 067/300 Read block 853 len 40960
19559 068/300 Read block 1417 len 12288
19560 069/300 Write block 86 len 40960 data: 9 7 7 5 3 3 2 2 2 3
19561 070/300 Read block 3267 len 16384
19562 071/300 Write block 3226 len 8192 data: 2 3
19563 072/300 Read block 1181 len 24576
19564 073/300 Write block 2417 len 12288 data: 6 6 7
19565 074/300 Read block 2755 len 36864
19566 075/300 Read block 4712 len 8192
19567 076/300 Flush
19568 077/300 Read block 412 len 24576
19569 078/300 Write block 2482 len 20480 data: 4 4 4 3 3
19570 079/300 Read block 777 len 8192
19571 080/300 Write block 1095 len 4096 data: 3
19572 081/300 Write block 2586 len 40960 data: 8 5 5 5 3 3 3 3 3 3
19573 082/300 Write block 2983 len 32768 data: 3 3 3 3 2 2 2 3
19574 083/300 Flush
19575 084/300 Write block 1918 len 12288 data: 4 5 5
19576 085/300 Write block 2795 len 36864 data: 3 2 2 3 2 2 2 2 2
19577 086/300 Write block 811 len 36864 data: 4 4 4 5 4 3 4 4 5
19578 087/300 Read block 1824 len 36864
19579 088/300 Write block 930 len 20480 data: 4 4 3 4 4
19580 089/300 Read block 1766 len 24576
19581 090/300 Read block 757 len 24576
19582 091/300 Read block 3832 len 32768
19583 092/300 Read block 1951 len 24576
19584 093/300 Read block 1757 len 36864
19585 094/300 Write block 2904 len 12288 data: 6 4 3
19586 095/300 Flush
19587 096/300 Flush
19588 097/300 Read block 3264 len 28672
19589 098/300 Write block 4639 len 12288 data: 2 3 5
19590 099/300 Read block 2178 len 24576
19591 100/300 Read block 4702 len 28672
19592 101/300 Read block 3640 len 20480
19593 102/300 Write block 3795 len 28672 data: 2 2 2 2 3 2 2
19594 103/300 Write block 4888 len 36864 data: 3 3 3 2 2 2 3 4 5
19595 104/300 Flush
19596 105/300 Flush
19597 106/300 Write block 3622 len 12288 data: 5 5 4
19598 107/300 Read block 3777 len 12288
19599 108/300 Write block 2306 len 36864 data: 3 3 3 3 4 6 6 8 9
19600 109/300 Flush
19601 110/300 Read block 162 len 24576
19602 111/300 Write block 1128 len 20480 data: 4 3 3 4 5
19603 112/300 Read block 2529 len 28672
19604 113/300 Write block 3826 len 32768 data: 4 5 5 4 5 3 3 4
19605 114/300 Write block 3272 len 8192 data: 3 3
19606 115/300 Read block 1411 len 20480
19607 116/300 Read block 4814 len 28672
19608 117/300 Write block 2734 len 40960 data: 3 5 5 6 4 5 4 3 6 4
19609 118/300 Read block 4855 len 32768
19610 119/300 Write block 2965 len 20480 data: 5 4 4 4 5
19611 120/300 Read block 3973 len 24576
19612 121/300 Read block 578 len 28672
19613 122/300 Read block 3922 len 40960
19614 123/300 Write block 2818 len 40960 data: 2 3 5 5 6 6 5 5 5 5
19615 124/300 Flush
19616 125/300 Write block 1723 len 40960 data: 5 6 6 7 5 6 5 5 6 4
19617 126/300 Read block 4174 len 4096
19618 127/300 Flush
19619 128/300 Write block 4075 len 12288 data: 6 5 5
19620 129/300 Write block 470 len 40960 data: 5 4 5 3 3 5 5 5 4 4
19621 130/300 Read block 4413 len 28672
19622 131/300 Flush
19623 132/300 Write block 1431 len 16384 data: 2 2 3 4
19624 133/300 Read block 97 len 12288
19625 134/300 Read block 3935 len 16384
19626 135/300 Read block 4327 len 20480
19627 136/300 Read block 2982 len 32768
19628 137/300 Write block 3176 len 28672 data: 5 4 6 4 5 5 4
19629 138/300 Read block 1330 len 24576
19630 139/300 Write block 3453 len 12288 data: 3 2 2
19631 140/300 Read block 4577 len 20480
19632 141/300 Write block 380 len 32768 data: 3 3 3 3 3 2 2 2
19633 142/300 Write block 2703 len 20480 data: 4 5 6 4 5
19634 143/300 Write block 2162 len 36864 data: 3 3 3 4 4 4 5 5 6
19635 144/300 Read block 2657 len 8192
19636 145/300 Read block 116 len 24576
19637 146/300 Read block 271 len 8192
19638 147/300 Write block 2262 len 8192 data: 7 5
19639 148/300 Read block 3900 len 36864
19640 149/300 Read block 1871 len 8192
19641 150/300 Read block 909 len 16384
19642 151/300 Write block 828 len 16384 data: 3 2 2 2
19643 152/300 Flush
19644 153/300 Write block 3435 len 28672 data: 3 3 5 5 4 5 4
19645 154/300 Read block 254 len 12288
19646 155/300 Write block 1684 len 20480 data: 2 2 3 4 4
19647 156/300 Write block 3694 len 24576 data: 5 5 5 4 5 4
19648 157/300 Write block 54 len 16384 data: 2 2 3 3
19649 158/300 Read block 749 len 16384
19650 159/300 Read block 3873 len 36864
19651 160/300 Read block 853 len 12288
19652 161/300 Write block 2681 len 40960 data: 3 5 5 4 5 4 4 5 5 3
19653 162/300 Write block 200 len 12288 data: 3 5 5
19654 163/300 Write block 4221 len 28672 data: 6 6 5 5 6 6 7
19655 164/300 Read block 1127 len 40960
19656 165/300 Write block 2259 len 36864 data: 6 6 7 8 6 4 3 3 3
19657 166/300 Read block 1294 len 40960
19658 167/300 Read block 4500 len 4096
19659 168/300 Write block 254 len 4096 data: 2
19660 169/300 Read block 2271 len 40960
19661 170/300 Write block 2499 len 24576 data: 3 2 2 2 2 2
19662 171/300 Read block 9 len 40960
19663 172/300 Write block 1241 len 8192 data: 5 5
19664 173/300 Write block 605 len 28672 data: 3 3 3 3 4 4 3
19665 174/300 Read block 865 len 28672
19666 175/300 Read block 3717 len 24576
19667 176/300 Read block 3651 len 28672
19668 177/300 Read block 1184 len 20480
19669 178/300 Read block 4065 len 4096
19670 179/300 Read block 2172 len 32768
19671 180/300 Read block 1351 len 36864
19672 181/300 Read block 2463 len 12288
19673 182/300 Write block 4359 len 28672 data: 5 6 6 5 4 4 3
19674 183/300 Write block 2726 len 20480 data: 3 4 4 4 4
19675 184/300 Read block 525 len 28672
19676 185/300 Write block 656 len 36864 data: 3 4 4 5 6 6 6 5 5
19677 186/300 Read block 899 len 28672
19678 187/300 Write block 3723 len 40960 data: 4 4 2 3 4 4 4 5 5 7
19679 188/300 Read block 3703 len 12288
19680 189/300 Read block 4019 len 12288
19681 190/300 Read block 626 len 24576
19682 191/300 Write block 3504 len 40960 data: 5 5 4 4 4 3 2 2 2 3
19683 192/300 Read block 3102 len 24576
19684 193/300 Write block 3138 len 12288 data: 7 6 6
19685 194/300 Read block 346 len 32768
19686 195/300 Read block 4423 len 12288
19687 196/300 Flush
19688 197/300 Read block 2115 len 12288
19689 198/300 Write block 89 len 36864 data: 6 4 4 3 3 3 4 3 4
19690 199/300 Read block 887 len 8192
19691 200/300 Read block 194 len 28672
19692 201/300 Flush
19693 202/300 Read block 3523 len 40960
19694 203/300 Write block 957 len 16384 data: 5 4 4 5
19695 204/300 Read block 4650 len 40960
19696 205/300 Flush
19697 206/300 Write block 2378 len 28672 data: 3 5 5 5 6 6 6
19698 207/300 Read block 2048 len 40960
19699 208/300 Read block 3579 len 24576
19700 209/300 Read block 4195 len 32768
19701 210/300 Read block 291 len 28672
19702 211/300 Read block 1760 len 32768
19703 212/300 Read block 33 len 8192
19704 213/300 Write block 3597 len 36864 data: 4 4 4 5 5 5 3 3 4
19705 214/300 Read block 4181 len 8192
19706 215/300 Read block 912 len 4096
19707 216/300 Read block 4208 len 24576
19708 217/300 Read block 3220 len 24576
19709 218/300 Write block 1588 len 16384 data: 7 8 6 5
19710 219/300 Read block 4465 len 16384
19711 220/300 Write block 2947 len 32768 data: 6 6 6 6 6 5 5 4
19712 221/300 Write block 3820 len 20480 data: 4 5 5 5 5
19713 222/300 Write block 2712 len 12288 data: 4 4 4
19714 223/300 Read block 1853 len 12288
19715 224/300 Write block 1263 len 16384 data: 2 2 3 4
19716 225/300 Read block 1174 len 24576
19717 226/300 Read block 3076 len 32768
19718 227/300 Read block 4815 len 40960
19719 228/300 Read block 1016 len 36864
19720 229/300 Read block 2944 len 40960
19721 230/300 Write block 420 len 24576 data: 7 7 5 5 4 4
19722 231/300 Read block 2861 len 12288
19723 232/300 Write block 4253 len 16384 data: 3 5 5 6
19724 233/300 Write block 1710 len 24576 data: 10 7 6 6 6 7
19725 234/300 Write block 1355 len 16384 data: 3 3 2 4
19726 235/300 Read block 4308 len 28672
19727 236/300 Read block 40 len 8192
19728 237/300 Read block 3868 len 40960
19729 238/300 Read block 1326 len 24576
19730 239/300 Read block 2898 len 32768
19731 240/300 Write block 2955 len 8192 data: 3 2
19732 241/300 Read block 2734 len 28672
19733 242/300 Flush
19734 243/300 Read block 3309 len 36864
19735 244/300 Read block 535 len 16384
19736 245/300 Read block 3749 len 32768
19737 246/300 Read block 2037 len 40960
19738 247/300 Read block 4008 len 40960
19739 248/300 Read block 4867 len 28672
19740 249/300 Read block 3249 len 12288
19741 250/300 Write block 4110 len 16384 data: 2 2 2 2
19742 251/300 Flush
19743 252/300 Write block 4925 len 36864 data: 4 4 3 3 3 3 5 6 6
19744 253/300 Write block 4945 len 12288 data: 4 5 4
19745 254/300 Flush
19746 255/300 Read block 3788 len 24576
19747 256/300 Write block 3868 len 40960 data: 2 2 2 2 2 3 5 5 5 4
19748 257/300 Write block 1029 len 32768 data: 3 3 4 4 3 3 2 2
19749 258/300 Write block 1488 len 28672 data: 3 3 3 3 3 4 4
19750 259/300 Read block 431 len 4096
19751 260/300 Read block 4620 len 12288
19752 261/300 Read block 3038 len 16384
19753 262/300 Write block 249 len 12288 data: 2 3 3
19754 263/300 Read block 185 len 20480
19755 264/300 Flush
19756 265/300 Read block 4717 len 32768
19757 266/300 Read block 781 len 4096
19758 267/300 Write block 4362 len 24576 data: 6 5 5 4 4 4
19759 268/300 Write block 2295 len 24576 data: 3 3 4 5 4 4
19760 269/300 Write block 4381 len 20480 data: 6 6 6 6 6
19761 270/300 Read block 4206 len 16384
19762 271/300 Read block 2497 len 40960
19763 272/300 Write block 650 len 12288 data: 3 4 2
19764 273/300 Read block 867 len 24576
19765 274/300 Write block 834 len 16384 data: 3 2 2 2
19766 275/300 Read block 4121 len 40960
19767 276/300 Read block 3259 len 8192
19768 277/300 Read block 2704 len 4096
19769 278/300 Read block 1415 len 36864
19770 279/300 Flush
19771 280/300 Write block 1681 len 8192 data: 2 2
19772 281/300 Flush
19773 282/300 Read block 3633 len 36864
19774 283/300 Read block 216 len 4096
19775 284/300 Write block 2809 len 8192 data: 2 2
19776 285/300 Read block 2730 len 16384
19777 286/300 Read block 1605 len 28672
19778 287/300 Write block 3051 len 12288 data: 4 3 3
19779 288/300 Write block 4609 len 28672 data: 2 2 2 2 2 2 2
19780 289/300 Write block 2509 len 12288 data: 3 3 2
19781 290/300 Read block 2305 len 16384
19782 291/300 Read block 695 len 36864
19783 292/300 Read block 1389 len 16384
19784 293/300 Read block 2110 len 32768
19785 294/300 Read block 279 len 36864
19786 295/300 Write block 3403 len 8192 data: 5 4
19787 296/300 Read block 3916 len 32768
19788 297/300 Flush
19789 298/300 Write block 2782 len 40960 data: 3 4 4 4 4 3 4 4 4 3
19790 299/300 Write block 2732 len 12288 data: 4 4 4
19791 300/300 Flush
19792 301/300 Flush
19793 Main task has work to do, go find it
19794 got dsc Start(1)
19795 start 1
19796 Mar 17 22:23:26.161 INFO request completed, latency_us: 240, response_code: 204, uri: /start/cid/1, method: POST, req_id: 4c197b07-3a68-4426-924d-af1143ca3055, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
19797 Replay: started 1, returned:Ok(())
19798 ----------------------------------------------------------------
19799 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19800 States: Active Offline Active
19801 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19802 5529 Acked Write 3 Done Sent Done false
19803 5530 Acked Read 4 Done Sent Done false
19804 5531 Acked Write 6 Done Sent Done false
19805 5532 Acked Read 4 Done Sent Done false
19806 5533 Acked Read 8 Done Sent Done false
19807 5534 Acked Read 3 Done Sent Done false
19808 5535 Acked Write 1 Done Sent Done false
19809 5536 Acked Read 1 Done Sent Done false
19810 5537 Acked Write 7 Done Sent Done false
19811 5538 Acked Write 9 Done Sent Done false
19812 5539 Acked Write 10 Done Sent Done false
19813 5540 Acked Write 7 Done Sent Done false
19814 5541 Acked Write 6 Done Sent Done false
19815 5542 Acked Write 1 Done Sent Done false
19816 5543 Acked Read 6 Done Sent Done false
19817 5544 Acked Write 5 Done Sent Done false
19818 5545 Acked Write 8 Done Sent Done false
19819 5546 Acked Write 5 Done Sent Done false
19820 5547 Acked Read 4 Done Sent Done false
19821 5548 Acked Read 2 Done Sent Done false
19822 5549 Acked Write 7 Done Sent Done false
19823 5550 Acked Read 8 Done Sent Done false
19824 5551 Acked Write 3 Done Sent Done false
19825 5552 Acked Write 2 Done Sent Done false
19826 5553 Acked Read 9 Done Sent Done false
19827 5554 Acked Read 1 Done Sent Done false
19828 5555 Acked Read 1 Done Sent Done false
19829 5556 Acked Write 7 Done Sent Done false
19830 5557 Acked Write 5 Done Sent Done false
19831 5558 Acked Read 3 Done Sent Done false
19832 5559 Acked Read 6 Done Sent Done false
19833 5560 Acked Write 9 Done Sent Done false
19834 5561 Acked Flush 0 Done Sent Done false
19835 5562 Acked Write 5 Done Sent Done false
19836 5563 Acked Write 1 Done Sent Done false
19837 5564 Acked Write 3 Done Sent Done false
19838 5565 Acked Read 9 Done Sent Done false
19839 5566 Acked Flush 0 Done Sent Done false
19840 5567 Acked Write 3 Done Sent Done false
19841 5568 Acked Flush 0 Done Sent Done false
19842 5569 Acked Read 1 Done Sent Done false
19843 5570 Acked Flush 0 Done Sent Done false
19844 5571 Acked Read 2 Done Sent Done false
19845 5572 Acked Flush 0 Done Sent Done false
19846 5573 Acked Write 10 Done Sent Done false
19847 5574 Acked Write 9 Done Sent Done false
19848 5575 Acked Flush 0 Done Sent Done false
19849 5576 Acked Read 10 Done Sent Done false
19850 5577 Acked Read 2 Done Sent Done false
19851 5578 Acked Read 7 Done Sent Done false
19852 5579 Acked Flush 0 Done Sent Done false
19853 5580 Acked Read 3 Done Sent Done false
19854 5581 Acked Write 1 Done Sent Done false
19855 5582 Acked Write 6 Done Sent Done false
19856 5583 Acked Read 8 Done Sent Done false
19857 5584 Acked Read 2 Done Sent Done false
19858 5585 Acked Write 5 Done Sent Done false
19859 5586 Acked Write 9 Done Sent Done false
19860 5587 Acked Write 2 Done Sent Done false
19861 5588 Acked Read 3 Done Sent Done false
19862 5589 Acked Flush 0 Done Sent Done false
19863 5590 Acked Write 6 Done Sent Done false
19864 5591 Acked Read 7 Done Sent Done false
19865 5592 Acked Write 1 Done Sent Done false
19866 5593 Acked Flush 0 Done Sent Done false
19867 5594 Acked Read 1 Done Sent Done false
19868 5595 Acked Read 10 Done Sent Done false
19869 5596 Acked Read 3 Done Sent Done false
19870 5597 Acked Write 10 Done Sent Done false
19871 5598 Acked Read 4 Done Sent Done false
19872 5599 Acked Write 2 Done Sent Done false
19873 5600 Acked Read 6 Done Sent Done false
19874 5601 Acked Write 3 Done Sent Done false
19875 5602 Acked Read 9 Done Sent Done false
19876 5603 Acked Read 2 Done Sent Done false
19877 5604 Acked Flush 0 Done Sent Done false
19878 5605 Acked Read 6 Done Sent Done false
19879 5606 Acked Write 5 Done Sent Done false
19880 5607 Acked Read 2 Done Sent Done false
19881 5608 Acked Write 1 Done Sent Done false
19882 5609 Acked Write 10 Done Sent Done false
19883 5610 Acked Write 8 Done Sent Done false
19884 5611 Acked Flush 0 Done Sent Done false
19885 5612 Acked Write 3 Done Sent Done false
19886 5613 Acked Write 9 Done Sent Done false
19887 5614 Acked Write 9 Done Sent Done false
19888 5615 Acked Read 9 Done Sent Done false
19889 5616 Acked Write 5 Done Sent Done false
19890 5617 Acked Read 6 Done Sent Done false
19891 5618 Acked Read 6 Done Sent Done false
19892 5619 Acked Read 8 Done Sent Done false
19893 5620 Acked Read 6 Done Sent Done false
19894 5621 Acked Read 9 Done Sent Done false
19895 5622 Acked Write 3 Done Sent Done false
19896 5623 Acked Flush 0 Done Sent Done false
19897 5624 Acked Flush 0 Done Sent Done false
19898 5625 Acked Read 7 Done Sent Done false
19899 5626 Acked Write 3 Done Sent Done false
19900 5627 Acked Read 6 Done Sent Done false
19901 5628 Acked Read 7 Done Sent Done false
19902 5629 Acked Read 5 Done Sent Done false
19903 5630 Acked Write 7 Done Sent Done false
19904 5631 Acked Write 9 Done Sent Done false
19905 5632 Acked Flush 0 Done Sent Done false
19906 5633 Acked Flush 0 Done Sent Done false
19907 5634 Acked Write 3 Done Sent Done false
19908 5635 Acked Read 3 Done Sent Done false
19909 5636 Acked Write 9 Done Sent Done false
19910 5637 Acked Flush 0 Done Sent Done false
19911 5638 Acked Read 6 Done Sent Done false
19912 5639 Acked Write 5 Done Sent Done false
19913 5640 Acked Read 7 Done Sent Done false
19914 5641 Acked Write 8 Done Sent Done false
19915 5642 Acked Write 2 Done Sent Done false
19916 5643 Acked Read 5 Done Sent Done false
19917 5644 Acked Read 7 Done Sent Done false
19918 5645 Acked Write 10 Done Sent Done false
19919 5646 Acked Read 8 Done Sent Done false
19920 5647 Acked Write 5 Done Sent Done false
19921 5648 Acked Read 6 Done Sent Done false
19922 5649 Acked Read 7 Done Sent Done false
19923 5650 Acked Read 10 Done Sent Done false
19924 5651 Acked Write 10 Done Sent Done false
19925 5652 Acked Flush 0 Done Sent Done false
19926 5653 Acked Write 10 Done Sent Done false
19927 5654 Acked Read 1 Done Sent Done false
19928 5655 Acked Flush 0 Done Sent Done false
19929 5656 Acked Write 3 Done Sent Done false
19930 5657 Acked Write 10 Done Sent Done false
19931 5658 Acked Read 7 Done Sent Done false
19932 5659 Acked Flush 0 Done Sent Done false
19933 5660 Acked Write 4 Done Sent Done false
19934 5661 Acked Read 3 Done Sent Done false
19935 5662 Acked Read 4 Done Sent Done false
19936 5663 Acked Read 5 Done Sent Done false
19937 5664 Acked Read 8 Done Sent Done false
19938 5665 Acked Write 7 Done Sent Done false
19939 5666 Acked Read 6 Done Sent Done false
19940 5667 Acked Write 3 Done Sent Done false
19941 5668 Acked Read 5 Done Sent Done false
19942 5669 Acked Write 8 Done Sent Done false
19943 5670 Acked Write 5 Done Sent Done false
19944 5671 Acked Write 9 Done Sent Done false
19945 5672 Acked Read 2 Done Sent Done false
19946 5673 Acked Read 6 Done Sent Done false
19947 5674 Acked Read 2 Done Sent Done false
19948 5675 Acked Write 2 Done Sent Done false
19949 5676 Acked Read 9 Done Sent Done false
19950 5677 Acked Read 2 Done Sent Done false
19951 5678 Acked Read 4 Done Sent Done false
19952 5679 Acked Write 4 Done Sent Done false
19953 5680 Acked Flush 0 Done Sent Done false
19954 5681 Acked Write 7 Done Sent Done false
19955 5682 Acked Read 3 Done Sent Done false
19956 5683 Acked Write 5 Done Sent Done false
19957 5684 Acked Write 6 Done Sent Done false
19958 5685 Acked Write 4 Done Sent Done false
19959 5686 Acked Read 4 Done Sent Done false
19960 5687 Acked Read 9 Done Sent Done false
19961 5688 Acked Read 3 Done Sent Done false
19962 5689 Acked Write 10 Done Sent Done false
19963 5690 Acked Write 3 Done Sent Done false
19964 5691 Acked Write 7 Done Sent Done false
19965 5692 Acked Read 10 Done Sent Done false
19966 5693 Acked Write 9 Done Sent Done false
19967 5694 Acked Read 10 Done Sent Done false
19968 5695 Acked Read 1 Done Sent Done false
19969 5696 Acked Write 1 Done Sent Done false
19970 5697 Acked Read 10 Done Sent Done false
19971 5698 Acked Write 6 Done Sent Done false
19972 5699 Acked Read 10 Done Sent Done false
19973 5700 Acked Write 2 Done Sent Done false
19974 5701 Acked Write 7 Done Sent Done false
19975 5702 Acked Read 7 Done Sent Done false
19976 5703 Acked Read 6 Done Sent Done false
19977 5704 Acked Read 7 Done Sent Done false
19978 5705 Acked Read 5 Done Sent Done false
19979 5706 Acked Read 1 Done Sent Done false
19980 5707 Acked Read 8 Done Sent Done false
19981 5708 Acked Read 9 Done Sent Done false
19982 5709 Acked Read 3 Done Sent Done false
19983 5710 Acked Write 7 Done Sent Done false
19984 5711 Acked Write 5 Done Sent Done false
19985 5712 Acked Read 7 Done Sent Done false
19986 5713 Acked Write 9 Done Sent Done false
19987 5714 Acked Read 7 Done Sent Done false
19988 5715 Acked Write 10 Done Sent Done false
19989 5716 Acked Read 3 Done Sent Done false
19990 5717 Acked Read 3 Done Sent Done false
19991 5718 Acked Read 6 Done Sent Done false
19992 5719 Acked Write 10 Done Sent Done false
19993 5720 Acked Read 6 Done Sent Done false
19994 5721 Acked Write 3 Done Sent Done false
19995 5722 Acked Read 8 Done Sent Done false
19996 5723 Acked Read 3 Done Sent Done false
19997 5724 Acked Flush 0 Done Sent Done false
19998 5725 Acked Read 3 Done Sent Done false
19999 5726 Acked Write 9 Done Sent Done false
20000 5727 Acked Read 2 Done Sent Done false
20001 5728 Acked Read 7 Done Sent Done false
20002 5729 Acked Flush 0 Done Sent Done false
20003 5730 Acked Read 10 Done Sent Done false
20004 5731 Acked Write 4 Done Sent Done false
20005 5732 Acked Read 10 Done Sent Done false
20006 5733 Acked Flush 0 Done Sent Done false
20007 5734 Acked Write 7 Done Sent Done false
20008 5735 Acked Read 10 Done Sent Done false
20009 5736 Acked Read 6 Done Sent Done false
20010 5737 Acked Read 8 Done Sent Done false
20011 5738 Acked Read 7 Done Sent Done false
20012 5739 Acked Read 8 Done Sent Done false
20013 5740 Acked Read 2 Done Sent Done false
20014 5741 Acked Write 9 Done Sent Done false
20015 5742 Acked Read 2 Done Sent Done false
20016 5743 Acked Read 1 Done Sent Done false
20017 5744 Acked Read 6 Done Sent Done false
20018 5745 Acked Read 6 Done Sent Done false
20019 5746 Acked Write 4 Done Sent Done false
20020 5747 Acked Read 4 Done Sent Done false
20021 5748 Acked Write 8 Done Sent Done false
20022 5749 Acked Write 5 Done Sent Done false
20023 5750 Acked Write 3 Done Sent Done false
20024 5751 Acked Read 3 Done Sent Done false
20025 5752 Acked Write 4 Done Sent Done false
20026 5753 Acked Read 6 Done Sent Done false
20027 5754 Acked Read 8 Done Sent Done false
20028 5755 Acked Read 10 Done Sent Done false
20029 5756 Acked Read 9 Done Sent Done false
20030 5757 Acked Read 10 Done Sent Done false
20031 5758 Acked Write 6 Done Sent Done false
20032 5759 Acked Read 3 Done Sent Done false
20033 5760 Acked Write 4 Done Sent Done false
20034 5761 Acked Write 6 Done Sent Done false
20035 5762 Acked Write 4 Done Sent Done false
20036 5763 Acked Read 7 Done Sent Done false
20037 5764 Acked Read 2 Done Sent Done false
20038 5765 Acked Read 10 Done Sent Done false
20039 5766 Acked Read 6 Done Sent Done false
20040 5767 Acked Read 8 Done Sent Done false
20041 5768 Acked Write 2 Done Sent Done false
20042 5769 Acked Read 7 Done Sent Done false
20043 5770 Acked Flush 0 Done Sent Done false
20044 5771 Acked Read 9 Done Sent Done false
20045 5772 Acked Read 4 Done Sent Done false
20046 5773 Acked Read 8 Done Sent Done false
20047 5774 Acked Read 10 Done Sent Done false
20048 5775 Acked Read 10 Done Sent Done false
20049 5776 Acked Read 7 Done Sent Done false
20050 5777 Acked Read 3 Done Sent Done false
20051 5778 Acked Write 4 Done Sent Done false
20052 5779 Acked Flush 0 Done Sent Done false
20053 5780 Acked Write 9 Done Sent Done false
20054 5781 Acked Write 3 Done Sent Done false
20055 5782 Acked Flush 0 Done Sent Done false
20056 5783 Acked Read 6 Done Sent Done false
20057 5784 Acked Write 10 Done Sent Done false
20058 5785 Acked Write 8 Done Sent Done false
20059 5786 Acked Write 7 Done Sent Done false
20060 5787 Acked Read 1 Done Sent Done false
20061 5788 Acked Read 3 Done Sent Done false
20062 5789 Acked Read 4 Done Sent Done false
20063 5790 Acked Write 3 Done Sent Done false
20064 5791 Acked Read 5 Done Sent Done false
20065 5792 Acked Flush 0 Done Sent Done false
20066 5793 Acked Read 8 Done Sent Done false
20067 5794 Acked Read 1 Done Sent Done false
20068 5795 Acked Write 6 Done Sent Done false
20069 5796 Acked Write 6 Done Sent Done false
20070 5797 Acked Write 5 Done Sent Done false
20071 5798 Acked Read 4 Done Sent Done false
20072 5799 Acked Read 10 Done Sent Done false
20073 5800 Acked Write 3 Done Sent Done false
20074 5801 Acked Read 6 Done Sent Done false
20075 5802 Acked Write 4 Done Sent Done false
20076 5803 Acked Read 10 Done Sent Done false
20077 5804 Acked Read 2 Done Sent Done false
20078 5805 Acked Read 1 Done Sent Done false
20079 5806 Acked Read 9 Done Sent Done false
20080 5807 Acked Flush 0 Done Sent Done false
20081 5808 Acked Write 2 Done Sent Done false
20082 5809 Acked Flush 0 Done Sent Done false
20083 5810 Acked Read 9 Done Sent Done false
20084 5811 Acked Read 1 Done Sent Done false
20085 5812 Acked Write 2 Done Sent Done false
20086 5813 Acked Read 4 Done Sent Done false
20087 5814 Acked Read 7 Done Sent Done false
20088 5815 Acked Write 3 Done Sent Done false
20089 5816 Acked Write 7 Done Sent Done false
20090 5817 Acked Write 3 Done Sent Done false
20091 5818 Acked Read 4 Done Sent Done false
20092 5819 Acked Read 9 Done Sent Done false
20093 5820 Acked Read 4 Done Sent Done false
20094 5821 Acked Read 8 Done Sent Done false
20095 5822 Acked Read 9 Done Sent Done false
20096 5823 Acked Write 2 Done Sent Done false
20097 5824 Acked Read 8 Done Sent Done false
20098 5825 Acked Flush 0 Done Sent Done false
20099 5826 Acked Write 10 Done Sent Done false
20100 5827 Acked Write 3 Done Sent Done false
20101 5828 Acked Flush 0 Done Sent Done false
20102 5829 Acked Flush 0 Done Sent Done false
20103 STATES DS:0 DS:1 DS:2 TOTAL
20104 Sent 0 301 0 301
20105 Done 301 0 301 602
20106 Skipped 0 0 0 0
20107 Error 0 0 0 0
20108 Last Flush: Some(JobId(5829)) Some(JobId(5528)) Some(JobId(5829))
20109 Downstairs last five completed: 5528 5527 5526 5525 5524
20110 Upstairs last five completed: 5829 5828 5827 5826 5825
20111 CLIENT: Up:0 ds:301 act:2
201122025-03-17T22:23:27.000ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
201132025-03-17T22:23:28.002ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
201142025-03-17T22:23:29.004ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
201152025-03-17T22:23:30.006ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
20116 ----------------------------------------------------------------
20117 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20118 States: Active Offline Active
20119 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20120 5529 Acked Write 3 Done Sent Done false
20121 5530 Acked Read 4 Done Sent Done false
20122 5531 Acked Write 6 Done Sent Done false
20123 5532 Acked Read 4 Done Sent Done false
20124 5533 Acked Read 8 Done Sent Done false
20125 5534 Acked Read 3 Done Sent Done false
20126 5535 Acked Write 1 Done Sent Done false
20127 5536 Acked Read 1 Done Sent Done false
20128 5537 Acked Write 7 Done Sent Done false
20129 5538 Acked Write 9 Done Sent Done false
20130 5539 Acked Write 10 Done Sent Done false
20131 5540 Acked Write 7 Done Sent Done false
20132 5541 Acked Write 6 Done Sent Done false
20133 5542 Acked Write 1 Done Sent Done false
20134 5543 Acked Read 6 Done Sent Done false
20135 5544 Acked Write 5 Done Sent Done false
20136 5545 Acked Write 8 Done Sent Done false
20137 5546 Acked Write 5 Done Sent Done false
20138 5547 Acked Read 4 Done Sent Done false
20139 5548 Acked Read 2 Done Sent Done false
20140 5549 Acked Write 7 Done Sent Done false
20141 5550 Acked Read 8 Done Sent Done false
20142 5551 Acked Write 3 Done Sent Done false
20143 5552 Acked Write 2 Done Sent Done false
20144 5553 Acked Read 9 Done Sent Done false
20145 5554 Acked Read 1 Done Sent Done false
20146 5555 Acked Read 1 Done Sent Done false
20147 5556 Acked Write 7 Done Sent Done false
20148 5557 Acked Write 5 Done Sent Done false
20149 5558 Acked Read 3 Done Sent Done false
20150 5559 Acked Read 6 Done Sent Done false
20151 5560 Acked Write 9 Done Sent Done false
20152 5561 Acked Flush 0 Done Sent Done false
20153 5562 Acked Write 5 Done Sent Done false
20154 5563 Acked Write 1 Done Sent Done false
20155 5564 Acked Write 3 Done Sent Done false
20156 5565 Acked Read 9 Done Sent Done false
20157 5566 Acked Flush 0 Done Sent Done false
20158 5567 Acked Write 3 Done Sent Done false
20159 5568 Acked Flush 0 Done Sent Done false
20160 5569 Acked Read 1 Done Sent Done false
20161 5570 Acked Flush 0 Done Sent Done false
20162 5571 Acked Read 2 Done Sent Done false
20163 5572 Acked Flush 0 Done Sent Done false
20164 5573 Acked Write 10 Done Sent Done false
20165 5574 Acked Write 9 Done Sent Done false
20166 5575 Acked Flush 0 Done Sent Done false
20167 5576 Acked Read 10 Done Sent Done false
20168 5577 Acked Read 2 Done Sent Done false
20169 5578 Acked Read 7 Done Sent Done false
20170 5579 Acked Flush 0 Done Sent Done false
20171 5580 Acked Read 3 Done Sent Done false
20172 5581 Acked Write 1 Done Sent Done false
20173 5582 Acked Write 6 Done Sent Done false
20174 5583 Acked Read 8 Done Sent Done false
20175 5584 Acked Read 2 Done Sent Done false
20176 5585 Acked Write 5 Done Sent Done false
20177 5586 Acked Write 9 Done Sent Done false
20178 5587 Acked Write 2 Done Sent Done false
20179 5588 Acked Read 3 Done Sent Done false
20180 5589 Acked Flush 0 Done Sent Done false
20181 5590 Acked Write 6 Done Sent Done false
20182 5591 Acked Read 7 Done Sent Done false
20183 5592 Acked Write 1 Done Sent Done false
20184 5593 Acked Flush 0 Done Sent Done false
20185 5594 Acked Read 1 Done Sent Done false
20186 5595 Acked Read 10 Done Sent Done false
20187 5596 Acked Read 3 Done Sent Done false
20188 5597 Acked Write 10 Done Sent Done false
20189 5598 Acked Read 4 Done Sent Done false
20190 5599 Acked Write 2 Done Sent Done false
20191 5600 Acked Read 6 Done Sent Done false
20192 5601 Acked Write 3 Done Sent Done false
20193 5602 Acked Read 9 Done Sent Done false
20194 5603 Acked Read 2 Done Sent Done false
20195 5604 Acked Flush 0 Done Sent Done false
20196 5605 Acked Read 6 Done Sent Done false
20197 5606 Acked Write 5 Done Sent Done false
20198 5607 Acked Read 2 Done Sent Done false
20199 5608 Acked Write 1 Done Sent Done false
20200 5609 Acked Write 10 Done Sent Done false
20201 5610 Acked Write 8 Done Sent Done false
20202 5611 Acked Flush 0 Done Sent Done false
20203 5612 Acked Write 3 Done Sent Done false
20204 5613 Acked Write 9 Done Sent Done false
20205 5614 Acked Write 9 Done Sent Done false
20206 5615 Acked Read 9 Done Sent Done false
20207 5616 Acked Write 5 Done Sent Done false
20208 5617 Acked Read 6 Done Sent Done false
20209 5618 Acked Read 6 Done Sent Done false
20210 5619 Acked Read 8 Done Sent Done false
20211 5620 Acked Read 6 Done Sent Done false
20212 5621 Acked Read 9 Done Sent Done false
20213 5622 Acked Write 3 Done Sent Done false
20214 5623 Acked Flush 0 Done Sent Done false
20215 5624 Acked Flush 0 Done Sent Done false
20216 5625 Acked Read 7 Done Sent Done false
20217 5626 Acked Write 3 Done Sent Done false
20218 5627 Acked Read 6 Done Sent Done false
20219 5628 Acked Read 7 Done Sent Done false
20220 5629 Acked Read 5 Done Sent Done false
20221 5630 Acked Write 7 Done Sent Done false
20222 5631 Acked Write 9 Done Sent Done false
20223 5632 Acked Flush 0 Done Sent Done false
20224 5633 Acked Flush 0 Done Sent Done false
20225 5634 Acked Write 3 Done Sent Done false
20226 5635 Acked Read 3 Done Sent Done false
20227 5636 Acked Write 9 Done Sent Done false
20228 5637 Acked Flush 0 Done Sent Done false
20229 5638 Acked Read 6 Done Sent Done false
20230 5639 Acked Write 5 Done Sent Done false
20231 5640 Acked Read 7 Done Sent Done false
20232 5641 Acked Write 8 Done Sent Done false
20233 5642 Acked Write 2 Done Sent Done false
20234 5643 Acked Read 5 Done Sent Done false
20235 5644 Acked Read 7 Done Sent Done false
20236 5645 Acked Write 10 Done Sent Done false
20237 5646 Acked Read 8 Done Sent Done false
20238 5647 Acked Write 5 Done Sent Done false
20239 5648 Acked Read 6 Done Sent Done false
20240 5649 Acked Read 7 Done Sent Done false
20241 5650 Acked Read 10 Done Sent Done false
20242 5651 Acked Write 10 Done Sent Done false
20243 5652 Acked Flush 0 Done Sent Done false
20244 5653 Acked Write 10 Done Sent Done false
20245 5654 Acked Read 1 Done Sent Done false
20246 5655 Acked Flush 0 Done Sent Done false
20247 5656 Acked Write 3 Done Sent Done false
20248 5657 Acked Write 10 Done Sent Done false
20249 5658 Acked Read 7 Done Sent Done false
20250 5659 Acked Flush 0 Done Sent Done false
20251 5660 Acked Write 4 Done Sent Done false
20252 5661 Acked Read 3 Done Sent Done false
20253 5662 Acked Read 4 Done Sent Done false
20254 5663 Acked Read 5 Done Sent Done false
20255 5664 Acked Read 8 Done Sent Done false
20256 5665 Acked Write 7 Done Sent Done false
20257 5666 Acked Read 6 Done Sent Done false
20258 5667 Acked Write 3 Done Sent Done false
20259 5668 Acked Read 5 Done Sent Done false
20260 5669 Acked Write 8 Done Sent Done false
20261 5670 Acked Write 5 Done Sent Done false
20262 5671 Acked Write 9 Done Sent Done false
20263 5672 Acked Read 2 Done Sent Done false
20264 5673 Acked Read 6 Done Sent Done false
20265 5674 Acked Read 2 Done Sent Done false
20266 5675 Acked Write 2 Done Sent Done false
20267 5676 Acked Read 9 Done Sent Done false
20268 5677 Acked Read 2 Done Sent Done false
20269 5678 Acked Read 4 Done Sent Done false
20270 5679 Acked Write 4 Done Sent Done false
20271 5680 Acked Flush 0 Done Sent Done false
20272 5681 Acked Write 7 Done Sent Done false
20273 5682 Acked Read 3 Done Sent Done false
20274 5683 Acked Write 5 Done Sent Done false
20275 5684 Acked Write 6 Done Sent Done false
20276 5685 Acked Write 4 Done Sent Done false
20277 5686 Acked Read 4 Done Sent Done false
20278 5687 Acked Read 9 Done Sent Done false
20279 5688 Acked Read 3 Done Sent Done false
20280 5689 Acked Write 10 Done Sent Done false
20281 5690 Acked Write 3 Done Sent Done false
20282 5691 Acked Write 7 Done Sent Done false
20283 5692 Acked Read 10 Done Sent Done false
20284 5693 Acked Write 9 Done Sent Done false
20285 5694 Acked Read 10 Done Sent Done false
20286 5695 Acked Read 1 Done Sent Done false
20287 5696 Acked Write 1 Done Sent Done false
20288 5697 Acked Read 10 Done Sent Done false
20289 5698 Acked Write 6 Done Sent Done false
20290 5699 Acked Read 10 Done Sent Done false
20291 5700 Acked Write 2 Done Sent Done false
20292 5701 Acked Write 7 Done Sent Done false
20293 5702 Acked Read 7 Done Sent Done false
20294 5703 Acked Read 6 Done Sent Done false
20295 5704 Acked Read 7 Done Sent Done false
20296 5705 Acked Read 5 Done Sent Done false
20297 5706 Acked Read 1 Done Sent Done false
20298 5707 Acked Read 8 Done Sent Done false
20299 5708 Acked Read 9 Done Sent Done false
20300 5709 Acked Read 3 Done Sent Done false
20301 5710 Acked Write 7 Done Sent Done false
20302 5711 Acked Write 5 Done Sent Done false
20303 5712 Acked Read 7 Done Sent Done false
20304 5713 Acked Write 9 Done Sent Done false
20305 5714 Acked Read 7 Done Sent Done false
20306 5715 Acked Write 10 Done Sent Done false
20307 5716 Acked Read 3 Done Sent Done false
20308 5717 Acked Read 3 Done Sent Done false
20309 5718 Acked Read 6 Done Sent Done false
20310 5719 Acked Write 10 Done Sent Done false
20311 5720 Acked Read 6 Done Sent Done false
20312 5721 Acked Write 3 Done Sent Done false
20313 5722 Acked Read 8 Done Sent Done false
20314 5723 Acked Read 3 Done Sent Done false
20315 5724 Acked Flush 0 Done Sent Done false
20316 5725 Acked Read 3 Done Sent Done false
20317 5726 Acked Write 9 Done Sent Done false
20318 5727 Acked Read 2 Done Sent Done false
20319 5728 Acked Read 7 Done Sent Done false
20320 5729 Acked Flush 0 Done Sent Done false
20321 5730 Acked Read 10 Done Sent Done false
20322 5731 Acked Write 4 Done Sent Done false
20323 5732 Acked Read 10 Done Sent Done false
20324 5733 Acked Flush 0 Done Sent Done false
20325 5734 Acked Write 7 Done Sent Done false
20326 5735 Acked Read 10 Done Sent Done false
20327 5736 Acked Read 6 Done Sent Done false
20328 5737 Acked Read 8 Done Sent Done false
20329 5738 Acked Read 7 Done Sent Done false
20330 5739 Acked Read 8 Done Sent Done false
20331 5740 Acked Read 2 Done Sent Done false
20332 5741 Acked Write 9 Done Sent Done false
20333 5742 Acked Read 2 Done Sent Done false
20334 5743 Acked Read 1 Done Sent Done false
20335 5744 Acked Read 6 Done Sent Done false
20336 5745 Acked Read 6 Done Sent Done false
20337 5746 Acked Write 4 Done Sent Done false
20338 5747 Acked Read 4 Done Sent Done false
20339 5748 Acked Write 8 Done Sent Done false
20340 5749 Acked Write 5 Done Sent Done false
20341 5750 Acked Write 3 Done Sent Done false
20342 5751 Acked Read 3 Done Sent Done false
20343 5752 Acked Write 4 Done Sent Done false
20344 5753 Acked Read 6 Done Sent Done false
20345 5754 Acked Read 8 Done Sent Done false
20346 5755 Acked Read 10 Done Sent Done false
20347 5756 Acked Read 9 Done Sent Done false
20348 5757 Acked Read 10 Done Sent Done false
20349 5758 Acked Write 6 Done Sent Done false
20350 5759 Acked Read 3 Done Sent Done false
20351 5760 Acked Write 4 Done Sent Done false
20352 5761 Acked Write 6 Done Sent Done false
20353 5762 Acked Write 4 Done Sent Done false
20354 5763 Acked Read 7 Done Sent Done false
20355 5764 Acked Read 2 Done Sent Done false
20356 5765 Acked Read 10 Done Sent Done false
20357 5766 Acked Read 6 Done Sent Done false
20358 5767 Acked Read 8 Done Sent Done false
20359 5768 Acked Write 2 Done Sent Done false
20360 5769 Acked Read 7 Done Sent Done false
20361 5770 Acked Flush 0 Done Sent Done false
20362 5771 Acked Read 9 Done Sent Done false
20363 5772 Acked Read 4 Done Sent Done false
20364 5773 Acked Read 8 Done Sent Done false
20365 5774 Acked Read 10 Done Sent Done false
20366 5775 Acked Read 10 Done Sent Done false
20367 5776 Acked Read 7 Done Sent Done false
20368 5777 Acked Read 3 Done Sent Done false
20369 5778 Acked Write 4 Done Sent Done false
20370 5779 Acked Flush 0 Done Sent Done false
20371 5780 Acked Write 9 Done Sent Done false
20372 5781 Acked Write 3 Done Sent Done false
20373 5782 Acked Flush 0 Done Sent Done false
20374 5783 Acked Read 6 Done Sent Done false
20375 5784 Acked Write 10 Done Sent Done false
20376 5785 Acked Write 8 Done Sent Done false
20377 5786 Acked Write 7 Done Sent Done false
20378 5787 Acked Read 1 Done Sent Done false
20379 5788 Acked Read 3 Done Sent Done false
20380 5789 Acked Read 4 Done Sent Done false
20381 5790 Acked Write 3 Done Sent Done false
20382 5791 Acked Read 5 Done Sent Done false
20383 5792 Acked Flush 0 Done Sent Done false
20384 5793 Acked Read 8 Done Sent Done false
20385 5794 Acked Read 1 Done Sent Done false
20386 5795 Acked Write 6 Done Sent Done false
20387 5796 Acked Write 6 Done Sent Done false
20388 5797 Acked Write 5 Done Sent Done false
20389 5798 Acked Read 4 Done Sent Done false
20390 5799 Acked Read 10 Done Sent Done false
20391 5800 Acked Write 3 Done Sent Done false
20392 5801 Acked Read 6 Done Sent Done false
20393 5802 Acked Write 4 Done Sent Done false
20394 5803 Acked Read 10 Done Sent Done false
20395 5804 Acked Read 2 Done Sent Done false
20396 5805 Acked Read 1 Done Sent Done false
20397 5806 Acked Read 9 Done Sent Done false
20398 5807 Acked Flush 0 Done Sent Done false
20399 5808 Acked Write 2 Done Sent Done false
20400 5809 Acked Flush 0 Done Sent Done false
20401 5810 Acked Read 9 Done Sent Done false
20402 5811 Acked Read 1 Done Sent Done false
20403 5812 Acked Write 2 Done Sent Done false
20404 5813 Acked Read 4 Done Sent Done false
20405 5814 Acked Read 7 Done Sent Done false
20406 5815 Acked Write 3 Done Sent Done false
20407 5816 Acked Write 7 Done Sent Done false
20408 5817 Acked Write 3 Done Sent Done false
20409 5818 Acked Read 4 Done Sent Done false
20410 5819 Acked Read 9 Done Sent Done false
20411 5820 Acked Read 4 Done Sent Done false
20412 5821 Acked Read 8 Done Sent Done false
20413 5822 Acked Read 9 Done Sent Done false
20414 5823 Acked Write 2 Done Sent Done false
20415 5824 Acked Read 8 Done Sent Done false
20416 5825 Acked Flush 0 Done Sent Done false
20417 5826 Acked Write 10 Done Sent Done false
20418 5827 Acked Write 3 Done Sent Done false
20419 5828 Acked Flush 0 Done Sent Done false
20420 5829 Acked Flush 0 Done Sent Done false
20421 STATES DS:0 DS:1 DS:2 TOTAL
20422 Sent 0 301 0 301
20423 Done 301 0 301 602
20424 Skipped 0 0 0 0
20425 Error 0 0 0 0
20426 Last Flush: Some(JobId(5829)) Some(JobId(5528)) Some(JobId(5829))
20427 Downstairs last five completed: 5528 5527 5526 5525 5524
20428 Upstairs last five completed: 5829 5828 5827 5826 5825
20429 CLIENT: Up:0 ds:301 act:2
204302025-03-17T22:23:31.394ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
204312025-03-17T22:23:31.907ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
204322025-03-17T22:23:31.907ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
204332025-03-17T22:23:31.907ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
204342025-03-17T22:23:31.907ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
204352025-03-17T22:23:31.907ZINFOcrucible: client task is exiting = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
204362025-03-17T22:23:32.408ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
204372025-03-17T22:23:33.012ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
20438 [8820] got start action
20439 [8820] I am going to restart
20440 Starting downstairs at port 8820
20441 Make output file at "/tmp/dsc/downstairs-8820.txt"
20442 [8820][1] reports Starting
20443 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1227)
20444 [8820][1] reports Running
204452025-03-17T22:23:34.014ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
20446 ----------------------------------------------------------------
20447 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20448 States: Active Offline Active
20449 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20450 5529 Acked Write 3 Done Sent Done false
20451 5530 Acked Read 4 Done Sent Done false
20452 5531 Acked Write 6 Done Sent Done false
20453 5532 Acked Read 4 Done Sent Done false
20454 5533 Acked Read 8 Done Sent Done false
20455 5534 Acked Read 3 Done Sent Done false
20456 5535 Acked Write 1 Done Sent Done false
20457 5536 Acked Read 1 Done Sent Done false
20458 5537 Acked Write 7 Done Sent Done false
20459 5538 Acked Write 9 Done Sent Done false
20460 5539 Acked Write 10 Done Sent Done false
20461 5540 Acked Write 7 Done Sent Done false
20462 5541 Acked Write 6 Done Sent Done false
20463 5542 Acked Write 1 Done Sent Done false
20464 5543 Acked Read 6 Done Sent Done false
20465 5544 Acked Write 5 Done Sent Done false
20466 5545 Acked Write 8 Done Sent Done false
20467 5546 Acked Write 5 Done Sent Done false
20468 5547 Acked Read 4 Done Sent Done false
20469 5548 Acked Read 2 Done Sent Done false
20470 5549 Acked Write 7 Done Sent Done false
20471 5550 Acked Read 8 Done Sent Done false
20472 5551 Acked Write 3 Done Sent Done false
20473 5552 Acked Write 2 Done Sent Done false
20474 5553 Acked Read 9 Done Sent Done false
20475 5554 Acked Read 1 Done Sent Done false
20476 5555 Acked Read 1 Done Sent Done false
20477 5556 Acked Write 7 Done Sent Done false
20478 5557 Acked Write 5 Done Sent Done false
20479 5558 Acked Read 3 Done Sent Done false
20480 5559 Acked Read 6 Done Sent Done false
20481 5560 Acked Write 9 Done Sent Done false
20482 5561 Acked Flush 0 Done Sent Done false
20483 5562 Acked Write 5 Done Sent Done false
20484 5563 Acked Write 1 Done Sent Done false
20485 5564 Acked Write 3 Done Sent Done false
20486 5565 Acked Read 9 Done Sent Done false
20487 5566 Acked Flush 0 Done Sent Done false
20488 5567 Acked Write 3 Done Sent Done false
20489 5568 Acked Flush 0 Done Sent Done false
20490 5569 Acked Read 1 Done Sent Done false
20491 5570 Acked Flush 0 Done Sent Done false
20492 5571 Acked Read 2 Done Sent Done false
20493 5572 Acked Flush 0 Done Sent Done false
20494 5573 Acked Write 10 Done Sent Done false
20495 5574 Acked Write 9 Done Sent Done false
20496 5575 Acked Flush 0 Done Sent Done false
20497 5576 Acked Read 10 Done Sent Done false
20498 5577 Acked Read 2 Done Sent Done false
20499 5578 Acked Read 7 Done Sent Done false
20500 5579 Acked Flush 0 Done Sent Done false
20501 5580 Acked Read 3 Done Sent Done false
20502 5581 Acked Write 1 Done Sent Done false
20503 5582 Acked Write 6 Done Sent Done false
20504 5583 Acked Read 8 Done Sent Done false
20505 5584 Acked Read 2 Done Sent Done false
20506 5585 Acked Write 5 Done Sent Done false
20507 5586 Acked Write 9 Done Sent Done false
20508 5587 Acked Write 2 Done Sent Done false
20509 5588 Acked Read 3 Done Sent Done false
20510 5589 Acked Flush 0 Done Sent Done false
20511 5590 Acked Write 6 Done Sent Done false
20512 5591 Acked Read 7 Done Sent Done false
20513 5592 Acked Write 1 Done Sent Done false
20514 5593 Acked Flush 0 Done Sent Done false
20515 5594 Acked Read 1 Done Sent Done false
20516 5595 Acked Read 10 Done Sent Done false
20517 5596 Acked Read 3 Done Sent Done false
20518 5597 Acked Write 10 Done Sent Done false
20519 5598 Acked Read 4 Done Sent Done false
20520 5599 Acked Write 2 Done Sent Done false
20521 5600 Acked Read 6 Done Sent Done false
20522 5601 Acked Write 3 Done Sent Done false
20523 5602 Acked Read 9 Done Sent Done false
20524 5603 Acked Read 2 Done Sent Done false
20525 5604 Acked Flush 0 Done Sent Done false
20526 5605 Acked Read 6 Done Sent Done false
20527 5606 Acked Write 5 Done Sent Done false
20528 5607 Acked Read 2 Done Sent Done false
20529 5608 Acked Write 1 Done Sent Done false
20530 5609 Acked Write 10 Done Sent Done false
20531 5610 Acked Write 8 Done Sent Done false
20532 5611 Acked Flush 0 Done Sent Done false
20533 5612 Acked Write 3 Done Sent Done false
20534 5613 Acked Write 9 Done Sent Done false
20535 5614 Acked Write 9 Done Sent Done false
20536 5615 Acked Read 9 Done Sent Done false
20537 5616 Acked Write 5 Done Sent Done false
20538 5617 Acked Read 6 Done Sent Done false
20539 5618 Acked Read 6 Done Sent Done false
20540 5619 Acked Read 8 Done Sent Done false
20541 5620 Acked Read 6 Done Sent Done false
20542 5621 Acked Read 9 Done Sent Done false
20543 5622 Acked Write 3 Done Sent Done false
20544 5623 Acked Flush 0 Done Sent Done false
20545 5624 Acked Flush 0 Done Sent Done false
20546 5625 Acked Read 7 Done Sent Done false
20547 5626 Acked Write 3 Done Sent Done false
20548 5627 Acked Read 6 Done Sent Done false
20549 5628 Acked Read 7 Done Sent Done false
20550 5629 Acked Read 5 Done Sent Done false
20551 5630 Acked Write 7 Done Sent Done false
20552 5631 Acked Write 9 Done Sent Done false
20553 5632 Acked Flush 0 Done Sent Done false
20554 5633 Acked Flush 0 Done Sent Done false
20555 5634 Acked Write 3 Done Sent Done false
20556 5635 Acked Read 3 Done Sent Done false
20557 5636 Acked Write 9 Done Sent Done false
20558 5637 Acked Flush 0 Done Sent Done false
20559 5638 Acked Read 6 Done Sent Done false
20560 5639 Acked Write 5 Done Sent Done false
20561 5640 Acked Read 7 Done Sent Done false
20562 5641 Acked Write 8 Done Sent Done false
20563 5642 Acked Write 2 Done Sent Done false
20564 5643 Acked Read 5 Done Sent Done false
20565 5644 Acked Read 7 Done Sent Done false
20566 5645 Acked Write 10 Done Sent Done false
20567 5646 Acked Read 8 Done Sent Done false
20568 5647 Acked Write 5 Done Sent Done false
20569 5648 Acked Read 6 Done Sent Done false
20570 5649 Acked Read 7 Done Sent Done false
20571 5650 Acked Read 10 Done Sent Done false
20572 5651 Acked Write 10 Done Sent Done false
20573 5652 Acked Flush 0 Done Sent Done false
20574 5653 Acked Write 10 Done Sent Done false
20575 5654 Acked Read 1 Done Sent Done false
20576 5655 Acked Flush 0 Done Sent Done false
20577 5656 Acked Write 3 Done Sent Done false
20578 5657 Acked Write 10 Done Sent Done false
20579 5658 Acked Read 7 Done Sent Done false
20580 5659 Acked Flush 0 Done Sent Done false
20581 5660 Acked Write 4 Done Sent Done false
20582 5661 Acked Read 3 Done Sent Done false
20583 5662 Acked Read 4 Done Sent Done false
20584 5663 Acked Read 5 Done Sent Done false
20585 5664 Acked Read 8 Done Sent Done false
20586 5665 Acked Write 7 Done Sent Done false
20587 5666 Acked Read 6 Done Sent Done false
20588 5667 Acked Write 3 Done Sent Done false
20589 5668 Acked Read 5 Done Sent Done false
20590 5669 Acked Write 8 Done Sent Done false
20591 5670 Acked Write 5 Done Sent Done false
20592 5671 Acked Write 9 Done Sent Done false
20593 5672 Acked Read 2 Done Sent Done false
20594 5673 Acked Read 6 Done Sent Done false
20595 5674 Acked Read 2 Done Sent Done false
20596 5675 Acked Write 2 Done Sent Done false
20597 5676 Acked Read 9 Done Sent Done false
20598 5677 Acked Read 2 Done Sent Done false
20599 5678 Acked Read 4 Done Sent Done false
20600 5679 Acked Write 4 Done Sent Done false
20601 5680 Acked Flush 0 Done Sent Done false
20602 5681 Acked Write 7 Done Sent Done false
20603 5682 Acked Read 3 Done Sent Done false
20604 5683 Acked Write 5 Done Sent Done false
20605 5684 Acked Write 6 Done Sent Done false
20606 5685 Acked Write 4 Done Sent Done false
20607 5686 Acked Read 4 Done Sent Done false
20608 5687 Acked Read 9 Done Sent Done false
20609 5688 Acked Read 3 Done Sent Done false
20610 5689 Acked Write 10 Done Sent Done false
20611 5690 Acked Write 3 Done Sent Done false
20612 5691 Acked Write 7 Done Sent Done false
20613 5692 Acked Read 10 Done Sent Done false
20614 5693 Acked Write 9 Done Sent Done false
20615 5694 Acked Read 10 Done Sent Done false
20616 5695 Acked Read 1 Done Sent Done false
20617 5696 Acked Write 1 Done Sent Done false
20618 5697 Acked Read 10 Done Sent Done false
20619 5698 Acked Write 6 Done Sent Done false
20620 5699 Acked Read 10 Done Sent Done false
20621 5700 Acked Write 2 Done Sent Done false
20622 5701 Acked Write 7 Done Sent Done false
20623 5702 Acked Read 7 Done Sent Done false
20624 5703 Acked Read 6 Done Sent Done false
20625 5704 Acked Read 7 Done Sent Done false
20626 5705 Acked Read 5 Done Sent Done false
20627 5706 Acked Read 1 Done Sent Done false
20628 5707 Acked Read 8 Done Sent Done false
20629 5708 Acked Read 9 Done Sent Done false
20630 5709 Acked Read 3 Done Sent Done false
20631 5710 Acked Write 7 Done Sent Done false
20632 5711 Acked Write 5 Done Sent Done false
20633 5712 Acked Read 7 Done Sent Done false
20634 5713 Acked Write 9 Done Sent Done false
20635 5714 Acked Read 7 Done Sent Done false
20636 5715 Acked Write 10 Done Sent Done false
20637 5716 Acked Read 3 Done Sent Done false
20638 5717 Acked Read 3 Done Sent Done false
20639 5718 Acked Read 6 Done Sent Done false
20640 5719 Acked Write 10 Done Sent Done false
20641 5720 Acked Read 6 Done Sent Done false
20642 5721 Acked Write 3 Done Sent Done false
20643 5722 Acked Read 8 Done Sent Done false
20644 5723 Acked Read 3 Done Sent Done false
20645 5724 Acked Flush 0 Done Sent Done false
20646 5725 Acked Read 3 Done Sent Done false
20647 5726 Acked Write 9 Done Sent Done false
20648 5727 Acked Read 2 Done Sent Done false
20649 5728 Acked Read 7 Done Sent Done false
20650 5729 Acked Flush 0 Done Sent Done false
20651 5730 Acked Read 10 Done Sent Done false
20652 5731 Acked Write 4 Done Sent Done false
20653 5732 Acked Read 10 Done Sent Done false
20654 5733 Acked Flush 0 Done Sent Done false
20655 5734 Acked Write 7 Done Sent Done false
20656 5735 Acked Read 10 Done Sent Done false
20657 5736 Acked Read 6 Done Sent Done false
20658 5737 Acked Read 8 Done Sent Done false
20659 5738 Acked Read 7 Done Sent Done false
20660 5739 Acked Read 8 Done Sent Done false
20661 5740 Acked Read 2 Done Sent Done false
20662 5741 Acked Write 9 Done Sent Done false
20663 5742 Acked Read 2 Done Sent Done false
20664 5743 Acked Read 1 Done Sent Done false
20665 5744 Acked Read 6 Done Sent Done false
20666 5745 Acked Read 6 Done Sent Done false
20667 5746 Acked Write 4 Done Sent Done false
20668 5747 Acked Read 4 Done Sent Done false
20669 5748 Acked Write 8 Done Sent Done false
20670 5749 Acked Write 5 Done Sent Done false
20671 5750 Acked Write 3 Done Sent Done false
20672 5751 Acked Read 3 Done Sent Done false
20673 5752 Acked Write 4 Done Sent Done false
20674 5753 Acked Read 6 Done Sent Done false
20675 5754 Acked Read 8 Done Sent Done false
20676 5755 Acked Read 10 Done Sent Done false
20677 5756 Acked Read 9 Done Sent Done false
20678 5757 Acked Read 10 Done Sent Done false
20679 5758 Acked Write 6 Done Sent Done false
20680 5759 Acked Read 3 Done Sent Done false
20681 5760 Acked Write 4 Done Sent Done false
20682 5761 Acked Write 6 Done Sent Done false
20683 5762 Acked Write 4 Done Sent Done false
20684 5763 Acked Read 7 Done Sent Done false
20685 5764 Acked Read 2 Done Sent Done false
20686 5765 Acked Read 10 Done Sent Done false
20687 5766 Acked Read 6 Done Sent Done false
20688 5767 Acked Read 8 Done Sent Done false
20689 5768 Acked Write 2 Done Sent Done false
20690 5769 Acked Read 7 Done Sent Done false
20691 5770 Acked Flush 0 Done Sent Done false
20692 5771 Acked Read 9 Done Sent Done false
20693 5772 Acked Read 4 Done Sent Done false
20694 5773 Acked Read 8 Done Sent Done false
20695 5774 Acked Read 10 Done Sent Done false
20696 5775 Acked Read 10 Done Sent Done false
20697 5776 Acked Read 7 Done Sent Done false
20698 5777 Acked Read 3 Done Sent Done false
20699 5778 Acked Write 4 Done Sent Done false
20700 5779 Acked Flush 0 Done Sent Done false
20701 5780 Acked Write 9 Done Sent Done false
20702 5781 Acked Write 3 Done Sent Done false
20703 5782 Acked Flush 0 Done Sent Done false
20704 5783 Acked Read 6 Done Sent Done false
20705 5784 Acked Write 10 Done Sent Done false
20706 5785 Acked Write 8 Done Sent Done false
20707 5786 Acked Write 7 Done Sent Done false
20708 5787 Acked Read 1 Done Sent Done false
20709 5788 Acked Read 3 Done Sent Done false
20710 5789 Acked Read 4 Done Sent Done false
20711 5790 Acked Write 3 Done Sent Done false
20712 5791 Acked Read 5 Done Sent Done false
20713 5792 Acked Flush 0 Done Sent Done false
20714 5793 Acked Read 8 Done Sent Done false
20715 5794 Acked Read 1 Done Sent Done false
20716 5795 Acked Write 6 Done Sent Done false
20717 5796 Acked Write 6 Done Sent Done false
20718 5797 Acked Write 5 Done Sent Done false
20719 5798 Acked Read 4 Done Sent Done false
20720 5799 Acked Read 10 Done Sent Done false
20721 5800 Acked Write 3 Done Sent Done false
20722 5801 Acked Read 6 Done Sent Done false
20723 5802 Acked Write 4 Done Sent Done false
20724 5803 Acked Read 10 Done Sent Done false
20725 5804 Acked Read 2 Done Sent Done false
20726 5805 Acked Read 1 Done Sent Done false
20727 5806 Acked Read 9 Done Sent Done false
20728 5807 Acked Flush 0 Done Sent Done false
20729 5808 Acked Write 2 Done Sent Done false
20730 5809 Acked Flush 0 Done Sent Done false
20731 5810 Acked Read 9 Done Sent Done false
20732 5811 Acked Read 1 Done Sent Done false
20733 5812 Acked Write 2 Done Sent Done false
20734 5813 Acked Read 4 Done Sent Done false
20735 5814 Acked Read 7 Done Sent Done false
20736 5815 Acked Write 3 Done Sent Done false
20737 5816 Acked Write 7 Done Sent Done false
20738 5817 Acked Write 3 Done Sent Done false
20739 5818 Acked Read 4 Done Sent Done false
20740 5819 Acked Read 9 Done Sent Done false
20741 5820 Acked Read 4 Done Sent Done false
20742 5821 Acked Read 8 Done Sent Done false
20743 5822 Acked Read 9 Done Sent Done false
20744 5823 Acked Write 2 Done Sent Done false
20745 5824 Acked Read 8 Done Sent Done false
20746 5825 Acked Flush 0 Done Sent Done false
20747 5826 Acked Write 10 Done Sent Done false
20748 5827 Acked Write 3 Done Sent Done false
20749 5828 Acked Flush 0 Done Sent Done false
20750 5829 Acked Flush 0 Done Sent Done false
20751 STATES DS:0 DS:1 DS:2 TOTAL
20752 Sent 0 301 0 301
20753 Done 301 0 301 602
20754 Skipped 0 0 0 0
20755 Error 0 0 0 0
20756 Last Flush: Some(JobId(5829)) Some(JobId(5528)) Some(JobId(5829))
20757 Downstairs last five completed: 5528 5527 5526 5525 5524
20758 Upstairs last five completed: 5829 5828 5827 5826 5825
20759 CLIENT: Up:0 ds:301 act:2
207602025-03-17T22:23:35.015ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
207612025-03-17T22:23:36.395ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
207622025-03-17T22:23:37.018ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
207632025-03-17T22:23:38.020ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
20764 ----------------------------------------------------------------
20765 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20766 States: Active Offline Active
20767 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20768 5529 Acked Write 3 Done Sent Done false
20769 5530 Acked Read 4 Done Sent Done false
20770 5531 Acked Write 6 Done Sent Done false
20771 5532 Acked Read 4 Done Sent Done false
20772 5533 Acked Read 8 Done Sent Done false
20773 5534 Acked Read 3 Done Sent Done false
20774 5535 Acked Write 1 Done Sent Done false
20775 5536 Acked Read 1 Done Sent Done false
20776 5537 Acked Write 7 Done Sent Done false
20777 5538 Acked Write 9 Done Sent Done false
20778 5539 Acked Write 10 Done Sent Done false
20779 5540 Acked Write 7 Done Sent Done false
20780 5541 Acked Write 6 Done Sent Done false
20781 5542 Acked Write 1 Done Sent Done false
20782 5543 Acked Read 6 Done Sent Done false
20783 5544 Acked Write 5 Done Sent Done false
20784 5545 Acked Write 8 Done Sent Done false
20785 5546 Acked Write 5 Done Sent Done false
20786 5547 Acked Read 4 Done Sent Done false
20787 5548 Acked Read 2 Done Sent Done false
20788 5549 Acked Write 7 Done Sent Done false
20789 5550 Acked Read 8 Done Sent Done false
20790 5551 Acked Write 3 Done Sent Done false
20791 5552 Acked Write 2 Done Sent Done false
20792 5553 Acked Read 9 Done Sent Done false
20793 5554 Acked Read 1 Done Sent Done false
20794 5555 Acked Read 1 Done Sent Done false
20795 5556 Acked Write 7 Done Sent Done false
20796 5557 Acked Write 5 Done Sent Done false
20797 5558 Acked Read 3 Done Sent Done false
20798 5559 Acked Read 6 Done Sent Done false
20799 5560 Acked Write 9 Done Sent Done false
20800 5561 Acked Flush 0 Done Sent Done false
20801 5562 Acked Write 5 Done Sent Done false
20802 5563 Acked Write 1 Done Sent Done false
20803 5564 Acked Write 3 Done Sent Done false
20804 5565 Acked Read 9 Done Sent Done false
20805 5566 Acked Flush 0 Done Sent Done false
20806 5567 Acked Write 3 Done Sent Done false
20807 5568 Acked Flush 0 Done Sent Done false
20808 5569 Acked Read 1 Done Sent Done false
20809 5570 Acked Flush 0 Done Sent Done false
20810 5571 Acked Read 2 Done Sent Done false
20811 5572 Acked Flush 0 Done Sent Done false
20812 5573 Acked Write 10 Done Sent Done false
20813 5574 Acked Write 9 Done Sent Done false
20814 5575 Acked Flush 0 Done Sent Done false
20815 5576 Acked Read 10 Done Sent Done false
20816 5577 Acked Read 2 Done Sent Done false
20817 5578 Acked Read 7 Done Sent Done false
20818 5579 Acked Flush 0 Done Sent Done false
20819 5580 Acked Read 3 Done Sent Done false
20820 5581 Acked Write 1 Done Sent Done false
20821 5582 Acked Write 6 Done Sent Done false
20822 5583 Acked Read 8 Done Sent Done false
20823 5584 Acked Read 2 Done Sent Done false
20824 5585 Acked Write 5 Done Sent Done false
20825 5586 Acked Write 9 Done Sent Done false
20826 5587 Acked Write 2 Done Sent Done false
20827 5588 Acked Read 3 Done Sent Done false
20828 5589 Acked Flush 0 Done Sent Done false
20829 5590 Acked Write 6 Done Sent Done false
20830 5591 Acked Read 7 Done Sent Done false
20831 5592 Acked Write 1 Done Sent Done false
20832 5593 Acked Flush 0 Done Sent Done false
20833 5594 Acked Read 1 Done Sent Done false
20834 5595 Acked Read 10 Done Sent Done false
20835 5596 Acked Read 3 Done Sent Done false
20836 5597 Acked Write 10 Done Sent Done false
20837 5598 Acked Read 4 Done Sent Done false
20838 5599 Acked Write 2 Done Sent Done false
20839 5600 Acked Read 6 Done Sent Done false
20840 5601 Acked Write 3 Done Sent Done false
20841 5602 Acked Read 9 Done Sent Done false
20842 5603 Acked Read 2 Done Sent Done false
20843 5604 Acked Flush 0 Done Sent Done false
20844 5605 Acked Read 6 Done Sent Done false
20845 5606 Acked Write 5 Done Sent Done false
20846 5607 Acked Read 2 Done Sent Done false
20847 5608 Acked Write 1 Done Sent Done false
20848 5609 Acked Write 10 Done Sent Done false
20849 5610 Acked Write 8 Done Sent Done false
20850 5611 Acked Flush 0 Done Sent Done false
20851 5612 Acked Write 3 Done Sent Done false
20852 5613 Acked Write 9 Done Sent Done false
20853 5614 Acked Write 9 Done Sent Done false
20854 5615 Acked Read 9 Done Sent Done false
20855 5616 Acked Write 5 Done Sent Done false
20856 5617 Acked Read 6 Done Sent Done false
20857 5618 Acked Read 6 Done Sent Done false
20858 5619 Acked Read 8 Done Sent Done false
20859 5620 Acked Read 6 Done Sent Done false
20860 5621 Acked Read 9 Done Sent Done false
20861 5622 Acked Write 3 Done Sent Done false
20862 5623 Acked Flush 0 Done Sent Done false
20863 5624 Acked Flush 0 Done Sent Done false
20864 5625 Acked Read 7 Done Sent Done false
20865 5626 Acked Write 3 Done Sent Done false
20866 5627 Acked Read 6 Done Sent Done false
20867 5628 Acked Read 7 Done Sent Done false
20868 5629 Acked Read 5 Done Sent Done false
20869 5630 Acked Write 7 Done Sent Done false
20870 5631 Acked Write 9 Done Sent Done false
20871 5632 Acked Flush 0 Done Sent Done false
20872 5633 Acked Flush 0 Done Sent Done false
20873 5634 Acked Write 3 Done Sent Done false
20874 5635 Acked Read 3 Done Sent Done false
20875 5636 Acked Write 9 Done Sent Done false
20876 5637 Acked Flush 0 Done Sent Done false
20877 5638 Acked Read 6 Done Sent Done false
20878 5639 Acked Write 5 Done Sent Done false
20879 5640 Acked Read 7 Done Sent Done false
20880 5641 Acked Write 8 Done Sent Done false
20881 5642 Acked Write 2 Done Sent Done false
20882 5643 Acked Read 5 Done Sent Done false
20883 5644 Acked Read 7 Done Sent Done false
20884 5645 Acked Write 10 Done Sent Done false
20885 5646 Acked Read 8 Done Sent Done false
20886 5647 Acked Write 5 Done Sent Done false
20887 5648 Acked Read 6 Done Sent Done false
20888 5649 Acked Read 7 Done Sent Done false
20889 5650 Acked Read 10 Done Sent Done false
20890 5651 Acked Write 10 Done Sent Done false
20891 5652 Acked Flush 0 Done Sent Done false
20892 5653 Acked Write 10 Done Sent Done false
20893 5654 Acked Read 1 Done Sent Done false
20894 5655 Acked Flush 0 Done Sent Done false
20895 5656 Acked Write 3 Done Sent Done false
20896 5657 Acked Write 10 Done Sent Done false
20897 5658 Acked Read 7 Done Sent Done false
20898 5659 Acked Flush 0 Done Sent Done false
20899 5660 Acked Write 4 Done Sent Done false
20900 5661 Acked Read 3 Done Sent Done false
20901 5662 Acked Read 4 Done Sent Done false
20902 5663 Acked Read 5 Done Sent Done false
20903 5664 Acked Read 8 Done Sent Done false
20904 5665 Acked Write 7 Done Sent Done false
20905 5666 Acked Read 6 Done Sent Done false
20906 5667 Acked Write 3 Done Sent Done false
20907 5668 Acked Read 5 Done Sent Done false
20908 5669 Acked Write 8 Done Sent Done false
20909 5670 Acked Write 5 Done Sent Done false
20910 5671 Acked Write 9 Done Sent Done false
20911 5672 Acked Read 2 Done Sent Done false
20912 5673 Acked Read 6 Done Sent Done false
20913 5674 Acked Read 2 Done Sent Done false
20914 5675 Acked Write 2 Done Sent Done false
20915 5676 Acked Read 9 Done Sent Done false
20916 5677 Acked Read 2 Done Sent Done false
20917 5678 Acked Read 4 Done Sent Done false
20918 5679 Acked Write 4 Done Sent Done false
20919 5680 Acked Flush 0 Done Sent Done false
20920 5681 Acked Write 7 Done Sent Done false
20921 5682 Acked Read 3 Done Sent Done false
20922 5683 Acked Write 5 Done Sent Done false
20923 5684 Acked Write 6 Done Sent Done false
20924 5685 Acked Write 4 Done Sent Done false
20925 5686 Acked Read 4 Done Sent Done false
20926 5687 Acked Read 9 Done Sent Done false
20927 5688 Acked Read 3 Done Sent Done false
20928 5689 Acked Write 10 Done Sent Done false
20929 5690 Acked Write 3 Done Sent Done false
20930 5691 Acked Write 7 Done Sent Done false
20931 5692 Acked Read 10 Done Sent Done false
20932 5693 Acked Write 9 Done Sent Done false
20933 5694 Acked Read 10 Done Sent Done false
20934 5695 Acked Read 1 Done Sent Done false
20935 5696 Acked Write 1 Done Sent Done false
20936 5697 Acked Read 10 Done Sent Done false
20937 5698 Acked Write 6 Done Sent Done false
20938 5699 Acked Read 10 Done Sent Done false
20939 5700 Acked Write 2 Done Sent Done false
20940 5701 Acked Write 7 Done Sent Done false
20941 5702 Acked Read 7 Done Sent Done false
20942 5703 Acked Read 6 Done Sent Done false
20943 5704 Acked Read 7 Done Sent Done false
20944 5705 Acked Read 5 Done Sent Done false
20945 5706 Acked Read 1 Done Sent Done false
20946 5707 Acked Read 8 Done Sent Done false
20947 5708 Acked Read 9 Done Sent Done false
20948 5709 Acked Read 3 Done Sent Done false
20949 5710 Acked Write 7 Done Sent Done false
20950 5711 Acked Write 5 Done Sent Done false
20951 5712 Acked Read 7 Done Sent Done false
20952 5713 Acked Write 9 Done Sent Done false
20953 5714 Acked Read 7 Done Sent Done false
20954 5715 Acked Write 10 Done Sent Done false
20955 5716 Acked Read 3 Done Sent Done false
20956 5717 Acked Read 3 Done Sent Done false
20957 5718 Acked Read 6 Done Sent Done false
20958 5719 Acked Write 10 Done Sent Done false
20959 5720 Acked Read 6 Done Sent Done false
20960 5721 Acked Write 3 Done Sent Done false
20961 5722 Acked Read 8 Done Sent Done false
20962 5723 Acked Read 3 Done Sent Done false
20963 5724 Acked Flush 0 Done Sent Done false
20964 5725 Acked Read 3 Done Sent Done false
20965 5726 Acked Write 9 Done Sent Done false
20966 5727 Acked Read 2 Done Sent Done false
20967 5728 Acked Read 7 Done Sent Done false
20968 5729 Acked Flush 0 Done Sent Done false
20969 5730 Acked Read 10 Done Sent Done false
20970 5731 Acked Write 4 Done Sent Done false
20971 5732 Acked Read 10 Done Sent Done false
20972 5733 Acked Flush 0 Done Sent Done false
20973 5734 Acked Write 7 Done Sent Done false
20974 5735 Acked Read 10 Done Sent Done false
20975 5736 Acked Read 6 Done Sent Done false
20976 5737 Acked Read 8 Done Sent Done false
20977 5738 Acked Read 7 Done Sent Done false
20978 5739 Acked Read 8 Done Sent Done false
20979 5740 Acked Read 2 Done Sent Done false
20980 5741 Acked Write 9 Done Sent Done false
20981 5742 Acked Read 2 Done Sent Done false
20982 5743 Acked Read 1 Done Sent Done false
20983 5744 Acked Read 6 Done Sent Done false
20984 5745 Acked Read 6 Done Sent Done false
20985 5746 Acked Write 4 Done Sent Done false
20986 5747 Acked Read 4 Done Sent Done false
20987 5748 Acked Write 8 Done Sent Done false
20988 5749 Acked Write 5 Done Sent Done false
20989 5750 Acked Write 3 Done Sent Done false
20990 5751 Acked Read 3 Done Sent Done false
20991 5752 Acked Write 4 Done Sent Done false
20992 5753 Acked Read 6 Done Sent Done false
20993 5754 Acked Read 8 Done Sent Done false
20994 5755 Acked Read 10 Done Sent Done false
20995 5756 Acked Read 9 Done Sent Done false
20996 5757 Acked Read 10 Done Sent Done false
20997 5758 Acked Write 6 Done Sent Done false
20998 5759 Acked Read 3 Done Sent Done false
20999 5760 Acked Write 4 Done Sent Done false
21000 5761 Acked Write 6 Done Sent Done false
21001 5762 Acked Write 4 Done Sent Done false
21002 5763 Acked Read 7 Done Sent Done false
21003 5764 Acked Read 2 Done Sent Done false
21004 5765 Acked Read 10 Done Sent Done false
21005 5766 Acked Read 6 Done Sent Done false
21006 5767 Acked Read 8 Done Sent Done false
21007 5768 Acked Write 2 Done Sent Done false
21008 5769 Acked Read 7 Done Sent Done false
21009 5770 Acked Flush 0 Done Sent Done false
21010 5771 Acked Read 9 Done Sent Done false
21011 5772 Acked Read 4 Done Sent Done false
21012 5773 Acked Read 8 Done Sent Done false
21013 5774 Acked Read 10 Done Sent Done false
21014 5775 Acked Read 10 Done Sent Done false
21015 5776 Acked Read 7 Done Sent Done false
21016 5777 Acked Read 3 Done Sent Done false
21017 5778 Acked Write 4 Done Sent Done false
21018 5779 Acked Flush 0 Done Sent Done false
21019 5780 Acked Write 9 Done Sent Done false
21020 5781 Acked Write 3 Done Sent Done false
21021 5782 Acked Flush 0 Done Sent Done false
21022 5783 Acked Read 6 Done Sent Done false
21023 5784 Acked Write 10 Done Sent Done false
21024 5785 Acked Write 8 Done Sent Done false
21025 5786 Acked Write 7 Done Sent Done false
21026 5787 Acked Read 1 Done Sent Done false
21027 5788 Acked Read 3 Done Sent Done false
21028 5789 Acked Read 4 Done Sent Done false
21029 5790 Acked Write 3 Done Sent Done false
21030 5791 Acked Read 5 Done Sent Done false
21031 5792 Acked Flush 0 Done Sent Done false
21032 5793 Acked Read 8 Done Sent Done false
21033 5794 Acked Read 1 Done Sent Done false
21034 5795 Acked Write 6 Done Sent Done false
21035 5796 Acked Write 6 Done Sent Done false
21036 5797 Acked Write 5 Done Sent Done false
21037 5798 Acked Read 4 Done Sent Done false
21038 5799 Acked Read 10 Done Sent Done false
21039 5800 Acked Write 3 Done Sent Done false
21040 5801 Acked Read 6 Done Sent Done false
21041 5802 Acked Write 4 Done Sent Done false
21042 5803 Acked Read 10 Done Sent Done false
21043 5804 Acked Read 2 Done Sent Done false
21044 5805 Acked Read 1 Done Sent Done false
21045 5806 Acked Read 9 Done Sent Done false
21046 5807 Acked Flush 0 Done Sent Done false
21047 5808 Acked Write 2 Done Sent Done false
21048 5809 Acked Flush 0 Done Sent Done false
21049 5810 Acked Read 9 Done Sent Done false
21050 5811 Acked Read 1 Done Sent Done false
21051 5812 Acked Write 2 Done Sent Done false
21052 5813 Acked Read 4 Done Sent Done false
21053 5814 Acked Read 7 Done Sent Done false
21054 5815 Acked Write 3 Done Sent Done false
21055 5816 Acked Write 7 Done Sent Done false
21056 5817 Acked Write 3 Done Sent Done false
21057 5818 Acked Read 4 Done Sent Done false
21058 5819 Acked Read 9 Done Sent Done false
21059 5820 Acked Read 4 Done Sent Done false
21060 5821 Acked Read 8 Done Sent Done false
21061 5822 Acked Read 9 Done Sent Done false
21062 5823 Acked Write 2 Done Sent Done false
21063 5824 Acked Read 8 Done Sent Done false
21064 5825 Acked Flush 0 Done Sent Done false
21065 5826 Acked Write 10 Done Sent Done false
21066 5827 Acked Write 3 Done Sent Done false
21067 5828 Acked Flush 0 Done Sent Done false
21068 5829 Acked Flush 0 Done Sent Done false
21069 STATES DS:0 DS:1 DS:2 TOTAL
21070 Sent 0 301 0 301
21071 Done 301 0 301 602
21072 Skipped 0 0 0 0
21073 Error 0 0 0 0
21074 Last Flush: Some(JobId(5829)) Some(JobId(5528)) Some(JobId(5829))
21075 Downstairs last five completed: 5528 5527 5526 5525 5524
21076 Upstairs last five completed: 5829 5828 5827 5826 5825
21077 CLIENT: Up:0 ds:301 act:2
210782025-03-17T22:23:39.021ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
210792025-03-17T22:23:40.023ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
210802025-03-17T22:23:41.396ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
210812025-03-17T22:23:41.908ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
210822025-03-17T22:23:41.908ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:65264) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
210832025-03-17T22:23:41.909ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
210842025-03-17T22:23:41.910ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
210852025-03-17T22:23:41.910ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
210862025-03-17T22:23:41.910ZINFOcrucible: send last flush ID to this DS: Some(JobId(5528)) = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
210872025-03-17T22:23:41.910ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
210882025-03-17T22:23:41.910ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(5528)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
210892025-03-17T22:23:41.910ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(5528)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
21090 ----------------------------------------------------------------
21091 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
21092 Downstairs last five completed: 5829 5828 5827 5826 5825
21093 Upstairs last five completed: 5829 5828 5827 5826 5825
21094 CLIENT: Up:0 ds:0 act:3
21095 Replay: All jobs finished
21096 Main task has work to do, go find it
21097 got dsc Stop(2)
21098 stop 2
21099 Mar 17 22:23:42.171 INFO request completed, latency_us: 343, response_code: 204, uri: /stop/cid/2, method: POST, req_id:[8830] Got stop action so:false kr:false
21100 3b7b3a44-7fdb-4d13-9d10-de03e6604e97, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
211012025-03-17T22:23:42.172ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
211022025-03-17T22:23:42.172ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
211032025-03-17T22:23:42.172ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
211042025-03-17T22:23:42.172ZINFOcrucible: client task is exiting = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
21105 Mar 17 22:23:42.172 INFO request completed, latency_us: 153, response_code: 200, uri: /state/cid/2, method: GET, req_id: 48bc9cfa-ba11-455d-b2c5-0e06ede498d1, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
21106 [8830] Exited with: ExitStatus(unix_wait_status(9))
21107 [8830][2] reports Exit
21108 Mar 17 22:23:46.174 INFO request completed, latency_us: 209, response_code: 200, uri: /state/cid/2, method: GET, req_id: 2612fc34-2bac-4187-a519-9f5781e9c8d7, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
21109 001/300 Flush
21110 002/300 Write block 2186 len 8192 data: 2 4
21111 003/300 Read block 3687 len 8192
21112 004/300 Read block 1119 len 40960
21113 005/300 Read block 556 len 28672
21114 006/300 Write block 1943 len 36864 data: 2 2 2 2 3 3 3 3 3
21115 007/300 Write block 2135 len 36864 data: 2 2 3 3 4 5 7 6 5
21116 008/300 Read block 2882 len 8192
21117 009/300 Read block 1058 len 20480
21118 010/300 Write block 3802 len 36864 data: 4 4 3 3 5 5 5 5 5
21119 011/300 Read block 3542 len 4096
21120 012/300 Write block 3226 len 40960 data: 3 4 3 3 4 4 5 4 4 3
21121 013/300 Read block 1549 len 20480
21122 014/300 Read block 1039 len 24576
21123 015/300 Write block 2955 len 32768 data: 4 3 2 2 4 5 4 5
21124 016/300 Read block 252 len 12288
21125 017/300 Read block 3928 len 32768
21126 018/300 Write block 2660 len 32768 data: 3 3 3 3 3 3 3 4
21127 019/300 Write block 422 len 12288 data: 6 6 5
21128 020/300 Flush
21129 021/300 Read block 4943 len 24576
21130 022/300 Flush
21131 023/300 Flush
21132 024/300 Read block 1772 len 16384
21133 025/300 Read block 2118 len 36864
21134 026/300 Write block 2857 len 20480 data: 4 5 5 5 5
21135 027/300 Read block 1716 len 28672
21136 028/300 Read block 2832 len 24576
21137 029/300 Write block 2957 len 16384 data: 3 3 5 6
21138 030/300 Write block 4959 len 36864 data: 2 3 5 5 5 3 4 4 4
21139 031/300 Write block 2460 len 8192 data: 3 2
21140 032/300 Read block 1889 len 40960
21141 033/300 Write block 4736 len 28672 data: 5 6 4 6 6 6 4
21142 034/300 Read block 847 len 4096
21143 035/300 Write block 3180 len 16384 data: 6 6 5 4
21144 036/300 Read block 4648 len 8192
21145 037/300 Write block 3843 len 40960 data: 3 3 2 2 2 4 5 6 5 7
21146 038/300 Flush
21147 039/300 Read block 4815 len 4096
21148 040/300 Read block 2532 len 4096
21149 041/300 Read block 2090 len 8192
21150 042/300 Write block 2596 len 32768 data: 3 5 5 5 5 4 7 7
21151 043/300 Write block 3426 len 40960 data: 5 4 5 5 5 5 4 4 4 4
21152 044/300 Flush
21153 045/300 Write block 4243 len 40960 data: 4 4 4 3 4 4 4 3 4 5
21154 046/300 Flush
21155 047/300 Read block 1843 len 24576
21156 048/300 Write block 260 len 8192 data: 2 2
21157 049/300 Read block 1221 len 4096
21158 050/300 Flush
21159 051/300 Read block 4330 len 8192
21160 052/300 Read block 2040 len 8192
21161 053/300 Read block 2217 len 20480
21162 054/300 Read block 4745 len 40960
21163 055/300 Read block 2751 len 8192
21164 056/300 Flush
21165 057/300 Write block 446 len 24576 data: 5 6 5 6 7 7
21166 058/300 Read block 4470 len 32768
21167 059/300 Read block 3366 len 32768
21168 060/300 Write block 897 len 40960 data: 3 3 3 3 3 3 4 3 4 4
21169 061/300 Read block 266 len 8192
21170 062/300 Flush
21171 063/300 Read block 4401 len 20480
21172 064/300 Write block 1695 len 4096 data: 4
21173 065/300 Write block 4042 len 8192 data: 2 2
21174 066/300 Write block 908 len 16384 data: 6 6 7 7
21175 067/300 Read block 3980 len 28672
21176 068/300 Flush
21177 069/300 Read block 1891 len 12288
21178 070/300 Write block 4592 len 40960 data: 5 3 3 4 4 4 4 3 3 3
21179 071/300 Read block 3662 len 40960
21180 072/300 Write block 2555 len 32768 data: 6 5 6 6 6 5 4 3
21181 073/300 Write block 99 len 32768 data: 4 4 4 4 4 3 3 4
21182 074/300 Read block 3170 len 20480
21183 075/300 Write block 1123 len 24576 data: 4 4 4 5 5 5
21184 076/300 Flush
21185 077/300 Read block 2397 len 8192
21186 078/300 Flush
21187 079/300 Flush
21188 080/300 Write block 4700 len 8192 data: 4 4
21189 081/300 Read block 158 len 4096
21190 082/300 Read block 4377 len 20480
21191 083/300 Read block 3204 len 36864
21192 084/300 Read block 1301 len 8192
21193 085/300 Write block 2630 len 4096 data: 3
21194 086/300 Read block 2766 len 4096
21195 087/300 Flush
21196 088/300 Flush
21197 089/300 Write block 921 len 4096 data: 5
21198 090/300 Flush
21199 091/300 Read block 1323 len 40960
21200 092/300 Write block 3914 len 8192 data: 2 3
21201 093/300 Read block 395 len 20480
21202 094/300 Write block 1918 len 8192 data: 5 6
21203 095/300 Write block 2924 len 28672 data: 5 5 4 4 4 4 3
21204 096/300 Write block 4617 len 24576 data: 2 2 2 3 3 3
21205 097/300 Flush
21206 098/300 Flush
21207 099/300 Read block 2879 len 24576
21208 100/300 Read block 4734 len 12288
21209 101/300 Read block 1709 len 36864
21210 102/300 Read block 993 len 24576
21211 103/300 Read block 1833 len 4096
21212 104/300 Read block 4034 len 4096
21213 105/300 Flush
21214 106/300 Write block 3179 len 12288 data: 5 7 7
21215 107/300 Write block 408 len 4096 data: 4
21216 108/300 Write block 4664 len 32768 data: 5 4 5 5 4 5 5 5
21217 109/300 Write block 4120 len 16384 data: 4 4 3 4
21218 110/300 Write block 1019 len 24576 data: 3 3 2 2 2 2
21219 111/300 Write block 836 len 40960 data: 3 3 4 4 4 3 3 3 3 3
21220 112/300 Write block 3845 len 24576 data: 3 3 3 5 6 7
21221 113/300 Read block 890 len 36864
21222 114/300 Write block 537 len 16384 data: 4 4 5 4
21223 115/300 Write block 3518 len 40960 data: 5 6 6 4 5 6 4 7 5 5
21224 116/300 Write block 1153 len 16384 data: 5 5 5 5
21225 117/300 Write block 1327 len 28672 data: 5 4 4 5 6 5 5
21226 118/300 Read block 1298 len 12288
21227 119/300 Read block 1552 len 16384
21228 120/300 Write block 3472 len 32768 data: 4 5 5 5 4 4 3 4
21229 121/300 Read block 1043 len 4096
21230 122/300 Write block 1045 len 40960 data: 5 7 6 7 7 7 5 4 8 8
21231 123/300 Read block 1173 len 20480
21232 124/300 Write block 3930 len 36864 data: 4 4 3 4 4 4 3 3 5
21233 125/300 Flush
21234 126/300 Write block 3315 len 40960 data: 5 5 3 4 4 4 6 5 5 4
21235 127/300 Write block 1904 len 20480 data: 4 5 6 6 7
21236 128/300 Read block 3532 len 28672
21237 129/300 Flush
21238 130/300 Read block 3819 len 8192
21239 131/300 Write block 841 len 4096 data: 4
21240 132/300 Write block 158 len 28672 data: 6 6 5 5 6 7 7
21241 133/300 Read block 565 len 36864
21242 134/300 Write block 3723 len 4096 data: 5
21243 135/300 Read block 2119 len 32768
21244 136/300 Write block 1790 len 40960 data: 2 4 3 4 4 4 3 2 2 5
21245 137/300 Read block 2791 len 8192
21246 138/300 Write block 4761 len 40960 data: 4 5 4 5 5 6 4 4 5 5
21247 139/300 Write block 3274 len 12288 data: 3 3 2
21248 140/300 Write block 4797 len 12288 data: 3 3 3
21249 141/300 Read block 3593 len 8192
21250 142/300 Read block 823 len 16384
21251 143/300 Write block 1936 len 32768 data: 5 4 3 3 2 2 2 3
21252 144/300 Read block 4003 len 36864
21253 145/300 Flush
21254 146/300 Write block 593 len 32768 data: 7 7 7 6 5 4 3 2
21255 147/300 Write block 1614 len 4096 data: 5
21256 148/300 Read block 195 len 40960
21257 149/300 Read block 4908 len 24576
21258 150/300 Read block 4632 len 28672
21259 151/300 Read block 4021 len 4096
21260 152/300 Read block 669 len 20480
21261 153/300 Flush
21262 154/300 Flush
21263 155/300 Write block 911 len 32768 data: 8 6 5 3 4 4 4 6
21264 156/300 Write block 1251 len 32768 data: 5 5 5 5 5 5 5 3
21265 157/300 Read block 1411 len 12288
21266 158/300 Write block 4091 len 24576 data: 3 3 3 3 3 3
21267 159/300 Read block 3299 len 32768
21268 160/300 Read block 4444 len 8192
21269 161/300 Read block 2371 len 28672
21270 162/300 Write block 4568 len 40960 data: 3 4 3 3 3 3 3 3 3 3
21271 163/300 Flush
21272 164/300 Read block 3620 len 20480
21273 165/300 Flush
21274 166/300 Write block 1075 len 40960 data: 5 6 6 5 5 4 4 4 4 4
21275 167/300 Read block 2177 len 24576
21276 168/300 Read block 418 len 12288
21277 169/300 Read block 4964 len 12288
21278 170/300 Flush
21279 171/300 Flush
21280 172/300 Write block 1209 len 28672 data: 2 3 3 4 4 3 4
21281 173/300 Read block 4839 len 36864
21282 174/300 Read block 1529 len 28672
21283 175/300 Read block 1968 len 32768
21284 176/300 Write block 3775 len 28672 data: 7 8 6 5 5 5 5
21285 177/300 Read block 4843 len 20480
21286 178/300 Read block 3230 len 36864
21287 179/300 Write block 2172 len 16384 data: 4 4 3 3
21288 180/300 Read block 3787 len 12288
21289 181/300 Read block 433 len 24576
21290 182/300 Write block 1511 len 28672 data: 4 4 5 4 4 4 4
21291 183/300 Write block 737 len 32768 data: 3 5 5 9 9 9 9 9
21292 184/300 Read block 3817 len 12288
21293 185/300 Write block 1569 len 28672 data: 4 4 3 3 2 2 2
21294 186/300 Read block 4954 len 20480
21295 187/300 Write block 3092 len 8192 data: 4 3
21296 188/300 Write block 1957 len 24576 data: 6 6 6 3 3 5
21297 189/300 Flush
21298 190/300 Read block 4432 len 12288
21299 191/300 Read block 764 len 4096
21300 192/300 Read block 1041 len 40960
21301 193/300 Read block 3039 len 28672
21302 194/300 Write block 4488 len 12288 data: 3 3 4
21303 195/300 Read block 2462 len 36864
21304 196/300 Flush
21305 197/300 Read block 4825 len 12288
21306 198/300 Read block 4284 len 12288
21307 199/300 Read block 2772 len 28672
21308 200/300 Read block 2173 len 16384
21309 201/300 Read block 559 len 4096
21310 202/300 Write block 1862 len 20480 data: 4 4 4 4 6
21311 203/300 Flush
21312 204/300 Read block 452 len 24576
21313 205/300 Write block 3510 len 8192 data: 3 3
21314 206/300 Write block 3502 len 16384 data: 5 5 6 6
21315 207/300 Read block 2032 len 20480
21316 208/300 Write block 963 len 12288 data: 4 5 3
21317 209/300 Flush
21318 210/300 Write block 2655 len 4096 data: 2
21319 211/300 Read block 3847 len 4096
21320 212/300 Read block 2397 len 12288
21321 213/300 Flush
21322 214/300 Read block 1662 len 4096
21323 215/300 Read block 2519 len 32768
21324 216/300 Write block 757 len 32768 data: 4 4 4 4 4 4 3 4
21325 217/300 Write block 2937 len 36864 data: 4 4 5 4 3 4 2 2 3
21326 218/300 Write block 2867 len 40960 data: 5 5 5 6 6 5 3 3 4 4
21327 219/300 Write block 340 len 36864 data: 3 5 5 5 4 3 5 5 4
21328 220/300 Read block 2648 len 20480
21329 221/300 Read block 1053 len 8192
21330 222/300 Write block 2597 len 8192 data: 6 6
21331 223/300 Write block 2498 len 40960 data: 3 4 3 3 3 3 3 3 3 3
21332 224/300 Write block 3110 len 20480 data: 4 5 5 6 7
21333 225/300 Read block 2845 len 40960
21334 226/300 Flush
21335 227/300 Write block 443 len 32768 data: 3 3 3 6 7 6 7 8
21336 228/300 Write block 527 len 24576 data: 4 3 3 4 2 2
21337 229/300 Write block 2046 len 16384 data: 4 3 3 3
21338 230/300 Read block 1616 len 8192
21339 231/300 Write block 2241 len 36864 data: 5 5 5 4 3 3 3 4 5
21340 232/300 Write block 3828 len 8192 data: 6 5
21341 233/300 Write block 3784 len 4096 data: 7
21342 234/300 Read block 3248 len 20480
21343 235/300 Read block 2092 len 8192
21344 236/300 Write block 1852 len 8192 data: 2 2
21345 237/300 Read block 3314 len 24576
21346 238/300 Write block 4319 len 40960 data: 4 3 4 4 4 4 5 6 5 5
21347 239/300 Flush
21348 240/300 Read block 3206 len 24576
21349 241/300 Write block 1469 len 20480 data: 2 3 6 7 7
21350 242/300 Write block 1077 len 16384 data: 7 6 6 5
21351 243/300 Flush
21352 244/300 Read block 953 len 24576
21353 245/300 Flush
21354 246/300 Read block 359 len 20480
21355 247/300 Read block 1014 len 16384
21356 248/300 Write block 4903 len 16384 data: 8 8 6 6
21357 249/300 Read block 3717 len 24576
21358 250/300 Write block 4069 len 12288 data: 2 2 2
21359 251/300 Write block 2046 len 4096 data: 5
21360 252/300 Read block 3885 len 4096
21361 253/300 Write block 2622 len 32768 data: 4 4 4 4 5 5 4 3
21362 254/300 Read block 65 len 4096
21363 255/300 Write block 2082 len 24576 data: 3 3 3 3 4 5
21364 256/300 Write block 1098 len 24576 data: 2 2 3 3 3 3
21365 257/300 Write block 1935 len 12288 data: 5 6 5
21366 258/300 Write block 3931 len 24576 data: 5 4 5 5 5 4
21367 259/300 Read block 3641 len 32768
21368 260/300 Write block 3002 len 32768 data: 4 4 5 4 4 4 4 4
21369 261/300 Read block 1862 len 12288
21370 262/300 Read block 4423 len 28672
21371 263/300 Read block 902 len 20480
21372 264/300 Read block 1788 len 36864
21373 265/300 Read block 3746 len 4096
21374 266/300 Flush
21375 267/300 Write block 582 len 36864 data: 3 3 2 2 2 3 5 3 4
21376 268/300 Read block 1549 len 16384
21377 269/300 Write block 4359 len 4096 data: 6
21378 270/300 Read block 2323 len 8192
21379 271/300 Read block 1720 len 40960
21380 272/300 Write block 4727 len 8192 data: 3 4
21381 273/300 Read block 3254 len 28672
21382 274/300 Flush
21383 275/300 Read block 2710 len 4096
21384 276/300 Write block 2912 len 36864 data: 3 3 4 2 2 2 3 3 3
21385 277/300 Write block 2969 len 12288 data: 6 5 5
21386 278/300 Write block 3241 len 32768 data: 3 3 2 3 3 3 4 6
21387 279/300 Read block 585 len 36864
21388 280/300 Write block 4454 len 40960 data: 6 5 3 3 2 2 2 2 2 2
21389 281/300 Write block 4699 len 16384 data: 4 5 5 3
21390 282/300 Write block 1105 len 32768 data: 4 4 4 5 6 7 6 5
21391 283/300 Write block 3924 len 12288 data: 4 5 5
21392 284/300 Read block 1447 len 8192
21393 285/300 Read block 1546 len 8192
21394 286/300 Read block 3174 len 20480
21395 287/300 Read block 2810 len 8192
21396 288/300 Write block 1588 len 4096 data: 8
21397 289/300 Read block 537 len 28672
21398 290/300 Flush
21399 291/300 Flush
21400 292/300 Flush
21401 293/300 Read block 1356 len 8192
21402 294/300 Read block 1560 len 28672
21403 295/300 Write block 1394 len 16384 data: 2 2 3 4
21404 296/300 Write block 609 len 24576 data: 5 5 4 4 5 5
21405 297/300 Write block 1766 len 4096 data: 2
21406 298/300 Write block 981 len 40960 data: 4 4 3 3 3 3 4 4 4 4
21407 299/300 Read block 3075 len 12288
21408 300/300 Read block 2181 len 16384
21409 301/300 Write block 2823 len 16384 data: 7 6 6 6
21410 Main task has work to do, go find it
21411 got dsc Start(2)
21412 start 2
21413 Mar 17 22:23:46.402 INFO request completed, latency_us: 244, response_code: 204, uri: /start/cid/2, method: POST, req_id: 09b912f7-e5ba-4ef9-bc0a-ea7c5fc098fa, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
21414 Replay: started 2, returned:Ok(())
21415 ----------------------------------------------------------------
21416 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21417 States: Active Active Offline
21418 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21419 5830 Acked Flush 0 Done Done Sent false
21420 5831 Acked Write 2 Done Done Sent false
21421 5832 Acked Read 2 Done Done Sent false
21422 5833 Acked Read 10 Done Done Sent false
21423 5834 Acked Read 7 Done Done Sent false
21424 5835 Acked Write 9 Done Done Sent false
21425 5836 Acked Write 9 Done Done Sent false
21426 5837 Acked Read 2 Done Done Sent false
21427 5838 Acked Read 5 Done Done Sent false
21428 5839 Acked Write 9 Done Done Sent false
21429 5840 Acked Read 1 Done Done Sent false
21430 5841 Acked Write 10 Done Done Sent false
21431 5842 Acked Read 5 Done Done Sent false
21432 5843 Acked Read 6 Done Done Sent false
21433 5844 Acked Write 8 Done Done Sent false
21434 5845 Acked Read 3 Done Done Sent false
21435 5846 Acked Read 8 Done Done Sent false
21436 5847 Acked Write 8 Done Done Sent false
21437 5848 Acked Write 3 Done Done Sent false
21438 5849 Acked Flush 0 Done Done Sent false
21439 5850 Acked Read 6 Done Done Sent false
21440 5851 Acked Flush 0 Done Done Sent false
21441 5852 Acked Flush 0 Done Done Sent false
21442 5853 Acked Read 4 Done Done Sent false
21443 5854 Acked Read 9 Done Done Sent false
21444 5855 Acked Write 5 Done Done Sent false
21445 5856 Acked Read 7 Done Done Sent false
21446 5857 Acked Read 6 Done Done Sent false
21447 5858 Acked Write 4 Done Done Sent false
21448 5859 Acked Write 9 Done Done Sent false
21449 5860 Acked Write 2 Done Done Sent false
21450 5861 Acked Read 10 Done Done Sent false
21451 5862 Acked Write 7 Done Done Sent false
21452 5863 Acked Read 1 Done Done Sent false
21453 5864 Acked Write 4 Done Done Sent false
21454 5865 Acked Read 2 Done Done Sent false
21455 5866 Acked Write 10 Done Done Sent false
21456 5867 Acked Flush 0 Done Done Sent false
21457 5868 Acked Read 1 Done Done Sent false
21458 5869 Acked Read 1 Done Done Sent false
21459 5870 Acked Read 2 Done Done Sent false
21460 5871 Acked Write 8 Done Done Sent false
21461 5872 Acked Write 10 Done Done Sent false
21462 5873 Acked Flush 0 Done Done Sent false
21463 5874 Acked Write 10 Done Done Sent false
21464 5875 Acked Flush 0 Done Done Sent false
21465 5876 Acked Read 6 Done Done Sent false
21466 5877 Acked Write 2 Done Done Sent false
21467 5878 Acked Read 1 Done Done Sent false
21468 5879 Acked Flush 0 Done Done Sent false
21469 5880 Acked Read 2 Done Done Sent false
21470 5881 Acked Read 2 Done Done Sent false
21471 5882 Acked Read 5 Done Done Sent false
21472 5883 Acked Read 10 Done Done Sent false
21473 5884 Acked Read 2 Done Done Sent false
21474 5885 Acked Flush 0 Done Done Sent false
21475 5886 Acked Write 6 Done Done Sent false
21476 5887 Acked Read 8 Done Done Sent false
21477 5888 Acked Read 8 Done Done Sent false
21478 5889 Acked Write 10 Done Done Sent false
21479 5890 Acked Read 2 Done Done Sent false
21480 5891 Acked Flush 0 Done Done Sent false
21481 5892 Acked Read 5 Done Done Sent false
21482 5893 Acked Write 1 Done Done Sent false
21483 5894 Acked Write 2 Done Done Sent false
21484 5895 Acked Write 4 Done Done Sent false
21485 5896 Acked Read 7 Done Done Sent false
21486 5897 Acked Flush 0 Done Done Sent false
21487 5898 Acked Read 3 Done Done Sent false
21488 5899 Acked Write 10 Done Done Sent false
21489 5900 Acked Read 10 Done Done Sent false
21490 5901 Acked Write 8 Done Done Sent false
21491 5902 Acked Write 8 Done Done Sent false
21492 5903 Acked Read 5 Done Done Sent false
21493 5904 Acked Write 6 Done Done Sent false
21494 5905 Acked Flush 0 Done Done Sent false
21495 5906 Acked Read 2 Done Done Sent false
21496 5907 Acked Flush 0 Done Done Sent false
21497 5908 Acked Flush 0 Done Done Sent false
21498 5909 Acked Write 2 Done Done Sent false
21499 5910 Acked Read 1 Done Done Sent false
21500 5911 Acked Read 5 Done Done Sent false
21501 5912 Acked Read 9 Done Done Sent false
21502 5913 Acked Read 2 Done Done Sent false
21503 5914 Acked Write 1 Done Done Sent false
21504 5915 Acked Read 1 Done Done Sent false
21505 5916 Acked Flush 0 Done Done Sent false
21506 5917 Acked Flush 0 Done Done Sent false
21507 5918 Acked Write 1 Done Done Sent false
21508 5919 Acked Flush 0 Done Done Sent false
21509 5920 Acked Read 10 Done Done Sent false
21510 5921 Acked Write 2 Done Done Sent false
21511 5922 Acked Read 5 Done Done Sent false
21512 5923 Acked Write 2 Done Done Sent false
21513 5924 Acked Write 7 Done Done Sent false
21514 5925 Acked Write 6 Done Done Sent false
21515 5926 Acked Flush 0 Done Done Sent false
21516 5927 Acked Flush 0 Done Done Sent false
21517 5928 Acked Read 6 Done Done Sent false
21518 5929 Acked Read 3 Done Done Sent false
21519 5930 Acked Read 9 Done Done Sent false
21520 5931 Acked Read 6 Done Done Sent false
21521 5932 Acked Read 1 Done Done Sent false
21522 5933 Acked Read 1 Done Done Sent false
21523 5934 Acked Flush 0 Done Done Sent false
21524 5935 Acked Write 3 Done Done Sent false
21525 5936 Acked Write 1 Done Done Sent false
21526 5937 Acked Write 8 Done Done Sent false
21527 5938 Acked Write 4 Done Done Sent false
21528 5939 Acked Write 6 Done Done Sent false
21529 5940 Acked Write 10 Done Done Sent false
21530 5941 Acked Write 6 Done Done Sent false
21531 5942 Acked Read 9 Done Done Sent false
21532 5943 Acked Write 4 Done Done Sent false
21533 5944 Acked Write 10 Done Done Sent false
21534 5945 Acked Write 4 Done Done Sent false
21535 5946 Acked Write 7 Done Done Sent false
21536 5947 Acked Read 3 Done Done Sent false
21537 5948 Acked Read 4 Done Done Sent false
21538 5949 Acked Write 8 Done Done Sent false
21539 5950 Acked Read 1 Done Done Sent false
21540 5951 Acked Write 10 Done Done Sent false
21541 5952 Acked Read 5 Done Done Sent false
21542 5953 Acked Write 9 Done Done Sent false
21543 5954 Acked Flush 0 Done Done Sent false
21544 5955 Acked Write 10 Done Done Sent false
21545 5956 Acked Write 5 Done Done Sent false
21546 5957 Acked Read 7 Done Done Sent false
21547 5958 Acked Flush 0 Done Done Sent false
21548 5959 Acked Read 2 Done Done Sent false
21549 5960 Acked Write 1 Done Done Sent false
21550 5961 Acked Write 7 Done Done Sent false
21551 5962 Acked Read 9 Done Done Sent false
21552 5963 Acked Write 1 Done Done Sent false
21553 5964 Acked Read 8 Done Done Sent false
21554 5965 Acked Write 10 Done Done Sent false
21555 5966 Acked Read 2 Done Done Sent false
21556 5967 Acked Write 10 Done Done Sent false
21557 5968 Acked Write 3 Done Done Sent false
21558 5969 Acked Write 3 Done Done Sent false
21559 5970 Acked Read 2 Done Done Sent false
21560 5971 Acked Read 4 Done Done Sent false
21561 5972 Acked Write 8 Done Done Sent false
21562 5973 Acked Read 9 Done Done Sent false
21563 5974 Acked Flush 0 Done Done Sent false
21564 5975 Acked Write 8 Done Done Sent false
21565 5976 Acked Write 1 Done Done Sent false
21566 5977 Acked Read 10 Done Done Sent false
21567 5978 Acked Read 6 Done Done Sent false
21568 5979 Acked Read 7 Done Done Sent false
21569 5980 Acked Read 1 Done Done Sent false
21570 5981 Acked Read 5 Done Done Sent false
21571 5982 Acked Flush 0 Done Done Sent false
21572 5983 Acked Flush 0 Done Done Sent false
21573 5984 Acked Write 8 Done Done Sent false
21574 5985 Acked Write 8 Done Done Sent false
21575 5986 Acked Read 3 Done Done Sent false
21576 5987 Acked Write 6 Done Done Sent false
21577 5988 Acked Read 8 Done Done Sent false
21578 5989 Acked Read 2 Done Done Sent false
21579 5990 Acked Read 7 Done Done Sent false
21580 5991 Acked Write 10 Done Done Sent false
21581 5992 Acked Flush 0 Done Done Sent false
21582 5993 Acked Read 5 Done Done Sent false
21583 5994 Acked Flush 0 Done Done Sent false
21584 5995 Acked Write 10 Done Done Sent false
21585 5996 Acked Read 6 Done Done Sent false
21586 5997 Acked Read 3 Done Done Sent false
21587 5998 Acked Read 3 Done Done Sent false
21588 5999 Acked Flush 0 Done Done Sent false
21589 6000 Acked Flush 0 Done Done Sent false
21590 6001 Acked Write 7 Done Done Sent false
21591 6002 Acked Read 9 Done Done Sent false
21592 6003 Acked Read 7 Done Done Sent false
21593 6004 Acked Read 8 Done Done Sent false
21594 6005 Acked Write 7 Done Done Sent false
21595 6006 Acked Read 5 Done Done Sent false
21596 6007 Acked Read 9 Done Done Sent false
21597 6008 Acked Write 4 Done Done Sent false
21598 6009 Acked Read 3 Done Done Sent false
21599 6010 Acked Read 6 Done Done Sent false
21600 6011 Acked Write 7 Done Done Sent false
21601 6012 Acked Write 8 Done Done Sent false
21602 6013 Acked Read 3 Done Done Sent false
21603 6014 Acked Write 7 Done Done Sent false
21604 6015 Acked Read 5 Done Done Sent false
21605 6016 Acked Write 2 Done Done Sent false
21606 6017 Acked Write 6 Done Done Sent false
21607 6018 Acked Flush 0 Done Done Sent false
21608 6019 Acked Read 3 Done Done Sent false
21609 6020 Acked Read 1 Done Done Sent false
21610 6021 Acked Read 10 Done Done Sent false
21611 6022 Acked Read 7 Done Done Sent false
21612 6023 Acked Write 3 Done Done Sent false
21613 6024 Acked Read 9 Done Done Sent false
21614 6025 Acked Flush 0 Done Done Sent false
21615 6026 Acked Read 3 Done Done Sent false
21616 6027 Acked Read 3 Done Done Sent false
21617 6028 Acked Read 7 Done Done Sent false
21618 6029 Acked Read 4 Done Done Sent false
21619 6030 Acked Read 1 Done Done Sent false
21620 6031 Acked Write 5 Done Done Sent false
21621 6032 Acked Flush 0 Done Done Sent false
21622 6033 Acked Read 6 Done Done Sent false
21623 6034 Acked Write 2 Done Done Sent false
21624 6035 Acked Write 4 Done Done Sent false
21625 6036 Acked Read 5 Done Done Sent false
21626 6037 Acked Write 3 Done Done Sent false
21627 6038 Acked Flush 0 Done Done Sent false
21628 6039 Acked Write 1 Done Done Sent false
21629 6040 Acked Read 1 Done Done Sent false
21630 6041 Acked Read 3 Done Done Sent false
21631 6042 Acked Flush 0 Done Done Sent false
21632 6043 Acked Read 1 Done Done Sent false
21633 6044 Acked Read 8 Done Done Sent false
21634 6045 Acked Write 8 Done Done Sent false
21635 6046 Acked Write 9 Done Done Sent false
21636 6047 Acked Write 10 Done Done Sent false
21637 6048 Acked Write 9 Done Done Sent false
21638 6049 Acked Read 5 Done Done Sent false
21639 6050 Acked Read 2 Done Done Sent false
21640 6051 Acked Write 2 Done Done Sent false
21641 6052 Acked Write 10 Done Done Sent false
21642 6053 Acked Write 5 Done Done Sent false
21643 6054 Acked Read 10 Done Done Sent false
21644 6055 Acked Flush 0 Done Done Sent false
21645 6056 Acked Write 8 Done Done Sent false
21646 6057 Acked Write 6 Done Done Sent false
21647 6058 Acked Write 4 Done Done Sent false
21648 6059 Acked Read 2 Done Done Sent false
21649 6060 Acked Write 9 Done Done Sent false
21650 6061 Acked Write 2 Done Done Sent false
21651 6062 Acked Write 1 Done Done Sent false
21652 6063 Acked Read 5 Done Done Sent false
21653 6064 Acked Read 2 Done Done Sent false
21654 6065 Acked Write 2 Done Done Sent false
21655 6066 Acked Read 6 Done Done Sent false
21656 6067 Acked Write 10 Done Done Sent false
21657 6068 Acked Flush 0 Done Done Sent false
21658 6069 Acked Read 6 Done Done Sent false
21659 6070 Acked Write 5 Done Done Sent false
21660 6071 Acked Write 4 Done Done Sent false
21661 6072 Acked Flush 0 Done Done Sent false
21662 6073 Acked Read 6 Done Done Sent false
21663 6074 Acked Flush 0 Done Done Sent false
21664 6075 Acked Read 5 Done Done Sent false
21665 6076 Acked Read 4 Done Done Sent false
21666 6077 Acked Write 4 Done Done Sent false
21667 6078 Acked Read 6 Done Done Sent false
21668 6079 Acked Write 3 Done Done Sent false
21669 6080 Acked Write 1 Done Done Sent false
21670 6081 Acked Read 1 Done Done Sent false
21671 6082 Acked Write 8 Done Done Sent false
21672 6083 Acked Read 1 Done Done Sent false
21673 6084 Acked Write 6 Done Done Sent false
21674 6085 Acked Write 6 Done Done Sent false
21675 6086 Acked Write 3 Done Done Sent false
21676 6087 Acked Write 6 Done Done Sent false
21677 6088 Acked Read 8 Done Done Sent false
21678 6089 Acked Write 8 Done Done Sent false
21679 6090 Acked Read 3 Done Done Sent false
21680 6091 Acked Read 7 Done Done Sent false
21681 6092 Acked Read 5 Done Done Sent false
21682 6093 Acked Read 9 Done Done Sent false
21683 6094 Acked Read 1 Done Done Sent false
21684 6095 Acked Flush 0 Done Done Sent false
21685 6096 Acked Write 9 Done Done Sent false
21686 6097 Acked Read 4 Done Done Sent false
21687 6098 Acked Write 1 Done Done Sent false
21688 6099 Acked Read 2 Done Done Sent false
21689 6100 Acked Read 10 Done Done Sent false
21690 6101 Acked Write 2 Done Done Sent false
21691 6102 Acked Read 7 Done Done Sent false
21692 6103 Acked Flush 0 Done Done Sent false
21693 6104 Acked Read 1 Done Done Sent false
21694 6105 Acked Write 9 Done Done Sent false
21695 6106 Acked Write 3 Done Done Sent false
21696 6107 Acked Write 8 Done Done Sent false
21697 6108 Acked Read 9 Done Done Sent false
21698 6109 Acked Write 10 Done Done Sent false
21699 6110 Acked Write 4 Done Done Sent false
21700 6111 Acked Write 8 Done Done Sent false
21701 6112 Acked Write 3 Done Done Sent false
21702 6113 Acked Read 2 Done Done Sent false
21703 6114 Acked Read 2 Done Done Sent false
21704 6115 Acked Read 5 Done Done Sent false
21705 6116 Acked Read 2 Done Done Sent false
21706 6117 Acked Write 1 Done Done Sent false
21707 6118 Acked Read 7 Done Done Sent false
21708 6119 Acked Flush 0 Done Done Sent false
21709 6120 Acked Flush 0 Done Done Sent false
21710 6121 Acked Flush 0 Done Done Sent false
21711 6122 Acked Read 2 Done Done Sent false
21712 6123 Acked Read 7 Done Done Sent false
21713 6124 Acked Write 4 Done Done Sent false
21714 6125 Acked Write 6 Done Done Sent false
21715 6126 Acked Write 1 Done Done Sent false
21716 6127 Acked Write 10 Done Done Sent false
21717 6128 Acked Read 3 Done Done Sent false
21718 6129 Acked Read 4 Done Done Sent false
21719 6130 Acked Write 4 Done Done Sent false
21720 STATES DS:0 DS:1 DS:2 TOTAL
21721 Sent 0 0 301 301
21722 Done 301 301 0 602
21723 Skipped 0 0 0 0
21724 Error 0 0 0 0
21725 Last Flush: Some(JobId(6121)) Some(JobId(6121)) Some(JobId(5829))
21726 Downstairs last five completed: 5829 5828 5827 5826 5825
21727 Upstairs last five completed: 6130 6129 6128 6127 6126
21728 CLIENT: Up:0 ds:301 act:2
217292025-03-17T22:23:47.411ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
217302025-03-17T22:23:48.036ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
21731 [8830] got start action
21732 [8830] I am going to restart
21733 Starting downstairs at port 8830
21734 Make output file at "/tmp/dsc/downstairs-8830.txt"
21735 [8830][2] reports Starting
21736 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1228)
21737 [8830][2] reports Running
217382025-03-17T22:23:49.039ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
217392025-03-17T22:23:50.040ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
21740 ----------------------------------------------------------------
21741 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21742 States: Active Active Offline
21743 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21744 5830 Acked Flush 0 Done Done Sent false
21745 5831 Acked Write 2 Done Done Sent false
21746 5832 Acked Read 2 Done Done Sent false
21747 5833 Acked Read 10 Done Done Sent false
21748 5834 Acked Read 7 Done Done Sent false
21749 5835 Acked Write 9 Done Done Sent false
21750 5836 Acked Write 9 Done Done Sent false
21751 5837 Acked Read 2 Done Done Sent false
21752 5838 Acked Read 5 Done Done Sent false
21753 5839 Acked Write 9 Done Done Sent false
21754 5840 Acked Read 1 Done Done Sent false
21755 5841 Acked Write 10 Done Done Sent false
21756 5842 Acked Read 5 Done Done Sent false
21757 5843 Acked Read 6 Done Done Sent false
21758 5844 Acked Write 8 Done Done Sent false
21759 5845 Acked Read 3 Done Done Sent false
21760 5846 Acked Read 8 Done Done Sent false
21761 5847 Acked Write 8 Done Done Sent false
21762 5848 Acked Write 3 Done Done Sent false
21763 5849 Acked Flush 0 Done Done Sent false
21764 5850 Acked Read 6 Done Done Sent false
21765 5851 Acked Flush 0 Done Done Sent false
21766 5852 Acked Flush 0 Done Done Sent false
21767 5853 Acked Read 4 Done Done Sent false
21768 5854 Acked Read 9 Done Done Sent false
21769 5855 Acked Write 5 Done Done Sent false
21770 5856 Acked Read 7 Done Done Sent false
21771 5857 Acked Read 6 Done Done Sent false
21772 5858 Acked Write 4 Done Done Sent false
21773 5859 Acked Write 9 Done Done Sent false
21774 5860 Acked Write 2 Done Done Sent false
21775 5861 Acked Read 10 Done Done Sent false
21776 5862 Acked Write 7 Done Done Sent false
21777 5863 Acked Read 1 Done Done Sent false
21778 5864 Acked Write 4 Done Done Sent false
21779 5865 Acked Read 2 Done Done Sent false
21780 5866 Acked Write 10 Done Done Sent false
21781 5867 Acked Flush 0 Done Done Sent false
21782 5868 Acked Read 1 Done Done Sent false
21783 5869 Acked Read 1 Done Done Sent false
21784 5870 Acked Read 2 Done Done Sent false
21785 5871 Acked Write 8 Done Done Sent false
21786 5872 Acked Write 10 Done Done Sent false
21787 5873 Acked Flush 0 Done Done Sent false
21788 5874 Acked Write 10 Done Done Sent false
21789 5875 Acked Flush 0 Done Done Sent false
21790 5876 Acked Read 6 Done Done Sent false
21791 5877 Acked Write 2 Done Done Sent false
21792 5878 Acked Read 1 Done Done Sent false
21793 5879 Acked Flush 0 Done Done Sent false
21794 5880 Acked Read 2 Done Done Sent false
21795 5881 Acked Read 2 Done Done Sent false
21796 5882 Acked Read 5 Done Done Sent false
21797 5883 Acked Read 10 Done Done Sent false
21798 5884 Acked Read 2 Done Done Sent false
21799 5885 Acked Flush 0 Done Done Sent false
21800 5886 Acked Write 6 Done Done Sent false
21801 5887 Acked Read 8 Done Done Sent false
21802 5888 Acked Read 8 Done Done Sent false
21803 5889 Acked Write 10 Done Done Sent false
21804 5890 Acked Read 2 Done Done Sent false
21805 5891 Acked Flush 0 Done Done Sent false
21806 5892 Acked Read 5 Done Done Sent false
21807 5893 Acked Write 1 Done Done Sent false
21808 5894 Acked Write 2 Done Done Sent false
21809 5895 Acked Write 4 Done Done Sent false
21810 5896 Acked Read 7 Done Done Sent false
21811 5897 Acked Flush 0 Done Done Sent false
21812 5898 Acked Read 3 Done Done Sent false
21813 5899 Acked Write 10 Done Done Sent false
21814 5900 Acked Read 10 Done Done Sent false
21815 5901 Acked Write 8 Done Done Sent false
21816 5902 Acked Write 8 Done Done Sent false
21817 5903 Acked Read 5 Done Done Sent false
21818 5904 Acked Write 6 Done Done Sent false
21819 5905 Acked Flush 0 Done Done Sent false
21820 5906 Acked Read 2 Done Done Sent false
21821 5907 Acked Flush 0 Done Done Sent false
21822 5908 Acked Flush 0 Done Done Sent false
21823 5909 Acked Write 2 Done Done Sent false
21824 5910 Acked Read 1 Done Done Sent false
21825 5911 Acked Read 5 Done Done Sent false
21826 5912 Acked Read 9 Done Done Sent false
21827 5913 Acked Read 2 Done Done Sent false
21828 5914 Acked Write 1 Done Done Sent false
21829 5915 Acked Read 1 Done Done Sent false
21830 5916 Acked Flush 0 Done Done Sent false
21831 5917 Acked Flush 0 Done Done Sent false
21832 5918 Acked Write 1 Done Done Sent false
21833 5919 Acked Flush 0 Done Done Sent false
21834 5920 Acked Read 10 Done Done Sent false
21835 5921 Acked Write 2 Done Done Sent false
21836 5922 Acked Read 5 Done Done Sent false
21837 5923 Acked Write 2 Done Done Sent false
21838 5924 Acked Write 7 Done Done Sent false
21839 5925 Acked Write 6 Done Done Sent false
21840 5926 Acked Flush 0 Done Done Sent false
21841 5927 Acked Flush 0 Done Done Sent false
21842 5928 Acked Read 6 Done Done Sent false
21843 5929 Acked Read 3 Done Done Sent false
21844 5930 Acked Read 9 Done Done Sent false
21845 5931 Acked Read 6 Done Done Sent false
21846 5932 Acked Read 1 Done Done Sent false
21847 5933 Acked Read 1 Done Done Sent false
21848 5934 Acked Flush 0 Done Done Sent false
21849 5935 Acked Write 3 Done Done Sent false
21850 5936 Acked Write 1 Done Done Sent false
21851 5937 Acked Write 8 Done Done Sent false
21852 5938 Acked Write 4 Done Done Sent false
21853 5939 Acked Write 6 Done Done Sent false
21854 5940 Acked Write 10 Done Done Sent false
21855 5941 Acked Write 6 Done Done Sent false
21856 5942 Acked Read 9 Done Done Sent false
21857 5943 Acked Write 4 Done Done Sent false
21858 5944 Acked Write 10 Done Done Sent false
21859 5945 Acked Write 4 Done Done Sent false
21860 5946 Acked Write 7 Done Done Sent false
21861 5947 Acked Read 3 Done Done Sent false
21862 5948 Acked Read 4 Done Done Sent false
21863 5949 Acked Write 8 Done Done Sent false
21864 5950 Acked Read 1 Done Done Sent false
21865 5951 Acked Write 10 Done Done Sent false
21866 5952 Acked Read 5 Done Done Sent false
21867 5953 Acked Write 9 Done Done Sent false
21868 5954 Acked Flush 0 Done Done Sent false
21869 5955 Acked Write 10 Done Done Sent false
21870 5956 Acked Write 5 Done Done Sent false
21871 5957 Acked Read 7 Done Done Sent false
21872 5958 Acked Flush 0 Done Done Sent false
21873 5959 Acked Read 2 Done Done Sent false
21874 5960 Acked Write 1 Done Done Sent false
21875 5961 Acked Write 7 Done Done Sent false
21876 5962 Acked Read 9 Done Done Sent false
21877 5963 Acked Write 1 Done Done Sent false
21878 5964 Acked Read 8 Done Done Sent false
21879 5965 Acked Write 10 Done Done Sent false
21880 5966 Acked Read 2 Done Done Sent false
21881 5967 Acked Write 10 Done Done Sent false
21882 5968 Acked Write 3 Done Done Sent false
21883 5969 Acked Write 3 Done Done Sent false
21884 5970 Acked Read 2 Done Done Sent false
21885 5971 Acked Read 4 Done Done Sent false
21886 5972 Acked Write 8 Done Done Sent false
21887 5973 Acked Read 9 Done Done Sent false
21888 5974 Acked Flush 0 Done Done Sent false
21889 5975 Acked Write 8 Done Done Sent false
21890 5976 Acked Write 1 Done Done Sent false
21891 5977 Acked Read 10 Done Done Sent false
21892 5978 Acked Read 6 Done Done Sent false
21893 5979 Acked Read 7 Done Done Sent false
21894 5980 Acked Read 1 Done Done Sent false
21895 5981 Acked Read 5 Done Done Sent false
21896 5982 Acked Flush 0 Done Done Sent false
21897 5983 Acked Flush 0 Done Done Sent false
21898 5984 Acked Write 8 Done Done Sent false
21899 5985 Acked Write 8 Done Done Sent false
21900 5986 Acked Read 3 Done Done Sent false
21901 5987 Acked Write 6 Done Done Sent false
21902 5988 Acked Read 8 Done Done Sent false
21903 5989 Acked Read 2 Done Done Sent false
21904 5990 Acked Read 7 Done Done Sent false
21905 5991 Acked Write 10 Done Done Sent false
21906 5992 Acked Flush 0 Done Done Sent false
21907 5993 Acked Read 5 Done Done Sent false
21908 5994 Acked Flush 0 Done Done Sent false
21909 5995 Acked Write 10 Done Done Sent false
21910 5996 Acked Read 6 Done Done Sent false
21911 5997 Acked Read 3 Done Done Sent false
21912 5998 Acked Read 3 Done Done Sent false
21913 5999 Acked Flush 0 Done Done Sent false
21914 6000 Acked Flush 0 Done Done Sent false
21915 6001 Acked Write 7 Done Done Sent false
21916 6002 Acked Read 9 Done Done Sent false
21917 6003 Acked Read 7 Done Done Sent false
21918 6004 Acked Read 8 Done Done Sent false
21919 6005 Acked Write 7 Done Done Sent false
21920 6006 Acked Read 5 Done Done Sent false
21921 6007 Acked Read 9 Done Done Sent false
21922 6008 Acked Write 4 Done Done Sent false
21923 6009 Acked Read 3 Done Done Sent false
21924 6010 Acked Read 6 Done Done Sent false
21925 6011 Acked Write 7 Done Done Sent false
21926 6012 Acked Write 8 Done Done Sent false
21927 6013 Acked Read 3 Done Done Sent false
21928 6014 Acked Write 7 Done Done Sent false
21929 6015 Acked Read 5 Done Done Sent false
21930 6016 Acked Write 2 Done Done Sent false
21931 6017 Acked Write 6 Done Done Sent false
21932 6018 Acked Flush 0 Done Done Sent false
21933 6019 Acked Read 3 Done Done Sent false
21934 6020 Acked Read 1 Done Done Sent false
21935 6021 Acked Read 10 Done Done Sent false
21936 6022 Acked Read 7 Done Done Sent false
21937 6023 Acked Write 3 Done Done Sent false
21938 6024 Acked Read 9 Done Done Sent false
21939 6025 Acked Flush 0 Done Done Sent false
21940 6026 Acked Read 3 Done Done Sent false
21941 6027 Acked Read 3 Done Done Sent false
21942 6028 Acked Read 7 Done Done Sent false
21943 6029 Acked Read 4 Done Done Sent false
21944 6030 Acked Read 1 Done Done Sent false
21945 6031 Acked Write 5 Done Done Sent false
21946 6032 Acked Flush 0 Done Done Sent false
21947 6033 Acked Read 6 Done Done Sent false
21948 6034 Acked Write 2 Done Done Sent false
21949 6035 Acked Write 4 Done Done Sent false
21950 6036 Acked Read 5 Done Done Sent false
21951 6037 Acked Write 3 Done Done Sent false
21952 6038 Acked Flush 0 Done Done Sent false
21953 6039 Acked Write 1 Done Done Sent false
21954 6040 Acked Read 1 Done Done Sent false
21955 6041 Acked Read 3 Done Done Sent false
21956 6042 Acked Flush 0 Done Done Sent false
21957 6043 Acked Read 1 Done Done Sent false
21958 6044 Acked Read 8 Done Done Sent false
21959 6045 Acked Write 8 Done Done Sent false
21960 6046 Acked Write 9 Done Done Sent false
21961 6047 Acked Write 10 Done Done Sent false
21962 6048 Acked Write 9 Done Done Sent false
21963 6049 Acked Read 5 Done Done Sent false
21964 6050 Acked Read 2 Done Done Sent false
21965 6051 Acked Write 2 Done Done Sent false
21966 6052 Acked Write 10 Done Done Sent false
21967 6053 Acked Write 5 Done Done Sent false
21968 6054 Acked Read 10 Done Done Sent false
21969 6055 Acked Flush 0 Done Done Sent false
21970 6056 Acked Write 8 Done Done Sent false
21971 6057 Acked Write 6 Done Done Sent false
21972 6058 Acked Write 4 Done Done Sent false
21973 6059 Acked Read 2 Done Done Sent false
21974 6060 Acked Write 9 Done Done Sent false
21975 6061 Acked Write 2 Done Done Sent false
21976 6062 Acked Write 1 Done Done Sent false
21977 6063 Acked Read 5 Done Done Sent false
21978 6064 Acked Read 2 Done Done Sent false
21979 6065 Acked Write 2 Done Done Sent false
21980 6066 Acked Read 6 Done Done Sent false
21981 6067 Acked Write 10 Done Done Sent false
21982 6068 Acked Flush 0 Done Done Sent false
21983 6069 Acked Read 6 Done Done Sent false
21984 6070 Acked Write 5 Done Done Sent false
21985 6071 Acked Write 4 Done Done Sent false
21986 6072 Acked Flush 0 Done Done Sent false
21987 6073 Acked Read 6 Done Done Sent false
21988 6074 Acked Flush 0 Done Done Sent false
21989 6075 Acked Read 5 Done Done Sent false
21990 6076 Acked Read 4 Done Done Sent false
21991 6077 Acked Write 4 Done Done Sent false
21992 6078 Acked Read 6 Done Done Sent false
21993 6079 Acked Write 3 Done Done Sent false
21994 6080 Acked Write 1 Done Done Sent false
21995 6081 Acked Read 1 Done Done Sent false
21996 6082 Acked Write 8 Done Done Sent false
21997 6083 Acked Read 1 Done Done Sent false
21998 6084 Acked Write 6 Done Done Sent false
21999 6085 Acked Write 6 Done Done Sent false
22000 6086 Acked Write 3 Done Done Sent false
22001 6087 Acked Write 6 Done Done Sent false
22002 6088 Acked Read 8 Done Done Sent false
22003 6089 Acked Write 8 Done Done Sent false
22004 6090 Acked Read 3 Done Done Sent false
22005 6091 Acked Read 7 Done Done Sent false
22006 6092 Acked Read 5 Done Done Sent false
22007 6093 Acked Read 9 Done Done Sent false
22008 6094 Acked Read 1 Done Done Sent false
22009 6095 Acked Flush 0 Done Done Sent false
22010 6096 Acked Write 9 Done Done Sent false
22011 6097 Acked Read 4 Done Done Sent false
22012 6098 Acked Write 1 Done Done Sent false
22013 6099 Acked Read 2 Done Done Sent false
22014 6100 Acked Read 10 Done Done Sent false
22015 6101 Acked Write 2 Done Done Sent false
22016 6102 Acked Read 7 Done Done Sent false
22017 6103 Acked Flush 0 Done Done Sent false
22018 6104 Acked Read 1 Done Done Sent false
22019 6105 Acked Write 9 Done Done Sent false
22020 6106 Acked Write 3 Done Done Sent false
22021 6107 Acked Write 8 Done Done Sent false
22022 6108 Acked Read 9 Done Done Sent false
22023 6109 Acked Write 10 Done Done Sent false
22024 6110 Acked Write 4 Done Done Sent false
22025 6111 Acked Write 8 Done Done Sent false
22026 6112 Acked Write 3 Done Done Sent false
22027 6113 Acked Read 2 Done Done Sent false
22028 6114 Acked Read 2 Done Done Sent false
22029 6115 Acked Read 5 Done Done Sent false
22030 6116 Acked Read 2 Done Done Sent false
22031 6117 Acked Write 1 Done Done Sent false
22032 6118 Acked Read 7 Done Done Sent false
22033 6119 Acked Flush 0 Done Done Sent false
22034 6120 Acked Flush 0 Done Done Sent false
22035 6121 Acked Flush 0 Done Done Sent false
22036 6122 Acked Read 2 Done Done Sent false
22037 6123 Acked Read 7 Done Done Sent false
22038 6124 Acked Write 4 Done Done Sent false
22039 6125 Acked Write 6 Done Done Sent false
22040 6126 Acked Write 1 Done Done Sent false
22041 6127 Acked Write 10 Done Done Sent false
22042 6128 Acked Read 3 Done Done Sent false
22043 6129 Acked Read 4 Done Done Sent false
22044 6130 Acked Write 4 Done Done Sent false
22045 STATES DS:0 DS:1 DS:2 TOTAL
22046 Sent 0 0 301 301
22047 Done 301 301 0 602
22048 Skipped 0 0 0 0
22049 Error 0 0 0 0
22050 Last Flush: Some(JobId(6121)) Some(JobId(6121)) Some(JobId(5829))
22051 Downstairs last five completed: 5829 5828 5827 5826 5825
22052 Upstairs last five completed: 6130 6129 6128 6127 6126
22053 CLIENT: Up:0 ds:301 act:2
220542025-03-17T22:23:51.397ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
220552025-03-17T22:23:52.174ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
220562025-03-17T22:23:52.174ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:36811) = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
220572025-03-17T22:23:52.175ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
220582025-03-17T22:23:52.175ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
220592025-03-17T22:23:52.175ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
220602025-03-17T22:23:52.175ZINFOcrucible: send last flush ID to this DS: Some(JobId(5829)) = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
220612025-03-17T22:23:52.176ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
220622025-03-17T22:23:52.176ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(5829)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
220632025-03-17T22:23:52.176ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(5829)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
22064 ----------------------------------------------------------------
22065 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
22066 Downstairs last five completed: 6131 6130 6129 6128 6127
22067 Upstairs last five completed: 6131 6130 6129 6128 6127
22068 CLIENT: Up:0 ds:0 act:3
22069 Replay: All jobs finished
22070 Main task has work to do, go find it
22071 got dsc Stop(2)
22072 stop 2
22073 Mar 17 22:23:54.407 INFO request completed, latency_us: 348, response_code[8830] Got stop action so:false kr:false
22074 : 204, uri: /stop/cid/2, method: POST, req_id: fe769867-c511-4138-88c1-65fad131f67e, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
22075 Mar 17 22:23:54.408 INFO request completed, latency_us: 109, response_code: 200, uri: /state/cid/2, method: GET, req_id: f47795b9-243d-4819-89d8-dfad3dc67343, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
220762025-03-17T22:23:54.408ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
220772025-03-17T22:23:54.408ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
220782025-03-17T22:23:54.408ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
220792025-03-17T22:23:54.408ZINFOcrucible: client task is exiting = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
22080 [8830] Exited with: ExitStatus(unix_wait_status(9))
22081 [8830][2] reports Exit
22082 Mar 17 22:23:58.410 INFO request completed, latency_us: 189, response_code: 200, uri: /state/cid/2, method: GET, req_id: 0d92a0f8-6418-4cf4-ac77-ea751ddc0956, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
22083 001/300 Read block 4107 len 24576
22084 002/300 Read block 1116 len 28672
22085 003/300 Read block 2733 len 32768
22086 004/300 Write block 2352 len 32768 data: 2 2 2 3 3 3 4 5
22087 005/300 Write block 4362 len 32768 data: 7 6 6 5 5 5 4 4
22088 006/300 Read block 4259 len 32768
22089 007/300 Write block 756 len 40960 data: 5 5 5 5 5 5 5 4 5 4
22090 008/300 Write block 353 len 24576 data: 4 3 3 3 2 2
22091 009/300 Read block 4887 len 20480
22092 010/300 Write block 2978 len 16384 data: 3 5 4 4
22093 011/300 Read block 3598 len 36864
22094 012/300 Write block 3636 len 36864 data: 4 5 5 5 5 3 3 3 3
22095 013/300 Flush
22096 014/300 Write block 2383 len 16384 data: 7 7 6 6
22097 015/300 Read block 1005 len 24576
22098 016/300 Write block 126 len 12288 data: 4 4 5
22099 017/300 Read block 2949 len 24576
22100 018/300 Read block 4860 len 40960
22101 019/300 Read block 305 len 8192
22102 020/300 Flush
22103 021/300 Read block 1744 len 40960
22104 022/300 Read block 496 len 12288
22105 023/300 Read block 3899 len 12288
22106 024/300 Read block 58 len 28672
22107 025/300 Write block 3851 len 36864 data: 6 8 5 5 4 4 3 3 3
22108 026/300 Read block 817 len 36864
22109 027/300 Read block 1002 len 40960
22110 028/300 Write block 3756 len 16384 data: 6 6 6 6
22111 029/300 Flush
22112 030/300 Write block 2920 len 8192 data: 4 4
22113 031/300 Read block 4026 len 40960
22114 032/300 Write block 473 len 32768 data: 4 4 6 6 6 5 5 3
22115 033/300 Write block 275 len 28672 data: 5 5 4 4 4 4 5
22116 034/300 Read block 4493 len 12288
22117 035/300 Read block 1573 len 24576
22118 036/300 Flush
22119 037/300 Write block 1688 len 8192 data: 5 4
22120 038/300 Read block 1659 len 20480
22121 039/300 Read block 3889 len 36864
22122 040/300 Flush
22123 041/300 Write block 1934 len 12288 data: 6 6 7
22124 042/300 Read block 4752 len 36864
22125 043/300 Flush
22126 044/300 Flush
22127 045/300 Read block 1761 len 8192
22128 046/300 Write block 2019 len 8192 data: 6 4
22129 047/300 Read block 4039 len 16384
22130 048/300 Read block 570 len 24576
22131 049/300 Read block 1946 len 28672
22132 050/300 Write block 1268 len 24576 data: 3 4 3 4 5 4
22133 051/300 Write block 2531 len 4096 data: 5
22134 052/300 Read block 4839 len 16384
22135 053/300 Write block 935 len 12288 data: 4 3 2
22136 054/300 Write block 868 len 16384 data: 3 3 4 4
22137 055/300 Write block 2405 len 40960 data: 5 5 4 3 2 3 5 5 6 6
22138 056/300 Read block 398 len 4096
22139 057/300 Write block 3237 len 20480 data: 3 3 3 3 4
22140 058/300 Flush
22141 059/300 Read block 4050 len 28672
22142 060/300 Read block 1902 len 20480
22143 061/300 Write block 4539 len 28672 data: 6 6 3 3 4 5 4
22144 062/300 Write block 2210 len 16384 data: 3 4 4 2
22145 063/300 Read block 908 len 32768
22146 064/300 Read block 4134 len 20480
22147 065/300 Write block 1489 len 28672 data: 4 4 4 4 5 5 3
22148 066/300 Write block 421 len 36864 data: 8 7 7 6 5 3 2 2 2
22149 067/300 Read block 3460 len 32768
22150 068/300 Read block 3865 len 32768
22151 069/300 Write block 1759 len 4096 data: 5
22152 070/300 Read block 3816 len 4096
22153 071/300 Read block 3504 len 16384
22154 072/300 Write block 3730 len 8192 data: 6 6
22155 073/300 Write block 2610 len 8192 data: 6 5
22156 074/300 Write block 4819 len 12288 data: 5 6 6
22157 075/300 Read block 287 len 20480
22158 076/300 Read block 1002 len 32768
22159 077/300 Write block 2245 len 20480 data: 4 4 4 5 6
22160 078/300 Flush
22161 079/300 Write block 2144 len 24576 data: 4 3 5 5 5 3
22162 080/300 Read block 740 len 36864
22163 081/300 Write block 1587 len 16384 data: 7 9 9 7
22164 082/300 Write block 2850 len 12288 data: 3 3 4
22165 083/300 Write block 1354 len 36864 data: 3 4 4 3 5 4 3 4 6
22166 084/300 Read block 4557 len 12288
22167 085/300 Flush
22168 086/300 Flush
22169 087/300 Write block 1591 len 16384 data: 6 6 6 4
22170 088/300 Read block 4023 len 4096
22171 089/300 Read block 4435 len 12288
22172 090/300 Write block 895 len 4096 data: 5
22173 091/300 Read block 748 len 12288
22174 092/300 Write block 4814 len 28672 data: 5 6 6 6 6 6 7
22175 093/300 Read block 4558 len 20480
22176 094/300 Flush
22177 095/300 Read block 2280 len 20480
22178 096/300 Read block 3291 len 20480
22179 097/300 Read block 286 len 40960
22180 098/300 Read block 2087 len 16384
22181 099/300 Write block 4566 len 36864 data: 2 2 4 5 4 4 4 4 4
22182 100/300 Read block 3566 len 28672
22183 101/300 Read block 1249 len 32768
22184 102/300 Write block 3929 len 8192 data: 4 5
22185 103/300 Read block 195 len 28672
22186 104/300 Write block 3350 len 24576 data: 7 4 3 3 3 3
22187 105/300 Write block 2262 len 40960 data: 9 7 5 4 4 4 2 2 2 2
22188 106/300 Write block 1197 len 4096 data: 3
22189 107/300 Read block 2414 len 16384
22190 108/300 Read block 3000 len 8192
22191 109/300 Write block 4643 len 36864 data: 5 5 6 7 7 6 4 5 6
22192 110/300 Read block 4093 len 16384
22193 111/300 Flush
22194 112/300 Write block 214 len 4096 data: 4
22195 113/300 Flush
22196 114/300 Write block 4501 len 36864 data: 5 5 6 4 4 3 3 4 4
22197 115/300 Flush
22198 116/300 Write block 606 len 28672 data: 4 4 4 6 6 5 5
22199 117/300 Write block 1380 len 40960 data: 2 2 2 2 2 2 2 2 2 2
22200 118/300 Read block 3200 len 8192
22201 119/300 Read block 4559 len 12288
22202 120/300 Write block 316 len 40960 data: 5 5 7 5 5 6 4 4 4 5
22203 121/300 Write block 2092 len 8192 data: 3 3
22204 122/300 Read block 3989 len 12288
22205 123/300 Write block 2498 len 28672 data: 4 5 4 4 4 4 4
22206 124/300 Write block 1603 len 36864 data: 4 4 4 4 3 5 6 6 4
22207 125/300 Write block 419 len 16384 data: 7 8 9 8
22208 126/300 Read block 3486 len 40960
22209 127/300 Read block 4403 len 32768
22210 128/300 Read block 4749 len 20480
22211 129/300 Write block 3042 len 12288 data: 7 6 6
22212 130/300 Read block 2926 len 40960
22213 131/300 Write block 1246 len 28672 data: 2 2 3 4 4 6 6
22214 132/300 Write block 2038 len 28672 data: 4 4 5 5 4 3 3
22215 133/300 Write block 3875 len 28672 data: 6 6 5 4 4 3 3
22216 134/300 Read block 1934 len 8192
22217 135/300 Write block 2541 len 24576 data: 5 4 4 4 5 5
22218 136/300 Read block 2646 len 36864
22219 137/300 Read block 3173 len 40960
22220 138/300 Write block 2175 len 20480 data: 4 3 3 3 3
22221 139/300 Write block 2160 len 8192 data: 3 3
22222 140/300 Write block 2135 len 32768 data: 3 3 4 4 5 6 8 7
22223 141/300 Write block 769 len 28672 data: 6 4 4 5 5 4 3
22224 142/300 Read block 2948 len 40960
22225 143/300 Flush
22226 144/300 Read block 1117 len 28672
22227 145/300 Read block 4011 len 8192
22228 146/300 Read block 4366 len 32768
22229 147/300 Read block 877 len 28672
22230 148/300 Read block 3420 len 4096
22231 149/300 Read block 3978 len 40960
22232 150/300 Flush
22233 151/300 Read block 30 len 4096
22234 152/300 Read block 1020 len 4096
22235 153/300 Read block 1097 len 32768
22236 154/300 Write block 3185 len 36864 data: 5 5 5 4 4 3 2 3 4
22237 155/300 Read block 55 len 8192
22238 156/300 Read block 4241 len 20480
22239 157/300 Read block 1216 len 4096
22240 158/300 Write block 1165 len 4096 data: 3
22241 159/300 Write block 316 len 32768 data: 6 6 8 6 6 7 5 5
22242 160/300 Flush
22243 161/300 Read block 2350 len 20480
22244 162/300 Write block 73 len 12288 data: 6 5 4
22245 163/300 Read block 56 len 28672
22246 164/300 Write block 1974 len 28672 data: 3 3 2 2 3 4 5
22247 165/300 Write block 2897 len 24576 data: 4 6 6 8 8 7
22248 166/300 Write block 1611 len 32768 data: 5 4 4 6 5 5 4 3
22249 167/300 Write block 3852 len 16384 data: 9 6 6 5
22250 168/300 Read block 4188 len 32768
22251 169/300 Flush
22252 170/300 Read block 4380 len 20480
22253 171/300 Read block 153 len 36864
22254 172/300 Read block 2320 len 28672
22255 173/300 Write block 485 len 12288 data: 4 4 4
22256 174/300 Read block 1751 len 12288
22257 175/300 Read block 4389 len 36864
22258 176/300 Write block 2527 len 36864 data: 5 5 5 5 6 5 4 3 3
22259 177/300 Read block 2877 len 4096
22260 178/300 Read block 820 len 12288
22261 179/300 Read block 3246 len 24576
22262 180/300 Write block 4557 len 36864 data: 8 6 6 5 6 6 6 5 3
22263 181/300 Read block 4032 len 24576
22264 182/300 Write block 1221 len 8192 data: 2 2
22265 183/300 Read block 3902 len 20480
22266 184/300 Flush
22267 185/300 Write block 2894 len 36864 data: 3 3 4 5 7 7 9 9 8
22268 186/300 Read block 2372 len 16384
22269 187/300 Read block 567 len 16384
22270 188/300 Read block 4917 len 12288
22271 189/300 Read block 2720 len 36864
22272 190/300 Read block 648 len 40960
22273 191/300 Flush
22274 192/300 Read block 4783 len 28672
22275 193/300 Flush
22276 194/300 Read block 4567 len 32768
22277 195/300 Write block 1923 len 12288 data: 7 6 5
22278 196/300 Write block 2021 len 16384 data: 4 3 4 4
22279 197/300 Flush
22280 198/300 Write block 2884 len 32768 data: 4 3 3 3 3 3 2 2
22281 199/300 Read block 2434 len 32768
22282 200/300 Write block 346 len 28672 data: 6 6 5 4 3 4 4
22283 201/300 Write block 1683 len 16384 data: 2 3 3 4
22284 202/300 Read block 2604 len 8192
22285 203/300 Read block 2497 len 8192
22286 204/300 Read block 1334 len 12288
22287 205/300 Read block 2457 len 28672
22288 206/300 Write block 2397 len 16384 data: 6 6 6 6
22289 207/300 Write block 262 len 4096 data: 5
22290 208/300 Read block 2882 len 20480
22291 209/300 Read block 4357 len 20480
22292 210/300 Read block 433 len 12288
22293 211/300 Write block 3208 len 12288 data: 3 3 3
22294 212/300 Write block 1548 len 32768 data: 5 6 6 6 6 7 7 6
22295 213/300 Read block 55 len 4096
22296 214/300 Write block 3477 len 16384 data: 5 4 5 4
22297 215/300 Flush
22298 216/300 Read block 3577 len 40960
22299 217/300 Flush
22300 218/300 Write block 1426 len 20480 data: 4 5 4 3 2
22301 219/300 Write block 2955 len 36864 data: 5 4 4 4 6 7 5 6 5
22302 220/300 Read block 3349 len 8192
22303 221/300 Read block 606 len 12288
22304 222/300 Read block 1952 len 4096
22305 223/300 Read block 2534 len 40960
22306 224/300 Read block 1389 len 20480
22307 225/300 Read block 1379 len 40960
22308 226/300 Read block 373 len 32768
22309 227/300 Write block 1081 len 12288 data: 5 5 5
22310 228/300 Flush
22311 229/300 Write block 3786 len 24576 data: 5 4 5 5 5 5
22312 230/300 Read block 1185 len 32768
22313 231/300 Write block 3763 len 20480 data: 4 3 3 3 4
22314 232/300 Write block 166 len 40960 data: 4 4 4 4 3 3 2 2 2 3
22315 233/300 Read block 941 len 36864
22316 234/300 Write block 3887 len 4096 data: 3
22317 235/300 Write block 1500 len 4096 data: 4
22318 236/300 Write block 803 len 16384 data: 3 3 3 3
22319 237/300 Read block 18 len 36864
22320 238/300 Write block 3677 len 36864 data: 2 2 2 3 4 6 7 6 6
22321 239/300 Flush
22322 240/300 Write block 3202 len 28672 data: 6 6 6 5 5 3 4
22323 241/300 Read block 1991 len 28672
22324 242/300 Read block 4638 len 24576
22325 243/300 Write block 1233 len 8192 data: 5 6
22326 244/300 Read block 3812 len 8192
22327 245/300 Read block 1232 len 36864
22328 246/300 Flush
22329 247/300 Read block 2630 len 32768
22330 248/300 Flush
22331 249/300 Flush
22332 250/300 Read block 3084 len 4096
22333 251/300 Write block 4750 len 16384 data: 4 5 5 4
22334 252/300 Write block 2613 len 12288 data: 4 4 4
22335 253/300 Read block 3410 len 8192
22336 254/300 Write block 3602 len 24576 data: 6 4 4 5 3 3
22337 255/300 Flush
22338 256/300 Write block 3847 len 20480 data: 4 6 7 8 7
22339 257/300 Read block 4988 len 36864
22340 258/300 Read block 1885 len 8192
22341 259/300 Write block 1304 len 4096 data: 2
22342 260/300 Write block 3883 len 32768 data: 4 4 3 3 4 3 3 3
22343 261/300 Read block 2052 len 24576
22344 262/300 Read block 2132 len 40960
22345 263/300 Write block 1789 len 40960 data: 3 3 5 4 5 5 5 4 3 3
22346 264/300 Read block 3367 len 28672
22347 265/300 Write block 4248 len 8192 data: 5 5
22348 266/300 Write block 2114 len 4096 data: 4
22349 267/300 Write block 3428 len 40960 data: 6 6 6 6 5 5 5 5 4 6
22350 268/300 Write block 4991 len 12288 data: 5 4 4
22351 269/300 Read block 1249 len 16384
22352 270/300 Write block 3182 len 12288 data: 6 5 5
22353 271/300 Read block 4239 len 28672
22354 272/300 Write block 4537 len 24576 data: 4 7 7 7 4 4
22355 273/300 Read block 4039 len 24576
22356 274/300 Read block 1434 len 40960
22357 275/300 Write block 2325 len 28672 data: 3 2 2 2 2 2 2
22358 276/300 Write block 837 len 32768 data: 4 5 5 5 5 4 4 4
22359 277/300 Write block 1406 len 8192 data: 3 3
22360 278/300 Write block 592 len 40960 data: 7 8 8 8 7 6 5 4 3 2
22361 279/300 Write block 3891 len 24576 data: 4 3 2 2 2 2
22362 280/300 Read block 4623 len 16384
22363 281/300 Read block 3126 len 24576
22364 282/300 Read block 2867 len 16384
22365 283/300 Flush
22366 284/300 Write block 3524 len 8192 data: 5 8
22367 285/300 Write block 2826 len 12288 data: 7 6 5
22368 286/300 Read block 3776 len 12288
22369 287/300 Write block 4927 len 12288 data: 4 4 4
22370 288/300 Write block 301 len 24576 data: 4 4 2 2 2 2
22371 289/300 Read block 4061 len 16384
22372 290/300 Write block 4364 len 20480 data: 7 6 6 6 5
22373 291/300 Read block 1322 len 24576
22374 292/300 Read block 3420 len 36864
22375 293/300 Read block 3558 len 28672
22376 294/300 Write block 98 len 4096 data: 4
22377 295/300 Read block 3080 len 32768
22378 296/300 Read block 3460 len 40960
22379 297/300 Flush
22380 298/300 Flush
22381 299/300 Read block 3461 len 32768
22382 300/300 Read block 4743 len 28672
22383 301/300 Write block 40 len 4096 data: 5
22384 Main task has work to do, go find it
22385 got dsc Start(2)
22386 start 2
22387 Mar 17 22:23:58.657 INFO request completed, latency_us: 207, response_code: 204, uri: /start/cid/2, method: POST, req_id: 8cb68b45-9a95-4613-b4cd-048bb67e12a3, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
22388 Replay: started 2, returned:Ok(())
22389 ----------------------------------------------------------------
22390 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22391 States: Active Active Offline
22392 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22393 6132 Acked Read 6 Done Done Sent false
22394 6133 Acked Read 7 Done Done Sent false
22395 6134 Acked Read 8 Done Done Sent false
22396 6135 Acked Write 8 Done Done Sent false
22397 6136 Acked Write 8 Done Done Sent false
22398 6137 Acked Read 8 Done Done Sent false
22399 6138 Acked Write 10 Done Done Sent false
22400 6139 Acked Write 6 Done Done Sent false
22401 6140 Acked Read 5 Done Done Sent false
22402 6141 Acked Write 4 Done Done Sent false
22403 6142 Acked Read 9 Done Done Sent false
22404 6143 Acked Write 9 Done Done Sent false
22405 6144 Acked Flush 0 Done Done Sent false
22406 6145 Acked Write 4 Done Done Sent false
22407 6146 Acked Read 6 Done Done Sent false
22408 6147 Acked Write 3 Done Done Sent false
22409 6148 Acked Read 6 Done Done Sent false
22410 6149 Acked Read 10 Done Done Sent false
22411 6150 Acked Read 2 Done Done Sent false
22412 6151 Acked Flush 0 Done Done Sent false
22413 6152 Acked Read 10 Done Done Sent false
22414 6153 Acked Read 3 Done Done Sent false
22415 6154 Acked Read 3 Done Done Sent false
22416 6155 Acked Read 7 Done Done Sent false
22417 6156 Acked Write 9 Done Done Sent false
22418 6157 Acked Read 9 Done Done Sent false
22419 6158 Acked Read 10 Done Done Sent false
22420 6159 Acked Write 4 Done Done Sent false
22421 6160 Acked Flush 0 Done Done Sent false
22422 6161 Acked Write 2 Done Done Sent false
22423 6162 Acked Read 10 Done Done Sent false
22424 6163 Acked Write 8 Done Done Sent false
22425 6164 Acked Write 7 Done Done Sent false
22426 6165 Acked Read 3 Done Done Sent false
22427 6166 Acked Read 6 Done Done Sent false
22428 6167 Acked Flush 0 Done Done Sent false
22429 6168 Acked Write 2 Done Done Sent false
22430 6169 Acked Read 5 Done Done Sent false
22431 6170 Acked Read 9 Done Done Sent false
22432 6171 Acked Flush 0 Done Done Sent false
22433 6172 Acked Write 3 Done Done Sent false
22434 6173 Acked Read 9 Done Done Sent false
22435 6174 Acked Flush 0 Done Done Sent false
22436 6175 Acked Flush 0 Done Done Sent false
22437 6176 Acked Read 2 Done Done Sent false
22438 6177 Acked Write 2 Done Done Sent false
22439 6178 Acked Read 4 Done Done Sent false
22440 6179 Acked Read 6 Done Done Sent false
22441 6180 Acked Read 7 Done Done Sent false
22442 6181 Acked Write 6 Done Done Sent false
22443 6182 Acked Write 1 Done Done Sent false
22444 6183 Acked Read 4 Done Done Sent false
22445 6184 Acked Write 3 Done Done Sent false
22446 6185 Acked Write 4 Done Done Sent false
22447 6186 Acked Write 10 Done Done Sent false
22448 6187 Acked Read 1 Done Done Sent false
22449 6188 Acked Write 5 Done Done Sent false
22450 6189 Acked Flush 0 Done Done Sent false
22451 6190 Acked Read 7 Done Done Sent false
22452 6191 Acked Read 5 Done Done Sent false
22453 6192 Acked Write 7 Done Done Sent false
22454 6193 Acked Write 4 Done Done Sent false
22455 6194 Acked Read 8 Done Done Sent false
22456 6195 Acked Read 5 Done Done Sent false
22457 6196 Acked Write 7 Done Done Sent false
22458 6197 Acked Write 9 Done Done Sent false
22459 6198 Acked Read 8 Done Done Sent false
22460 6199 Acked Read 8 Done Done Sent false
22461 6200 Acked Write 1 Done Done Sent false
22462 6201 Acked Read 1 Done Done Sent false
22463 6202 Acked Read 4 Done Done Sent false
22464 6203 Acked Write 2 Done Done Sent false
22465 6204 Acked Write 2 Done Done Sent false
22466 6205 Acked Write 3 Done Done Sent false
22467 6206 Acked Read 5 Done Done Sent false
22468 6207 Acked Read 8 Done Done Sent false
22469 6208 Acked Write 5 Done Done Sent false
22470 6209 Acked Flush 0 Done Done Sent false
22471 6210 Acked Write 6 Done Done Sent false
22472 6211 Acked Read 9 Done Done Sent false
22473 6212 Acked Write 4 Done Done Sent false
22474 6213 Acked Write 3 Done Done Sent false
22475 6214 Acked Write 9 Done Done Sent false
22476 6215 Acked Read 3 Done Done Sent false
22477 6216 Acked Flush 0 Done Done Sent false
22478 6217 Acked Flush 0 Done Done Sent false
22479 6218 Acked Write 4 Done Done Sent false
22480 6219 Acked Read 1 Done Done Sent false
22481 6220 Acked Read 3 Done Done Sent false
22482 6221 Acked Write 1 Done Done Sent false
22483 6222 Acked Read 3 Done Done Sent false
22484 6223 Acked Write 7 Done Done Sent false
22485 6224 Acked Read 5 Done Done Sent false
22486 6225 Acked Flush 0 Done Done Sent false
22487 6226 Acked Read 5 Done Done Sent false
22488 6227 Acked Read 5 Done Done Sent false
22489 6228 Acked Read 10 Done Done Sent false
22490 6229 Acked Read 4 Done Done Sent false
22491 6230 Acked Write 9 Done Done Sent false
22492 6231 Acked Read 7 Done Done Sent false
22493 6232 Acked Read 8 Done Done Sent false
22494 6233 Acked Write 2 Done Done Sent false
22495 6234 Acked Read 7 Done Done Sent false
22496 6235 Acked Write 6 Done Done Sent false
22497 6236 Acked Write 10 Done Done Sent false
22498 6237 Acked Write 1 Done Done Sent false
22499 6238 Acked Read 4 Done Done Sent false
22500 6239 Acked Read 2 Done Done Sent false
22501 6240 Acked Write 9 Done Done Sent false
22502 6241 Acked Read 4 Done Done Sent false
22503 6242 Acked Flush 0 Done Done Sent false
22504 6243 Acked Write 1 Done Done Sent false
22505 6244 Acked Flush 0 Done Done Sent false
22506 6245 Acked Write 9 Done Done Sent false
22507 6246 Acked Flush 0 Done Done Sent false
22508 6247 Acked Write 7 Done Done Sent false
22509 6248 Acked Write 10 Done Done Sent false
22510 6249 Acked Read 2 Done Done Sent false
22511 6250 Acked Read 3 Done Done Sent false
22512 6251 Acked Write 10 Done Done Sent false
22513 6252 Acked Write 2 Done Done Sent false
22514 6253 Acked Read 3 Done Done Sent false
22515 6254 Acked Write 7 Done Done Sent false
22516 6255 Acked Write 9 Done Done Sent false
22517 6256 Acked Write 4 Done Done Sent false
22518 6257 Acked Read 10 Done Done Sent false
22519 6258 Acked Read 8 Done Done Sent false
22520 6259 Acked Read 5 Done Done Sent false
22521 6260 Acked Write 3 Done Done Sent false
22522 6261 Acked Read 10 Done Done Sent false
22523 6262 Acked Write 7 Done Done Sent false
22524 6263 Acked Write 7 Done Done Sent false
22525 6264 Acked Write 7 Done Done Sent false
22526 6265 Acked Read 2 Done Done Sent false
22527 6266 Acked Write 6 Done Done Sent false
22528 6267 Acked Read 9 Done Done Sent false
22529 6268 Acked Read 10 Done Done Sent false
22530 6269 Acked Write 5 Done Done Sent false
22531 6270 Acked Write 2 Done Done Sent false
22532 6271 Acked Write 8 Done Done Sent false
22533 6272 Acked Write 7 Done Done Sent false
22534 6273 Acked Read 10 Done Done Sent false
22535 6274 Acked Flush 0 Done Done Sent false
22536 6275 Acked Read 7 Done Done Sent false
22537 6276 Acked Read 2 Done Done Sent false
22538 6277 Acked Read 8 Done Done Sent false
22539 6278 Acked Read 7 Done Done Sent false
22540 6279 Acked Read 1 Done Done Sent false
22541 6280 Acked Read 10 Done Done Sent false
22542 6281 Acked Flush 0 Done Done Sent false
22543 6282 Acked Read 1 Done Done Sent false
22544 6283 Acked Read 1 Done Done Sent false
22545 6284 Acked Read 8 Done Done Sent false
22546 6285 Acked Write 9 Done Done Sent false
22547 6286 Acked Read 2 Done Done Sent false
22548 6287 Acked Read 5 Done Done Sent false
22549 6288 Acked Read 1 Done Done Sent false
22550 6289 Acked Write 1 Done Done Sent false
22551 6290 Acked Write 8 Done Done Sent false
22552 6291 Acked Flush 0 Done Done Sent false
22553 6292 Acked Read 5 Done Done Sent false
22554 6293 Acked Write 3 Done Done Sent false
22555 6294 Acked Read 7 Done Done Sent false
22556 6295 Acked Write 7 Done Done Sent false
22557 6296 Acked Write 6 Done Done Sent false
22558 6297 Acked Write 8 Done Done Sent false
22559 6298 Acked Write 4 Done Done Sent false
22560 6299 Acked Read 8 Done Done Sent false
22561 6300 Acked Flush 0 Done Done Sent false
22562 6301 Acked Read 5 Done Done Sent false
22563 6302 Acked Read 9 Done Done Sent false
22564 6303 Acked Read 7 Done Done Sent false
22565 6304 Acked Write 3 Done Done Sent false
22566 6305 Acked Read 3 Done Done Sent false
22567 6306 Acked Read 9 Done Done Sent false
22568 6307 Acked Write 9 Done Done Sent false
22569 6308 Acked Read 1 Done Done Sent false
22570 6309 Acked Read 3 Done Done Sent false
22571 6310 Acked Read 6 Done Done Sent false
22572 6311 Acked Write 9 Done Done Sent false
22573 6312 Acked Read 6 Done Done Sent false
22574 6313 Acked Write 2 Done Done Sent false
22575 6314 Acked Read 5 Done Done Sent false
22576 6315 Acked Flush 0 Done Done Sent false
22577 6316 Acked Write 9 Done Done Sent false
22578 6317 Acked Read 4 Done Done Sent false
22579 6318 Acked Read 4 Done Done Sent false
22580 6319 Acked Read 3 Done Done Sent false
22581 6320 Acked Read 9 Done Done Sent false
22582 6321 Acked Read 10 Done Done Sent false
22583 6322 Acked Flush 0 Done Done Sent false
22584 6323 Acked Read 7 Done Done Sent false
22585 6324 Acked Flush 0 Done Done Sent false
22586 6325 Acked Read 8 Done Done Sent false
22587 6326 Acked Write 3 Done Done Sent false
22588 6327 Acked Write 4 Done Done Sent false
22589 6328 Acked Flush 0 Done Done Sent false
22590 6329 Acked Write 8 Done Done Sent false
22591 6330 Acked Read 8 Done Done Sent false
22592 6331 Acked Write 7 Done Done Sent false
22593 6332 Acked Write 4 Done Done Sent false
22594 6333 Acked Read 2 Done Done Sent false
22595 6334 Acked Read 2 Done Done Sent false
22596 6335 Acked Read 3 Done Done Sent false
22597 6336 Acked Read 7 Done Done Sent false
22598 6337 Acked Write 4 Done Done Sent false
22599 6338 Acked Write 1 Done Done Sent false
22600 6339 Acked Read 5 Done Done Sent false
22601 6340 Acked Read 5 Done Done Sent false
22602 6341 Acked Read 3 Done Done Sent false
22603 6342 Acked Write 3 Done Done Sent false
22604 6343 Acked Write 8 Done Done Sent false
22605 6344 Acked Read 1 Done Done Sent false
22606 6345 Acked Write 4 Done Done Sent false
22607 6346 Acked Flush 0 Done Done Sent false
22608 6347 Acked Read 10 Done Done Sent false
22609 6348 Acked Flush 0 Done Done Sent false
22610 6349 Acked Write 5 Done Done Sent false
22611 6350 Acked Write 9 Done Done Sent false
22612 6351 Acked Read 2 Done Done Sent false
22613 6352 Acked Read 3 Done Done Sent false
22614 6353 Acked Read 1 Done Done Sent false
22615 6354 Acked Read 10 Done Done Sent false
22616 6355 Acked Read 5 Done Done Sent false
22617 6356 Acked Read 10 Done Done Sent false
22618 6357 Acked Read 8 Done Done Sent false
22619 6358 Acked Write 3 Done Done Sent false
22620 6359 Acked Flush 0 Done Done Sent false
22621 6360 Acked Write 6 Done Done Sent false
22622 6361 Acked Read 8 Done Done Sent false
22623 6362 Acked Write 5 Done Done Sent false
22624 6363 Acked Write 10 Done Done Sent false
22625 6364 Acked Read 9 Done Done Sent false
22626 6365 Acked Write 1 Done Done Sent false
22627 6366 Acked Write 1 Done Done Sent false
22628 6367 Acked Write 4 Done Done Sent false
22629 6368 Acked Read 9 Done Done Sent false
22630 6369 Acked Write 9 Done Done Sent false
22631 6370 Acked Flush 0 Done Done Sent false
22632 6371 Acked Write 7 Done Done Sent false
22633 6372 Acked Read 7 Done Done Sent false
22634 6373 Acked Read 6 Done Done Sent false
22635 6374 Acked Write 2 Done Done Sent false
22636 6375 Acked Read 2 Done Done Sent false
22637 6376 Acked Read 9 Done Done Sent false
22638 6377 Acked Flush 0 Done Done Sent false
22639 6378 Acked Read 8 Done Done Sent false
22640 6379 Acked Flush 0 Done Done Sent false
22641 6380 Acked Flush 0 Done Done Sent false
22642 6381 Acked Read 1 Done Done Sent false
22643 6382 Acked Write 4 Done Done Sent false
22644 6383 Acked Write 3 Done Done Sent false
22645 6384 Acked Read 2 Done Done Sent false
22646 6385 Acked Write 6 Done Done Sent false
22647 6386 Acked Flush 0 Done Done Sent false
22648 6387 Acked Write 5 Done Done Sent false
22649 6388 Acked Read 9 Done Done Sent false
22650 6389 Acked Read 2 Done Done Sent false
22651 6390 Acked Write 1 Done Done Sent false
22652 6391 Acked Write 8 Done Done Sent false
22653 6392 Acked Read 6 Done Done Sent false
22654 6393 Acked Read 10 Done Done Sent false
22655 6394 Acked Write 10 Done Done Sent false
22656 6395 Acked Read 7 Done Done Sent false
22657 6396 Acked Write 2 Done Done Sent false
22658 6397 Acked Write 1 Done Done Sent false
22659 6398 Acked Write 10 Done Done Sent false
22660 6399 Acked Write 3 Done Done Sent false
22661 6400 Acked Read 4 Done Done Sent false
22662 6401 Acked Write 3 Done Done Sent false
22663 6402 Acked Read 7 Done Done Sent false
22664 6403 Acked Write 6 Done Done Sent false
22665 6404 Acked Read 6 Done Done Sent false
22666 6405 Acked Read 10 Done Done Sent false
22667 6406 Acked Write 7 Done Done Sent false
22668 6407 Acked Write 8 Done Done Sent false
22669 6408 Acked Write 2 Done Done Sent false
22670 6409 Acked Write 10 Done Done Sent false
22671 6410 Acked Write 6 Done Done Sent false
22672 6411 Acked Read 4 Done Done Sent false
22673 6412 Acked Read 6 Done Done Sent false
22674 6413 Acked Read 4 Done Done Sent false
22675 6414 Acked Flush 0 Done Done Sent false
22676 6415 Acked Write 2 Done Done Sent false
22677 6416 Acked Write 3 Done Done Sent false
22678 6417 Acked Read 3 Done Done Sent false
22679 6418 Acked Write 3 Done Done Sent false
22680 6419 Acked Write 6 Done Done Sent false
22681 6420 Acked Read 4 Done Done Sent false
22682 6421 Acked Write 5 Done Done Sent false
22683 6422 Acked Read 6 Done Done Sent false
22684 6423 Acked Read 9 Done Done Sent false
22685 6424 Acked Read 7 Done Done Sent false
22686 6425 Acked Write 1 Done Done Sent false
22687 6426 Acked Read 8 Done Done Sent false
22688 6427 Acked Read 10 Done Done Sent false
22689 6428 Acked Flush 0 Done Done Sent false
22690 6429 Acked Flush 0 Done Done Sent false
22691 6430 Acked Read 8 Done Done Sent false
22692 6431 Acked Read 7 Done Done Sent false
22693 6432 Acked Write 1 Done Done Sent false
22694 STATES DS:0 DS:1 DS:2 TOTAL
22695 Sent 0 0 301 301
22696 Done 301 301 0 602
22697 Skipped 0 0 0 0
22698 Error 0 0 0 0
22699 Last Flush: Some(JobId(6429)) Some(JobId(6429)) Some(JobId(6131))
22700 Downstairs last five completed: 6131 6130 6129 6128 6127
22701 Upstairs last five completed: 6432 6431 6430 6429 6428
22702 CLIENT: Up:0 ds:301 act:2
227032025-03-17T22:23:59.159ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
227042025-03-17T22:24:00.056ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
227052025-03-17T22:24:01.400ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
227062025-03-17T22:24:02.415ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
22707 [8830] got start action
22708 [8830] I am going to restart
22709 Starting downstairs at port 8830
22710 Make output file at "/tmp/dsc/downstairs-8830.txt"
22711 [8830][2] reports Starting
22712 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1229)
22713 [8830][2] reports Running
22714 ----------------------------------------------------------------
22715 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22716 States: Active Active Offline
22717 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22718 6132 Acked Read 6 Done Done Sent false
22719 6133 Acked Read 7 Done Done Sent false
22720 6134 Acked Read 8 Done Done Sent false
22721 6135 Acked Write 8 Done Done Sent false
22722 6136 Acked Write 8 Done Done Sent false
22723 6137 Acked Read 8 Done Done Sent false
22724 6138 Acked Write 10 Done Done Sent false
22725 6139 Acked Write 6 Done Done Sent false
22726 6140 Acked Read 5 Done Done Sent false
22727 6141 Acked Write 4 Done Done Sent false
22728 6142 Acked Read 9 Done Done Sent false
22729 6143 Acked Write 9 Done Done Sent false
22730 6144 Acked Flush 0 Done Done Sent false
22731 6145 Acked Write 4 Done Done Sent false
22732 6146 Acked Read 6 Done Done Sent false
22733 6147 Acked Write 3 Done Done Sent false
22734 6148 Acked Read 6 Done Done Sent false
22735 6149 Acked Read 10 Done Done Sent false
22736 6150 Acked Read 2 Done Done Sent false
22737 6151 Acked Flush 0 Done Done Sent false
22738 6152 Acked Read 10 Done Done Sent false
22739 6153 Acked Read 3 Done Done Sent false
22740 6154 Acked Read 3 Done Done Sent false
22741 6155 Acked Read 7 Done Done Sent false
22742 6156 Acked Write 9 Done Done Sent false
22743 6157 Acked Read 9 Done Done Sent false
22744 6158 Acked Read 10 Done Done Sent false
22745 6159 Acked Write 4 Done Done Sent false
22746 6160 Acked Flush 0 Done Done Sent false
22747 6161 Acked Write 2 Done Done Sent false
22748 6162 Acked Read 10 Done Done Sent false
22749 6163 Acked Write 8 Done Done Sent false
22750 6164 Acked Write 7 Done Done Sent false
22751 6165 Acked Read 3 Done Done Sent false
22752 6166 Acked Read 6 Done Done Sent false
22753 6167 Acked Flush 0 Done Done Sent false
22754 6168 Acked Write 2 Done Done Sent false
22755 6169 Acked Read 5 Done Done Sent false
22756 6170 Acked Read 9 Done Done Sent false
22757 6171 Acked Flush 0 Done Done Sent false
22758 6172 Acked Write 3 Done Done Sent false
22759 6173 Acked Read 9 Done Done Sent false
22760 6174 Acked Flush 0 Done Done Sent false
22761 6175 Acked Flush 0 Done Done Sent false
22762 6176 Acked Read 2 Done Done Sent false
22763 6177 Acked Write 2 Done Done Sent false
22764 6178 Acked Read 4 Done Done Sent false
22765 6179 Acked Read 6 Done Done Sent false
22766 6180 Acked Read 7 Done Done Sent false
22767 6181 Acked Write 6 Done Done Sent false
22768 6182 Acked Write 1 Done Done Sent false
22769 6183 Acked Read 4 Done Done Sent false
22770 6184 Acked Write 3 Done Done Sent false
22771 6185 Acked Write 4 Done Done Sent false
22772 6186 Acked Write 10 Done Done Sent false
22773 6187 Acked Read 1 Done Done Sent false
22774 6188 Acked Write 5 Done Done Sent false
22775 6189 Acked Flush 0 Done Done Sent false
22776 6190 Acked Read 7 Done Done Sent false
22777 6191 Acked Read 5 Done Done Sent false
22778 6192 Acked Write 7 Done Done Sent false
22779 6193 Acked Write 4 Done Done Sent false
22780 6194 Acked Read 8 Done Done Sent false
22781 6195 Acked Read 5 Done Done Sent false
22782 6196 Acked Write 7 Done Done Sent false
22783 6197 Acked Write 9 Done Done Sent false
22784 6198 Acked Read 8 Done Done Sent false
22785 6199 Acked Read 8 Done Done Sent false
22786 6200 Acked Write 1 Done Done Sent false
22787 6201 Acked Read 1 Done Done Sent false
22788 6202 Acked Read 4 Done Done Sent false
22789 6203 Acked Write 2 Done Done Sent false
22790 6204 Acked Write 2 Done Done Sent false
22791 6205 Acked Write 3 Done Done Sent false
22792 6206 Acked Read 5 Done Done Sent false
22793 6207 Acked Read 8 Done Done Sent false
22794 6208 Acked Write 5 Done Done Sent false
22795 6209 Acked Flush 0 Done Done Sent false
22796 6210 Acked Write 6 Done Done Sent false
22797 6211 Acked Read 9 Done Done Sent false
22798 6212 Acked Write 4 Done Done Sent false
22799 6213 Acked Write 3 Done Done Sent false
22800 6214 Acked Write 9 Done Done Sent false
22801 6215 Acked Read 3 Done Done Sent false
22802 6216 Acked Flush 0 Done Done Sent false
22803 6217 Acked Flush 0 Done Done Sent false
22804 6218 Acked Write 4 Done Done Sent false
22805 6219 Acked Read 1 Done Done Sent false
22806 6220 Acked Read 3 Done Done Sent false
22807 6221 Acked Write 1 Done Done Sent false
22808 6222 Acked Read 3 Done Done Sent false
22809 6223 Acked Write 7 Done Done Sent false
22810 6224 Acked Read 5 Done Done Sent false
22811 6225 Acked Flush 0 Done Done Sent false
22812 6226 Acked Read 5 Done Done Sent false
22813 6227 Acked Read 5 Done Done Sent false
22814 6228 Acked Read 10 Done Done Sent false
22815 6229 Acked Read 4 Done Done Sent false
22816 6230 Acked Write 9 Done Done Sent false
22817 6231 Acked Read 7 Done Done Sent false
22818 6232 Acked Read 8 Done Done Sent false
22819 6233 Acked Write 2 Done Done Sent false
22820 6234 Acked Read 7 Done Done Sent false
22821 6235 Acked Write 6 Done Done Sent false
22822 6236 Acked Write 10 Done Done Sent false
22823 6237 Acked Write 1 Done Done Sent false
22824 6238 Acked Read 4 Done Done Sent false
22825 6239 Acked Read 2 Done Done Sent false
22826 6240 Acked Write 9 Done Done Sent false
22827 6241 Acked Read 4 Done Done Sent false
22828 6242 Acked Flush 0 Done Done Sent false
22829 6243 Acked Write 1 Done Done Sent false
22830 6244 Acked Flush 0 Done Done Sent false
22831 6245 Acked Write 9 Done Done Sent false
22832 6246 Acked Flush 0 Done Done Sent false
22833 6247 Acked Write 7 Done Done Sent false
22834 6248 Acked Write 10 Done Done Sent false
22835 6249 Acked Read 2 Done Done Sent false
22836 6250 Acked Read 3 Done Done Sent false
22837 6251 Acked Write 10 Done Done Sent false
22838 6252 Acked Write 2 Done Done Sent false
22839 6253 Acked Read 3 Done Done Sent false
22840 6254 Acked Write 7 Done Done Sent false
22841 6255 Acked Write 9 Done Done Sent false
22842 6256 Acked Write 4 Done Done Sent false
22843 6257 Acked Read 10 Done Done Sent false
22844 6258 Acked Read 8 Done Done Sent false
22845 6259 Acked Read 5 Done Done Sent false
22846 6260 Acked Write 3 Done Done Sent false
22847 6261 Acked Read 10 Done Done Sent false
22848 6262 Acked Write 7 Done Done Sent false
22849 6263 Acked Write 7 Done Done Sent false
22850 6264 Acked Write 7 Done Done Sent false
22851 6265 Acked Read 2 Done Done Sent false
22852 6266 Acked Write 6 Done Done Sent false
22853 6267 Acked Read 9 Done Done Sent false
22854 6268 Acked Read 10 Done Done Sent false
22855 6269 Acked Write 5 Done Done Sent false
22856 6270 Acked Write 2 Done Done Sent false
22857 6271 Acked Write 8 Done Done Sent false
22858 6272 Acked Write 7 Done Done Sent false
22859 6273 Acked Read 10 Done Done Sent false
22860 6274 Acked Flush 0 Done Done Sent false
22861 6275 Acked Read 7 Done Done Sent false
22862 6276 Acked Read 2 Done Done Sent false
22863 6277 Acked Read 8 Done Done Sent false
22864 6278 Acked Read 7 Done Done Sent false
22865 6279 Acked Read 1 Done Done Sent false
22866 6280 Acked Read 10 Done Done Sent false
22867 6281 Acked Flush 0 Done Done Sent false
22868 6282 Acked Read 1 Done Done Sent false
22869 6283 Acked Read 1 Done Done Sent false
22870 6284 Acked Read 8 Done Done Sent false
22871 6285 Acked Write 9 Done Done Sent false
22872 6286 Acked Read 2 Done Done Sent false
22873 6287 Acked Read 5 Done Done Sent false
22874 6288 Acked Read 1 Done Done Sent false
22875 6289 Acked Write 1 Done Done Sent false
22876 6290 Acked Write 8 Done Done Sent false
22877 6291 Acked Flush 0 Done Done Sent false
22878 6292 Acked Read 5 Done Done Sent false
22879 6293 Acked Write 3 Done Done Sent false
22880 6294 Acked Read 7 Done Done Sent false
22881 6295 Acked Write 7 Done Done Sent false
22882 6296 Acked Write 6 Done Done Sent false
22883 6297 Acked Write 8 Done Done Sent false
22884 6298 Acked Write 4 Done Done Sent false
22885 6299 Acked Read 8 Done Done Sent false
22886 6300 Acked Flush 0 Done Done Sent false
22887 6301 Acked Read 5 Done Done Sent false
22888 6302 Acked Read 9 Done Done Sent false
22889 6303 Acked Read 7 Done Done Sent false
22890 6304 Acked Write 3 Done Done Sent false
22891 6305 Acked Read 3 Done Done Sent false
22892 6306 Acked Read 9 Done Done Sent false
22893 6307 Acked Write 9 Done Done Sent false
22894 6308 Acked Read 1 Done Done Sent false
22895 6309 Acked Read 3 Done Done Sent false
22896 6310 Acked Read 6 Done Done Sent false
22897 6311 Acked Write 9 Done Done Sent false
22898 6312 Acked Read 6 Done Done Sent false
22899 6313 Acked Write 2 Done Done Sent false
22900 6314 Acked Read 5 Done Done Sent false
22901 6315 Acked Flush 0 Done Done Sent false
22902 6316 Acked Write 9 Done Done Sent false
22903 6317 Acked Read 4 Done Done Sent false
22904 6318 Acked Read 4 Done Done Sent false
22905 6319 Acked Read 3 Done Done Sent false
22906 6320 Acked Read 9 Done Done Sent false
22907 6321 Acked Read 10 Done Done Sent false
22908 6322 Acked Flush 0 Done Done Sent false
22909 6323 Acked Read 7 Done Done Sent false
22910 6324 Acked Flush 0 Done Done Sent false
22911 6325 Acked Read 8 Done Done Sent false
22912 6326 Acked Write 3 Done Done Sent false
22913 6327 Acked Write 4 Done Done Sent false
22914 6328 Acked Flush 0 Done Done Sent false
22915 6329 Acked Write 8 Done Done Sent false
22916 6330 Acked Read 8 Done Done Sent false
22917 6331 Acked Write 7 Done Done Sent false
22918 6332 Acked Write 4 Done Done Sent false
22919 6333 Acked Read 2 Done Done Sent false
22920 6334 Acked Read 2 Done Done Sent false
22921 6335 Acked Read 3 Done Done Sent false
22922 6336 Acked Read 7 Done Done Sent false
22923 6337 Acked Write 4 Done Done Sent false
22924 6338 Acked Write 1 Done Done Sent false
22925 6339 Acked Read 5 Done Done Sent false
22926 6340 Acked Read 5 Done Done Sent false
22927 6341 Acked Read 3 Done Done Sent false
22928 6342 Acked Write 3 Done Done Sent false
22929 6343 Acked Write 8 Done Done Sent false
22930 6344 Acked Read 1 Done Done Sent false
22931 6345 Acked Write 4 Done Done Sent false
22932 6346 Acked Flush 0 Done Done Sent false
22933 6347 Acked Read 10 Done Done Sent false
22934 6348 Acked Flush 0 Done Done Sent false
22935 6349 Acked Write 5 Done Done Sent false
22936 6350 Acked Write 9 Done Done Sent false
22937 6351 Acked Read 2 Done Done Sent false
22938 6352 Acked Read 3 Done Done Sent false
22939 6353 Acked Read 1 Done Done Sent false
22940 6354 Acked Read 10 Done Done Sent false
22941 6355 Acked Read 5 Done Done Sent false
22942 6356 Acked Read 10 Done Done Sent false
22943 6357 Acked Read 8 Done Done Sent false
22944 6358 Acked Write 3 Done Done Sent false
22945 6359 Acked Flush 0 Done Done Sent false
22946 6360 Acked Write 6 Done Done Sent false
22947 6361 Acked Read 8 Done Done Sent false
22948 6362 Acked Write 5 Done Done Sent false
22949 6363 Acked Write 10 Done Done Sent false
22950 6364 Acked Read 9 Done Done Sent false
22951 6365 Acked Write 1 Done Done Sent false
22952 6366 Acked Write 1 Done Done Sent false
22953 6367 Acked Write 4 Done Done Sent false
22954 6368 Acked Read 9 Done Done Sent false
22955 6369 Acked Write 9 Done Done Sent false
22956 6370 Acked Flush 0 Done Done Sent false
22957 6371 Acked Write 7 Done Done Sent false
22958 6372 Acked Read 7 Done Done Sent false
22959 6373 Acked Read 6 Done Done Sent false
22960 6374 Acked Write 2 Done Done Sent false
22961 6375 Acked Read 2 Done Done Sent false
22962 6376 Acked Read 9 Done Done Sent false
22963 6377 Acked Flush 0 Done Done Sent false
22964 6378 Acked Read 8 Done Done Sent false
22965 6379 Acked Flush 0 Done Done Sent false
22966 6380 Acked Flush 0 Done Done Sent false
22967 6381 Acked Read 1 Done Done Sent false
22968 6382 Acked Write 4 Done Done Sent false
22969 6383 Acked Write 3 Done Done Sent false
22970 6384 Acked Read 2 Done Done Sent false
22971 6385 Acked Write 6 Done Done Sent false
22972 6386 Acked Flush 0 Done Done Sent false
22973 6387 Acked Write 5 Done Done Sent false
22974 6388 Acked Read 9 Done Done Sent false
22975 6389 Acked Read 2 Done Done Sent false
22976 6390 Acked Write 1 Done Done Sent false
22977 6391 Acked Write 8 Done Done Sent false
22978 6392 Acked Read 6 Done Done Sent false
22979 6393 Acked Read 10 Done Done Sent false
22980 6394 Acked Write 10 Done Done Sent false
22981 6395 Acked Read 7 Done Done Sent false
22982 6396 Acked Write 2 Done Done Sent false
22983 6397 Acked Write 1 Done Done Sent false
22984 6398 Acked Write 10 Done Done Sent false
22985 6399 Acked Write 3 Done Done Sent false
22986 6400 Acked Read 4 Done Done Sent false
22987 6401 Acked Write 3 Done Done Sent false
22988 6402 Acked Read 7 Done Done Sent false
22989 6403 Acked Write 6 Done Done Sent false
22990 6404 Acked Read 6 Done Done Sent false
22991 6405 Acked Read 10 Done Done Sent false
22992 6406 Acked Write 7 Done Done Sent false
22993 6407 Acked Write 8 Done Done Sent false
22994 6408 Acked Write 2 Done Done Sent false
22995 6409 Acked Write 10 Done Done Sent false
22996 6410 Acked Write 6 Done Done Sent false
22997 6411 Acked Read 4 Done Done Sent false
22998 6412 Acked Read 6 Done Done Sent false
22999 6413 Acked Read 4 Done Done Sent false
23000 6414 Acked Flush 0 Done Done Sent false
23001 6415 Acked Write 2 Done Done Sent false
23002 6416 Acked Write 3 Done Done Sent false
23003 6417 Acked Read 3 Done Done Sent false
23004 6418 Acked Write 3 Done Done Sent false
23005 6419 Acked Write 6 Done Done Sent false
23006 6420 Acked Read 4 Done Done Sent false
23007 6421 Acked Write 5 Done Done Sent false
23008 6422 Acked Read 6 Done Done Sent false
23009 6423 Acked Read 9 Done Done Sent false
23010 6424 Acked Read 7 Done Done Sent false
23011 6425 Acked Write 1 Done Done Sent false
23012 6426 Acked Read 8 Done Done Sent false
23013 6427 Acked Read 10 Done Done Sent false
23014 6428 Acked Flush 0 Done Done Sent false
23015 6429 Acked Flush 0 Done Done Sent false
23016 6430 Acked Read 8 Done Done Sent false
23017 6431 Acked Read 7 Done Done Sent false
23018 6432 Acked Write 1 Done Done Sent false
23019 STATES DS:0 DS:1 DS:2 TOTAL
23020 Sent 0 0 301 301
23021 Done 301 301 0 602
23022 Skipped 0 0 0 0
23023 Error 0 0 0 0
23024 Last Flush: Some(JobId(6429)) Some(JobId(6429)) Some(JobId(6131))
23025 Downstairs last five completed: 6131 6130 6129 6128 6127
23026 Upstairs last five completed: 6432 6431 6430 6429 6428
23027 CLIENT: Up:0 ds:301 act:2
230282025-03-17T22:24:03.161ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
230292025-03-17T22:24:04.063ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
230302025-03-17T22:24:04.409ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
230312025-03-17T22:24:04.410ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:48727) = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
230322025-03-17T22:24:04.411ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
230332025-03-17T22:24:04.411ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
230342025-03-17T22:24:04.411ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
230352025-03-17T22:24:04.411ZINFOcrucible: send last flush ID to this DS: Some(JobId(6131)) = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
230362025-03-17T22:24:04.411ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
230372025-03-17T22:24:04.411ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(6131)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
230382025-03-17T22:24:04.412ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(6131)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
23039 ----------------------------------------------------------------
23040 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
23041 Downstairs last five completed: 6433 6432 6431 6430 6429
23042 Upstairs last five completed: 6433 6432 6431 6430 6429
23043 CLIENT: Up:0 ds:0 act:3
23044 Replay: All jobs finished
23045 Main task has work to do, go find it
23046 got dsc Stop(2)
23047 stop 2
23048 Mar 17 22:24:06.662 INFO request completed, latency_us: 330, response_code: [8830] Got stop action so:false kr:false
23049 204, uri: /stop/cid/2, method: POST, req_id: 770581c1-8c86-4542-bed2-a1c325e17344, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
23050 Mar 17 22:24:06.663 INFO request completed, latency_us: 141, response_code: 200, uri: /state/cid/2, method: GET, req_id: 730a1585-b653-4dee-ba7c-33329cc38824, remote_addr: 127.0.0.1:58317, local_addr: 127.{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:24:06.663589839Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
23051 0.0.1:9998
230522025-03-17T22:24:06.663ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
230532025-03-17T22:24:06.663ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
230542025-03-17T22:24:06.663ZINFOcrucible: client task is exiting = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
23055 [8830] Exited with: ExitStatus(unix_wait_status(9))
23056 [8830][2] reports Exit
23057 Mar 17 22:24:10.665 INFO request completed, latency_us: 213, response_code: 200, uri: /state/cid/2, method: GET, req_id: 7cbf4b9d-6c91-41be-a3cc-d6c6b01f506a, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
23058 001/300 Read block 4878 len 40960
23059 002/300 Flush
23060 003/300 Write block 637 len 28672 data: 3 3 3 3 3 4 3
23061 004/300 Flush
23062 005/300 Read block 4639 len 36864
23063 006/300 Flush
23064 007/300 Flush
23065 008/300 Read block 1183 len 20480
23066 009/300 Write block 308 len 8192 data: 2 3
23067 010/300 Flush
23068 011/300 Read block 1338 len 40960
23069 012/300 Write block 670 len 40960 data: 3 2 2 2 3 3 4 4 4 3
23070 013/300 Write block 1960 len 28672 data: 4 4 6 5 5 6 7
23071 014/300 Write block 3944 len 40960 data: 6 5 5 5 4 5 5 5 5 5
23072 015/300 Write block 4049 len 40960 data: 3 3 3 3 3 4 3 3 3 4
23073 016/300 Read block 180 len 36864
23074 017/300 Write block 1499 len 36864 data: 4 5 4 3 3 3 3 3 3
23075 018/300 Write block 1281 len 4096 data: 6
23076 019/300 Read block 4953 len 24576
23077 020/300 Write block 1781 len 28672 data: 3 5 5 5 4 3 2
23078 021/300 Write block 3340 len 32768 data: 3 3 3 4 5 7 6 7
23079 022/300 Flush
23080 023/300 Flush
23081 024/300 Write block 2635 len 12288 data: 2 3 5
23082 025/300 Write block 4093 len 20480 data: 4 4 4 4 2
23083 026/300 Read block 3480 len 12288
23084 027/300 Write block 2024 len 20480 data: 5 4 4 4 4
23085 028/300 Write block 2172 len 24576 data: 5 5 4 5 4 4
23086 029/300 Read block 2405 len 12288
23087 030/300 Flush
23088 031/300 Read block 3329 len 32768
23089 032/300 Read block 2582 len 12288
23090 033/300 Write block 1306 len 8192 data: 3 4
23091 034/300 Write block 3191 len 16384 data: 3 4 5 4
23092 035/300 Read block 290 len 24576
23093 036/300 Read block 307 len 24576
23094 037/300 Write block 3217 len 12288 data: 2 2 2
23095 038/300 Write block 1426 len 4096 data: 5
23096 039/300 Read block 4494 len 40960
23097 040/300 Flush
23098 041/300 Write block 3508 len 12288 data: 5 4 4
23099 042/300 Read block 1025 len 36864
23100 043/300 Write block 3920 len 36864 data: 3 3 3 3 5 6 6 5 6
23101 044/300 Write block 4269 len 28672 data: 5 3 3 4 5 4 4
23102 045/300 Flush
23103 046/300 Read block 148 len 32768
23104 047/300 Write block 4859 len 36864 data: 5 6 6 6 5 5 5 6 5
23105 048/300 Read block 2856 len 40960
23106 049/300 Write block 3546 len 24576 data: 7 6 5 6 6 6
23107 050/300 Read block 3872 len 28672
23108 051/300 Write block 2274 len 32768 data: 3 4 4 5 5 5 5 6
23109 052/300 Write block 2408 len 40960 data: 4 3 4 6 6 7 7 5 5 7
23110 053/300 Write block 1979 len 8192 data: 5 6
23111 054/300 Flush
23112 055/300 Write block 4972 len 32768 data: 3 3 3 3 3 3 3 4
23113 056/300 Write block 4491 len 40960 data: 3 3 3 5 5 5 6 5 4 5
23114 057/300 Read block 1770 len 12288
23115 058/300 Read block 1568 len 16384
23116 059/300 Read block 4327 len 20480
23117 060/300 Write block 4074 len 32768 data: 5 7 6 6 5 5 3 3
23118 061/300 Read block 4534 len 12288
23119 062/300 Write block 2195 len 36864 data: 4 4 5 4 2 2 2 3 5
23120 063/300 Write block 3038 len 12288 data: 5 6 7
23121 064/300 Write block 4868 len 8192 data: 6 6
23122 065/300 Flush
23123 066/300 Read block 3809 len 24576
23124 067/300 Read block 4228 len 32768
23125 068/300 Write block 3519 len 8192 data: 7 7
23126 069/300 Flush
23127 070/300 Read block 3438 len 20480
23128 071/300 Read block 3163 len 32768
23129 072/300 Read block 4416 len 32768
23130 073/300 Read block 1621 len 32768
23131 074/300 Flush
23132 075/300 Flush
23133 076/300 Write block 2772 len 40960 data: 3 3 3 3 2 2 2 3 3 3
23134 077/300 Write block 4101 len 8192 data: 4 3
23135 078/300 Flush
23136 079/300 Write block 2241 len 40960 data: 6 6 6 5 5 5 5 6 7 5
23137 080/300 Write block 976 len 12288 data: 6 5 4
23138 081/300 Read block 184 len 24576
23139 082/300 Read block 2455 len 16384
23140 083/300 Read block 269 len 20480
23141 084/300 Write block 1259 len 8192 data: 3 3
23142 085/300 Read block 389 len 32768
23143 086/300 Read block 426 len 24576
23144 087/300 Read block 1548 len 36864
23145 088/300 Flush
23146 089/300 Write block 3342 len 8192 data: 4 5
23147 090/300 Flush
23148 091/300 Read block 3890 len 40960
23149 092/300 Read block 435 len 8192
23150 093/300 Read block 1849 len 24576
23151 094/300 Read block 3484 len 12288
23152 095/300 Read block 464 len 24576
23153 096/300 Read block 4819 len 20480
23154 097/300 Flush
23155 098/300 Flush
23156 099/300 Write block 1680 len 16384 data: 2 3 3 3
23157 100/300 Flush
23158 101/300 Write block 3070 len 28672 data: 3 3 3 3 3 3 4
23159 102/300 Read block 3194 len 20480
23160 103/300 Read block 1033 len 24576
23161 104/300 Flush
23162 105/300 Read block 3761 len 36864
23163 106/300 Read block 2047 len 20480
23164 107/300 Read block 1697 len 32768
23165 108/300 Write block 1868 len 16384 data: 6 7 7 8
23166 109/300 Flush
23167 110/300 Read block 2381 len 16384
23168 111/300 Read block 4333 len 40960
23169 112/300 Read block 1295 len 40960
23170 113/300 Read block 200 len 28672
23171 114/300 Flush
23172 115/300 Flush
23173 116/300 Read block 1542 len 40960
23174 117/300 Read block 2389 len 12288
23175 118/300 Read block 1416 len 4096
23176 119/300 Write block 656 len 28672 data: 4 5 5 6 7 7 7
23177 120/300 Read block 861 len 36864
23178 121/300 Write block 1012 len 12288 data: 4 4 4
23179 122/300 Read block 3374 len 32768
23180 123/300 Read block 1739 len 36864
23181 124/300 Write block 2313 len 16384 data: 9 10 8 5
23182 125/300 Read block 2250 len 16384
23183 126/300 Write block 4339 len 20480 data: 3 3 4 3 4
23184 127/300 Read block 1371 len 12288
23185 128/300 Write block 2561 len 8192 data: 5 4
23186 129/300 Read block 4966 len 40960
23187 130/300 Read block 2593 len 4096
23188 131/300 Read block 1267 len 16384
23189 132/300 Read block 40 len 8192
23190 133/300 Read block 3621 len 20480
23191 134/300 Read block 2750 len 12288
23192 135/300 Write block 3438 len 32768 data: 6 5 6 5 3 4 5 3
23193 136/300 Write block 3510 len 16384 data: 5 4 3 4
23194 137/300 Write block 1751 len 20480 data: 5 5 4 5 5
23195 138/300 Write block 4252 len 12288 data: 6 4 6
23196 139/300 Read block 1577 len 28672
23197 140/300 Read block 4993 len 28672
23198 141/300 Read block 183 len 32768
23199 142/300 Write block 4637 len 24576 data: 3 3 3 4 6 5
23200 143/300 Read block 121 len 36864
23201 144/300 Read block 2763 len 28672
23202 145/300 Write block 4755 len 36864 data: 5 6 6 4 2 4 5 6 5
23203 146/300 Flush
23204 147/300 Write block 4279 len 32768 data: 4 4 5 4 4 5 4 4
23205 148/300 Read block 3657 len 40960
23206 149/300 Write block 2616 len 36864 data: 3 2 2 3 4 4 5 5 5
23207 150/300 Read block 3910 len 36864
23208 151/300 Write block 4291 len 16384 data: 4 3 3 3
23209 152/300 Write block 4864 len 40960 data: 6 6 7 6 7 7 5 5 4 3
23210 153/300 Read block 3994 len 12288
23211 154/300 Read block 4441 len 4096
23212 155/300 Write block 1851 len 8192 data: 3 3
23213 156/300 Flush
23214 157/300 Write block 1017 len 8192 data: 4 4
23215 158/300 Write block 3072 len 4096 data: 4
23216 159/300 Flush
23217 160/300 Write block 220 len 36864 data: 3 3 3 3 4 4 4 4 3
23218 161/300 Read block 3305 len 32768
23219 162/300 Read block 2562 len 12288
23220 163/300 Read block 1568 len 40960
23221 164/300 Read block 992 len 28672
23222 165/300 Write block 1827 len 20480 data: 4 4 3 3 4
23223 166/300 Write block 3242 len 36864 data: 4 3 4 4 4 5 7 6 6
23224 167/300 Write block 781 len 8192 data: 7 8
23225 168/300 Read block 4375 len 16384
23226 169/300 Read block 3527 len 12288
23227 170/300 Read block 2504 len 20480
23228 171/300 Write block 41 len 20480 data: 5 5 5 6 8
23229 172/300 Write block 2843 len 36864 data: 4 5 5 4 4 3 3 4 4
23230 173/300 Read block 3932 len 8192
23231 174/300 Read block 3142 len 20480
23232 175/300 Write block 4286 len 40960 data: 5 3 4 4 4 5 4 4 4 3
23233 176/300 Read block 131 len 36864
23234 177/300 Flush
23235 178/300 Write block 1669 len 20480 data: 6 7 6 4 5
23236 179/300 Read block 4563 len 36864
23237 180/300 Read block 1424 len 8192
23238 181/300 Read block 242 len 12288
23239 182/300 Read block 644 len 4096
23240 183/300 Write block 3082 len 12288 data: 8 5 4
23241 184/300 Write block 3369 len 20480 data: 2 4 4 5 5
23242 185/300 Read block 1914 len 4096
23243 186/300 Flush
23244 187/300 Flush
23245 188/300 Read block 4008 len 12288
23246 189/300 Read block 3631 len 40960
23247 190/300 Write block 3041 len 32768 data: 7 8 7 7 7 5 6 6
23248 191/300 Read block 1155 len 24576
23249 192/300 Flush
23250 193/300 Read block 179 len 16384
23251 194/300 Write block 24 len 40960 data: 3 4 4 4 4 5 6 5 5 4
23252 195/300 Write block 4648 len 4096 data: 7
23253 196/300 Flush
23254 197/300 Write block 4659 len 4096 data: 4
23255 198/300 Read block 1140 len 4096
23256 199/300 Write block 2163 len 4096 data: 4
23257 200/300 Write block 1340 len 40960 data: 2 2 4 3 4 3 3 5 5 4
23258 201/300 Read block 4876 len 8192
23259 202/300 Read block 2023 len 16384
23260 203/300 Write block 4828 len 8192 data: 2 3
23261 204/300 Read block 1848 len 32768
23262 205/300 Read block 494 len 40960
23263 206/300 Read block 2349 len 28672
23264 207/300 Read block 1819 len 24576
23265 208/300 Write block 1656 len 4096 data: 6
23266 209/300 Read block 2687 len 28672
23267 210/300 Read block 659 len 12288
23268 211/300 Read block 2935 len 24576
23269 212/300 Write block 4428 len 20480 data: 4 4 5 5 5
23270 213/300 Write block 4543 len 36864 data: 5 6 5 4 5 6 6 7 8
23271 214/300 Write block 216 len 32768 data: 2 3 2 2 4 4 4 4
23272 215/300 Read block 2014 len 28672
23273 216/300 Write block 759 len 36864 data: 6 6 6 6 5 6 5 4 4
23274 217/300 Write block 124 len 16384 data: 6 4 5 5
23275 218/300 Write block 651 len 24576 data: 5 3 2 2 3 5
23276 219/300 Write block 3250 len 32768 data: 7 6 4 4 4 3 4 3
23277 220/300 Write block 1126 len 32768 data: 6 6 6 4 4 5 6 6
23278 221/300 Flush
23279 222/300 Read block 2721 len 40960
23280 223/300 Write block 4962 len 20480 data: 6 6 4 5 5
23281 224/300 Read block 701 len 36864
23282 225/300 Write block 886 len 36864 data: 3 4 6 6 6 5 5 4 4
23283 226/300 Read block 687 len 16384
23284 227/300 Write block 2687 len 16384 data: 5 6 6 4
23285 228/300 Read block 1171 len 16384
23286 229/300 Read block 53 len 40960
23287 230/300 Write block 549 len 20480 data: 2 2 3 2 4
23288 231/300 Write block 4636 len 24576 data: 3 4 4 4 5 7
23289 232/300 Write block 2222 len 28672 data: 4 4 5 5 5 6 7
23290 233/300 Write block 388 len 32768 data: 4 5 5 5 5 5 5 5
23291 234/300 Read block 1739 len 32768
23292 235/300 Write block 2389 len 28672 data: 3 5 4 5 5 7 5
23293 236/300 Read block 4908 len 32768
23294 237/300 Write block 32 len 4096 data: 6
23295 238/300 Write block 4776 len 40960 data: 4 4 4 4 4 4 5 5 4 4
23296 239/300 Read block 3577 len 8192
23297 240/300 Write block 3618 len 16384 data: 3 3 3 3
23298 241/300 Read block 3729 len 40960
23299 242/300 Read block 1286 len 16384
23300 243/300 Write block 508 len 32768 data: 4 5 4 4 4 3 3 2
23301 244/300 Read block 3227 len 20480
23302 245/300 Write block 3721 len 12288 data: 4 4 6
23303 246/300 Read block 435 len 36864
23304 247/300 Write block 2530 len 32768 data: 6 7 6 5 4 4 4 4
23305 248/300 Read block 3847 len 12288
23306 249/300 Write block 1034 len 24576 data: 4 3 3 3 4 4
23307 250/300 Write block 4175 len 28672 data: 4 6 6 7 7 6 7
23308 251/300 Write block 2522 len 36864 data: 6 7 6 4 4 6 6 6 7
23309 252/300 Read block 948 len 16384
23310 253/300 Read block 3708 len 28672
23311 254/300 Read block 2625 len 4096
23312 255/300 Read block 1641 len 8192
23313 256/300 Read block 3749 len 40960
23314 257/300 Read block 4367 len 24576
23315 258/300 Read block 1323 len 20480
23316 259/300 Write block 4214 len 40960 data: 6 5 6 6 7 5 6 7 7 6
23317 260/300 Write block 2503 len 4096 data: 5
23318 261/300 Write block 3746 len 28672 data: 5 5 4 4 4 4 6
23319 262/300 Read block 549 len 20480
23320 263/300 Write block 1062 len 24576 data: 2 2 2 3 3 4
23321 264/300 Flush
23322 265/300 Read block 3260 len 36864
23323 266/300 Read block 1041 len 36864
23324 267/300 Flush
23325 268/300 Read block 1715 len 40960
23326 269/300 Read block 3552 len 32768
23327 270/300 Write block 1801 len 28672 data: 4 5 5 5 3 4 4
23328 271/300 Read block 4082 len 36864
23329 272/300 Write block 1378 len 40960 data: 3 3 3 3 3 3 3 3 3 3
23330 273/300 Write block 1094 len 4096 data: 3
23331 274/300 Read block 1882 len 20480
23332 275/300 Read block 2144 len 20480
23333 276/300 Write block 630 len 36864 data: 2 2 2 2 3 3 3 4 4
23334 277/300 Write block 1207 len 16384 data: 2 2 3 4
23335 278/300 Read block 4000 len 8192
23336 279/300 Read block 1778 len 8192
23337 280/300 Read block 3476 len 12288
23338 281/300 Write block 505 len 4096 data: 4
23339 282/300 Read block 2805 len 40960
23340 283/300 Read block 4752 len 12288
23341 284/300 Write block 3730 len 36864 data: 7 7 8 7 6 5 5 4 4
23342 285/300 Read block 4831 len 28672
23343 286/300 Read block 4651 len 24576
23344 287/300 Write block 3688 len 36864 data: 4 4 4 4 5 6 6 6 6
23345 288/300 Write block 2111 len 36864 data: 6 5 4 5 3 4 3 4 4
23346 289/300 Write block 2100 len 20480 data: 4 4 4 4 3
23347 290/300 Read block 3388 len 12288
23348 291/300 Write block 1667 len 28672 data: 6 5 7 8 7 5 6
23349 292/300 Flush
23350 293/300 Write block 1643 len 16384 data: 4 2 2 3
23351 294/300 Read block 3983 len 36864
23352 295/300 Read block 3628 len 12288
23353 296/300 Flush
23354 297/300 Read block 2611 len 20480
23355 298/300 Write block 4122 len 24576 data: 4 5 3 2 2 2
23356 299/300 Write block 4026 len 32768 data: 3 3 4 3 3 4 4 4
23357 300/300 Read block 2378 len 16384
23358 301/300 Read block 2974 len 28672
23359 Main task has work to do, go find it
23360 got dsc Start(2)
23361 start 2
23362 Mar 17 22:24:10.914 INFO request completed, latency_us: 188, response_code: 204, uri: /start/cid/2, method: POST, req_id: f13b4045-aeef-45b8-9c38-7522bb236e44, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
23363 Replay: started 2, returned:Ok(())
23364 ----------------------------------------------------------------
23365 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23366 States: Active Active Offline
23367 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23368 6434 Acked Read 10 Done Done Sent false
23369 6435 Acked Flush 0 Done Done Sent false
23370 6436 Acked Write 7 Done Done Sent false
23371 6437 Acked Flush 0 Done Done Sent false
23372 6438 Acked Read 9 Done Done Sent false
23373 6439 Acked Flush 0 Done Done Sent false
23374 6440 Acked Flush 0 Done Done Sent false
23375 6441 Acked Read 5 Done Done Sent false
23376 6442 Acked Write 2 Done Done Sent false
23377 6443 Acked Flush 0 Done Done Sent false
23378 6444 Acked Read 10 Done Done Sent false
23379 6445 Acked Write 10 Done Done Sent false
23380 6446 Acked Write 7 Done Done Sent false
23381 6447 Acked Write 10 Done Done Sent false
23382 6448 Acked Write 10 Done Done Sent false
23383 6449 Acked Read 9 Done Done Sent false
23384 6450 Acked Write 9 Done Done Sent false
23385 6451 Acked Write 1 Done Done Sent false
23386 6452 Acked Read 6 Done Done Sent false
23387 6453 Acked Write 7 Done Done Sent false
23388 6454 Acked Write 8 Done Done Sent false
23389 6455 Acked Flush 0 Done Done Sent false
23390 6456 Acked Flush 0 Done Done Sent false
23391 6457 Acked Write 3 Done Done Sent false
23392 6458 Acked Write 5 Done Done Sent false
23393 6459 Acked Read 3 Done Done Sent false
23394 6460 Acked Write 5 Done Done Sent false
23395 6461 Acked Write 6 Done Done Sent false
23396 6462 Acked Read 3 Done Done Sent false
23397 6463 Acked Flush 0 Done Done Sent false
23398 6464 Acked Read 8 Done Done Sent false
23399 6465 Acked Read 3 Done Done Sent false
23400 6466 Acked Write 2 Done Done Sent false
23401 6467 Acked Write 4 Done Done Sent false
23402 6468 Acked Read 6 Done Done Sent false
23403 6469 Acked Read 6 Done Done Sent false
23404 6470 Acked Write 3 Done Done Sent false
23405 6471 Acked Write 1 Done Done Sent false
23406 6472 Acked Read 10 Done Done Sent false
23407 6473 Acked Flush 0 Done Done Sent false
23408 6474 Acked Write 3 Done Done Sent false
23409 6475 Acked Read 9 Done Done Sent false
23410 6476 Acked Write 9 Done Done Sent false
23411 6477 Acked Write 7 Done Done Sent false
23412 6478 Acked Flush 0 Done Done Sent false
23413 6479 Acked Read 8 Done Done Sent false
23414 6480 Acked Write 9 Done Done Sent false
23415 6481 Acked Read 10 Done Done Sent false
23416 6482 Acked Write 6 Done Done Sent false
23417 6483 Acked Read 7 Done Done Sent false
23418 6484 Acked Write 8 Done Done Sent false
23419 6485 Acked Write 10 Done Done Sent false
23420 6486 Acked Write 2 Done Done Sent false
23421 6487 Acked Flush 0 Done Done Sent false
23422 6488 Acked Write 8 Done Done Sent false
23423 6489 Acked Write 10 Done Done Sent false
23424 6490 Acked Read 3 Done Done Sent false
23425 6491 Acked Read 4 Done Done Sent false
23426 6492 Acked Read 5 Done Done Sent false
23427 6493 Acked Write 8 Done Done Sent false
23428 6494 Acked Read 3 Done Done Sent false
23429 6495 Acked Write 9 Done Done Sent false
23430 6496 Acked Write 3 Done Done Sent false
23431 6497 Acked Write 2 Done Done Sent false
23432 6498 Acked Flush 0 Done Done Sent false
23433 6499 Acked Read 6 Done Done Sent false
23434 6500 Acked Read 8 Done Done Sent false
23435 6501 Acked Write 2 Done Done Sent false
23436 6502 Acked Flush 0 Done Done Sent false
23437 6503 Acked Read 5 Done Done Sent false
23438 6504 Acked Read 8 Done Done Sent false
23439 6505 Acked Read 8 Done Done Sent false
23440 6506 Acked Read 8 Done Done Sent false
23441 6507 Acked Flush 0 Done Done Sent false
23442 6508 Acked Flush 0 Done Done Sent false
23443 6509 Acked Write 10 Done Done Sent false
23444 6510 Acked Write 2 Done Done Sent false
23445 6511 Acked Flush 0 Done Done Sent false
23446 6512 Acked Write 10 Done Done Sent false
23447 6513 Acked Write 3 Done Done Sent false
23448 6514 Acked Read 6 Done Done Sent false
23449 6515 Acked Read 4 Done Done Sent false
23450 6516 Acked Read 5 Done Done Sent false
23451 6517 Acked Write 2 Done Done Sent false
23452 6518 Acked Read 8 Done Done Sent false
23453 6519 Acked Read 6 Done Done Sent false
23454 6520 Acked Read 9 Done Done Sent false
23455 6521 Acked Flush 0 Done Done Sent false
23456 6522 Acked Write 2 Done Done Sent false
23457 6523 Acked Flush 0 Done Done Sent false
23458 6524 Acked Read 10 Done Done Sent false
23459 6525 Acked Read 2 Done Done Sent false
23460 6526 Acked Read 6 Done Done Sent false
23461 6527 Acked Read 3 Done Done Sent false
23462 6528 Acked Read 6 Done Done Sent false
23463 6529 Acked Read 5 Done Done Sent false
23464 6530 Acked Flush 0 Done Done Sent false
23465 6531 Acked Flush 0 Done Done Sent false
23466 6532 Acked Write 4 Done Done Sent false
23467 6533 Acked Flush 0 Done Done Sent false
23468 6534 Acked Write 7 Done Done Sent false
23469 6535 Acked Read 5 Done Done Sent false
23470 6536 Acked Read 6 Done Done Sent false
23471 6537 Acked Flush 0 Done Done Sent false
23472 6538 Acked Read 9 Done Done Sent false
23473 6539 Acked Read 5 Done Done Sent false
23474 6540 Acked Read 8 Done Done Sent false
23475 6541 Acked Write 4 Done Done Sent false
23476 6542 Acked Flush 0 Done Done Sent false
23477 6543 Acked Read 4 Done Done Sent false
23478 6544 Acked Read 10 Done Done Sent false
23479 6545 Acked Read 10 Done Done Sent false
23480 6546 Acked Read 7 Done Done Sent false
23481 6547 Acked Flush 0 Done Done Sent false
23482 6548 Acked Flush 0 Done Done Sent false
23483 6549 Acked Read 10 Done Done Sent false
23484 6550 Acked Read 3 Done Done Sent false
23485 6551 Acked Read 1 Done Done Sent false
23486 6552 Acked Write 7 Done Done Sent false
23487 6553 Acked Read 9 Done Done Sent false
23488 6554 Acked Write 3 Done Done Sent false
23489 6555 Acked Read 8 Done Done Sent false
23490 6556 Acked Read 9 Done Done Sent false
23491 6557 Acked Write 4 Done Done Sent false
23492 6558 Acked Read 4 Done Done Sent false
23493 6559 Acked Write 5 Done Done Sent false
23494 6560 Acked Read 3 Done Done Sent false
23495 6561 Acked Write 2 Done Done Sent false
23496 6562 Acked Read 10 Done Done Sent false
23497 6563 Acked Read 1 Done Done Sent false
23498 6564 Acked Read 4 Done Done Sent false
23499 6565 Acked Read 2 Done Done Sent false
23500 6566 Acked Read 5 Done Done Sent false
23501 6567 Acked Read 3 Done Done Sent false
23502 6568 Acked Write 8 Done Done Sent false
23503 6569 Acked Write 4 Done Done Sent false
23504 6570 Acked Write 5 Done Done Sent false
23505 6571 Acked Write 3 Done Done Sent false
23506 6572 Acked Read 7 Done Done Sent false
23507 6573 Acked Read 7 Done Done Sent false
23508 6574 Acked Read 8 Done Done Sent false
23509 6575 Acked Write 6 Done Done Sent false
23510 6576 Acked Read 9 Done Done Sent false
23511 6577 Acked Read 7 Done Done Sent false
23512 6578 Acked Write 9 Done Done Sent false
23513 6579 Acked Flush 0 Done Done Sent false
23514 6580 Acked Write 8 Done Done Sent false
23515 6581 Acked Read 10 Done Done Sent false
23516 6582 Acked Write 9 Done Done Sent false
23517 6583 Acked Read 9 Done Done Sent false
23518 6584 Acked Write 4 Done Done Sent false
23519 6585 Acked Write 10 Done Done Sent false
23520 6586 Acked Read 3 Done Done Sent false
23521 6587 Acked Read 1 Done Done Sent false
23522 6588 Acked Write 2 Done Done Sent false
23523 6589 Acked Flush 0 Done Done Sent false
23524 6590 Acked Write 2 Done Done Sent false
23525 6591 Acked Write 1 Done Done Sent false
23526 6592 Acked Flush 0 Done Done Sent false
23527 6593 Acked Write 9 Done Done Sent false
23528 6594 Acked Read 8 Done Done Sent false
23529 6595 Acked Read 3 Done Done Sent false
23530 6596 Acked Read 10 Done Done Sent false
23531 6597 Acked Read 7 Done Done Sent false
23532 6598 Acked Write 5 Done Done Sent false
23533 6599 Acked Write 9 Done Done Sent false
23534 6600 Acked Write 2 Done Done Sent false
23535 6601 Acked Read 4 Done Done Sent false
23536 6602 Acked Read 3 Done Done Sent false
23537 6603 Acked Read 5 Done Done Sent false
23538 6604 Acked Write 5 Done Done Sent false
23539 6605 Acked Write 9 Done Done Sent false
23540 6606 Acked Read 2 Done Done Sent false
23541 6607 Acked Read 5 Done Done Sent false
23542 6608 Acked Write 10 Done Done Sent false
23543 6609 Acked Read 9 Done Done Sent false
23544 6610 Acked Flush 0 Done Done Sent false
23545 6611 Acked Write 5 Done Done Sent false
23546 6612 Acked Read 9 Done Done Sent false
23547 6613 Acked Read 2 Done Done Sent false
23548 6614 Acked Read 3 Done Done Sent false
23549 6615 Acked Read 1 Done Done Sent false
23550 6616 Acked Write 3 Done Done Sent false
23551 6617 Acked Write 5 Done Done Sent false
23552 6618 Acked Read 1 Done Done Sent false
23553 6619 Acked Flush 0 Done Done Sent false
23554 6620 Acked Flush 0 Done Done Sent false
23555 6621 Acked Read 3 Done Done Sent false
23556 6622 Acked Read 10 Done Done Sent false
23557 6623 Acked Write 8 Done Done Sent false
23558 6624 Acked Read 6 Done Done Sent false
23559 6625 Acked Flush 0 Done Done Sent false
23560 6626 Acked Read 4 Done Done Sent false
23561 6627 Acked Write 10 Done Done Sent false
23562 6628 Acked Write 1 Done Done Sent false
23563 6629 Acked Flush 0 Done Done Sent false
23564 6630 Acked Write 1 Done Done Sent false
23565 6631 Acked Read 1 Done Done Sent false
23566 6632 Acked Write 1 Done Done Sent false
23567 6633 Acked Write 10 Done Done Sent false
23568 6634 Acked Read 2 Done Done Sent false
23569 6635 Acked Read 4 Done Done Sent false
23570 6636 Acked Write 2 Done Done Sent false
23571 6637 Acked Read 8 Done Done Sent false
23572 6638 Acked Read 10 Done Done Sent false
23573 6639 Acked Read 7 Done Done Sent false
23574 6640 Acked Read 6 Done Done Sent false
23575 6641 Acked Write 1 Done Done Sent false
23576 6642 Acked Read 7 Done Done Sent false
23577 6643 Acked Read 3 Done Done Sent false
23578 6644 Acked Read 6 Done Done Sent false
23579 6645 Acked Write 5 Done Done Sent false
23580 6646 Acked Write 9 Done Done Sent false
23581 6647 Acked Write 8 Done Done Sent false
23582 6648 Acked Read 7 Done Done Sent false
23583 6649 Acked Write 9 Done Done Sent false
23584 6650 Acked Write 4 Done Done Sent false
23585 6651 Acked Write 6 Done Done Sent false
23586 6652 Acked Write 8 Done Done Sent false
23587 6653 Acked Write 8 Done Done Sent false
23588 6654 Acked Flush 0 Done Done Sent false
23589 6655 Acked Read 10 Done Done Sent false
23590 6656 Acked Write 5 Done Done Sent false
23591 6657 Acked Read 9 Done Done Sent false
23592 6658 Acked Write 9 Done Done Sent false
23593 6659 Acked Read 4 Done Done Sent false
23594 6660 Acked Write 4 Done Done Sent false
23595 6661 Acked Read 4 Done Done Sent false
23596 6662 Acked Read 10 Done Done Sent false
23597 6663 Acked Write 5 Done Done Sent false
23598 6664 Acked Write 6 Done Done Sent false
23599 6665 Acked Write 7 Done Done Sent false
23600 6666 Acked Write 8 Done Done Sent false
23601 6667 Acked Read 8 Done Done Sent false
23602 6668 Acked Write 7 Done Done Sent false
23603 6669 Acked Read 8 Done Done Sent false
23604 6670 Acked Write 1 Done Done Sent false
23605 6671 Acked Write 10 Done Done Sent false
23606 6672 Acked Read 2 Done Done Sent false
23607 6673 Acked Write 4 Done Done Sent false
23608 6674 Acked Read 10 Done Done Sent false
23609 6675 Acked Read 4 Done Done Sent false
23610 6676 Acked Write 8 Done Done Sent false
23611 6677 Acked Read 5 Done Done Sent false
23612 6678 Acked Write 3 Done Done Sent false
23613 6679 Acked Read 9 Done Done Sent false
23614 6680 Acked Write 8 Done Done Sent false
23615 6681 Acked Read 3 Done Done Sent false
23616 6682 Acked Write 6 Done Done Sent false
23617 6683 Acked Write 7 Done Done Sent false
23618 6684 Acked Write 9 Done Done Sent false
23619 6685 Acked Read 4 Done Done Sent false
23620 6686 Acked Read 7 Done Done Sent false
23621 6687 Acked Read 1 Done Done Sent false
23622 6688 Acked Read 2 Done Done Sent false
23623 6689 Acked Read 10 Done Done Sent false
23624 6690 Acked Read 6 Done Done Sent false
23625 6691 Acked Read 5 Done Done Sent false
23626 6692 Acked Write 10 Done Done Sent false
23627 6693 Acked Write 1 Done Done Sent false
23628 6694 Acked Write 7 Done Done Sent false
23629 6695 Acked Read 5 Done Done Sent false
23630 6696 Acked Write 6 Done Done Sent false
23631 6697 Acked Flush 0 Done Done Sent false
23632 6698 Acked Read 9 Done Done Sent false
23633 6699 Acked Read 9 Done Done Sent false
23634 6700 Acked Flush 0 Done Done Sent false
23635 6701 Acked Read 10 Done Done Sent false
23636 6702 Acked Read 8 Done Done Sent false
23637 6703 Acked Write 7 Done Done Sent false
23638 6704 Acked Read 9 Done Done Sent false
23639 6705 Acked Write 10 Done Done Sent false
23640 6706 Acked Write 1 Done Done Sent false
23641 6707 Acked Read 5 Done Done Sent false
23642 6708 Acked Read 5 Done Done Sent false
23643 6709 Acked Write 9 Done Done Sent false
23644 6710 Acked Write 4 Done Done Sent false
23645 6711 Acked Read 2 Done Done Sent false
23646 6712 Acked Read 2 Done Done Sent false
23647 6713 Acked Read 3 Done Done Sent false
23648 6714 Acked Write 1 Done Done Sent false
23649 6715 Acked Read 10 Done Done Sent false
23650 6716 Acked Read 3 Done Done Sent false
23651 6717 Acked Write 9 Done Done Sent false
23652 6718 Acked Read 7 Done Done Sent false
23653 6719 Acked Read 6 Done Done Sent false
23654 6720 Acked Write 9 Done Done Sent false
23655 6721 Acked Write 9 Done Done Sent false
23656 6722 Acked Write 5 Done Done Sent false
23657 6723 Acked Read 3 Done Done Sent false
23658 6724 Acked Write 7 Done Done Sent false
23659 6725 Acked Flush 0 Done Done Sent false
23660 6726 Acked Write 4 Done Done Sent false
23661 6727 Acked Read 9 Done Done Sent false
23662 6728 Acked Read 3 Done Done Sent false
23663 6729 Acked Flush 0 Done Done Sent false
23664 6730 Acked Read 5 Done Done Sent false
23665 6731 Acked Write 6 Done Done Sent false
23666 6732 Acked Write 8 Done Done Sent false
23667 6733 Acked Read 4 Done Done Sent false
23668 6734 Acked Read 7 Done Done Sent false
23669 STATES DS:0 DS:1 DS:2 TOTAL
23670 Sent 0 0 301 301
23671 Done 301 301 0 602
23672 Skipped 0 0 0 0
23673 Error 0 0 0 0
23674 Last Flush: Some(JobId(6729)) Some(JobId(6729)) Some(JobId(6433))
23675 Downstairs last five completed: 6433 6432 6431 6430 6429
23676 Upstairs last five completed: 6734 6733 6732 6731 6730
23677 CLIENT: Up:0 ds:301 act:2
236782025-03-17T22:24:11.416ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
236792025-03-17T22:24:12.418ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
23680 [8830] got start action
23681 [8830] I am going to restart
23682 Starting downstairs at port 8830
23683 Make output file at "/tmp/dsc/downstairs-8830.txt"
23684 [8830][2] reports Starting
23685 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1230)
23686 [8830][2] reports Running
236872025-03-17T22:24:13.078ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
236882025-03-17T22:24:14.079ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
23689 ----------------------------------------------------------------
23690 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23691 States: Active Active Offline
23692 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23693 6434 Acked Read 10 Done Done Sent false
23694 6435 Acked Flush 0 Done Done Sent false
23695 6436 Acked Write 7 Done Done Sent false
23696 6437 Acked Flush 0 Done Done Sent false
23697 6438 Acked Read 9 Done Done Sent false
23698 6439 Acked Flush 0 Done Done Sent false
23699 6440 Acked Flush 0 Done Done Sent false
23700 6441 Acked Read 5 Done Done Sent false
23701 6442 Acked Write 2 Done Done Sent false
23702 6443 Acked Flush 0 Done Done Sent false
23703 6444 Acked Read 10 Done Done Sent false
23704 6445 Acked Write 10 Done Done Sent false
23705 6446 Acked Write 7 Done Done Sent false
23706 6447 Acked Write 10 Done Done Sent false
23707 6448 Acked Write 10 Done Done Sent false
23708 6449 Acked Read 9 Done Done Sent false
23709 6450 Acked Write 9 Done Done Sent false
23710 6451 Acked Write 1 Done Done Sent false
23711 6452 Acked Read 6 Done Done Sent false
23712 6453 Acked Write 7 Done Done Sent false
23713 6454 Acked Write 8 Done Done Sent false
23714 6455 Acked Flush 0 Done Done Sent false
23715 6456 Acked Flush 0 Done Done Sent false
23716 6457 Acked Write 3 Done Done Sent false
23717 6458 Acked Write 5 Done Done Sent false
23718 6459 Acked Read 3 Done Done Sent false
23719 6460 Acked Write 5 Done Done Sent false
23720 6461 Acked Write 6 Done Done Sent false
23721 6462 Acked Read 3 Done Done Sent false
23722 6463 Acked Flush 0 Done Done Sent false
23723 6464 Acked Read 8 Done Done Sent false
23724 6465 Acked Read 3 Done Done Sent false
23725 6466 Acked Write 2 Done Done Sent false
23726 6467 Acked Write 4 Done Done Sent false
23727 6468 Acked Read 6 Done Done Sent false
23728 6469 Acked Read 6 Done Done Sent false
23729 6470 Acked Write 3 Done Done Sent false
23730 6471 Acked Write 1 Done Done Sent false
23731 6472 Acked Read 10 Done Done Sent false
23732 6473 Acked Flush 0 Done Done Sent false
23733 6474 Acked Write 3 Done Done Sent false
23734 6475 Acked Read 9 Done Done Sent false
23735 6476 Acked Write 9 Done Done Sent false
23736 6477 Acked Write 7 Done Done Sent false
23737 6478 Acked Flush 0 Done Done Sent false
23738 6479 Acked Read 8 Done Done Sent false
23739 6480 Acked Write 9 Done Done Sent false
23740 6481 Acked Read 10 Done Done Sent false
23741 6482 Acked Write 6 Done Done Sent false
23742 6483 Acked Read 7 Done Done Sent false
23743 6484 Acked Write 8 Done Done Sent false
23744 6485 Acked Write 10 Done Done Sent false
23745 6486 Acked Write 2 Done Done Sent false
23746 6487 Acked Flush 0 Done Done Sent false
23747 6488 Acked Write 8 Done Done Sent false
23748 6489 Acked Write 10 Done Done Sent false
23749 6490 Acked Read 3 Done Done Sent false
23750 6491 Acked Read 4 Done Done Sent false
23751 6492 Acked Read 5 Done Done Sent false
23752 6493 Acked Write 8 Done Done Sent false
23753 6494 Acked Read 3 Done Done Sent false
23754 6495 Acked Write 9 Done Done Sent false
23755 6496 Acked Write 3 Done Done Sent false
23756 6497 Acked Write 2 Done Done Sent false
23757 6498 Acked Flush 0 Done Done Sent false
23758 6499 Acked Read 6 Done Done Sent false
23759 6500 Acked Read 8 Done Done Sent false
23760 6501 Acked Write 2 Done Done Sent false
23761 6502 Acked Flush 0 Done Done Sent false
23762 6503 Acked Read 5 Done Done Sent false
23763 6504 Acked Read 8 Done Done Sent false
23764 6505 Acked Read 8 Done Done Sent false
23765 6506 Acked Read 8 Done Done Sent false
23766 6507 Acked Flush 0 Done Done Sent false
23767 6508 Acked Flush 0 Done Done Sent false
23768 6509 Acked Write 10 Done Done Sent false
23769 6510 Acked Write 2 Done Done Sent false
23770 6511 Acked Flush 0 Done Done Sent false
23771 6512 Acked Write 10 Done Done Sent false
23772 6513 Acked Write 3 Done Done Sent false
23773 6514 Acked Read 6 Done Done Sent false
23774 6515 Acked Read 4 Done Done Sent false
23775 6516 Acked Read 5 Done Done Sent false
23776 6517 Acked Write 2 Done Done Sent false
23777 6518 Acked Read 8 Done Done Sent false
23778 6519 Acked Read 6 Done Done Sent false
23779 6520 Acked Read 9 Done Done Sent false
23780 6521 Acked Flush 0 Done Done Sent false
23781 6522 Acked Write 2 Done Done Sent false
23782 6523 Acked Flush 0 Done Done Sent false
23783 6524 Acked Read 10 Done Done Sent false
23784 6525 Acked Read 2 Done Done Sent false
23785 6526 Acked Read 6 Done Done Sent false
23786 6527 Acked Read 3 Done Done Sent false
23787 6528 Acked Read 6 Done Done Sent false
23788 6529 Acked Read 5 Done Done Sent false
23789 6530 Acked Flush 0 Done Done Sent false
23790 6531 Acked Flush 0 Done Done Sent false
23791 6532 Acked Write 4 Done Done Sent false
23792 6533 Acked Flush 0 Done Done Sent false
23793 6534 Acked Write 7 Done Done Sent false
23794 6535 Acked Read 5 Done Done Sent false
23795 6536 Acked Read 6 Done Done Sent false
23796 6537 Acked Flush 0 Done Done Sent false
23797 6538 Acked Read 9 Done Done Sent false
23798 6539 Acked Read 5 Done Done Sent false
23799 6540 Acked Read 8 Done Done Sent false
23800 6541 Acked Write 4 Done Done Sent false
23801 6542 Acked Flush 0 Done Done Sent false
23802 6543 Acked Read 4 Done Done Sent false
23803 6544 Acked Read 10 Done Done Sent false
23804 6545 Acked Read 10 Done Done Sent false
23805 6546 Acked Read 7 Done Done Sent false
23806 6547 Acked Flush 0 Done Done Sent false
23807 6548 Acked Flush 0 Done Done Sent false
23808 6549 Acked Read 10 Done Done Sent false
23809 6550 Acked Read 3 Done Done Sent false
23810 6551 Acked Read 1 Done Done Sent false
23811 6552 Acked Write 7 Done Done Sent false
23812 6553 Acked Read 9 Done Done Sent false
23813 6554 Acked Write 3 Done Done Sent false
23814 6555 Acked Read 8 Done Done Sent false
23815 6556 Acked Read 9 Done Done Sent false
23816 6557 Acked Write 4 Done Done Sent false
23817 6558 Acked Read 4 Done Done Sent false
23818 6559 Acked Write 5 Done Done Sent false
23819 6560 Acked Read 3 Done Done Sent false
23820 6561 Acked Write 2 Done Done Sent false
23821 6562 Acked Read 10 Done Done Sent false
23822 6563 Acked Read 1 Done Done Sent false
23823 6564 Acked Read 4 Done Done Sent false
23824 6565 Acked Read 2 Done Done Sent false
23825 6566 Acked Read 5 Done Done Sent false
23826 6567 Acked Read 3 Done Done Sent false
23827 6568 Acked Write 8 Done Done Sent false
23828 6569 Acked Write 4 Done Done Sent false
23829 6570 Acked Write 5 Done Done Sent false
23830 6571 Acked Write 3 Done Done Sent false
23831 6572 Acked Read 7 Done Done Sent false
23832 6573 Acked Read 7 Done Done Sent false
23833 6574 Acked Read 8 Done Done Sent false
23834 6575 Acked Write 6 Done Done Sent false
23835 6576 Acked Read 9 Done Done Sent false
23836 6577 Acked Read 7 Done Done Sent false
23837 6578 Acked Write 9 Done Done Sent false
23838 6579 Acked Flush 0 Done Done Sent false
23839 6580 Acked Write 8 Done Done Sent false
23840 6581 Acked Read 10 Done Done Sent false
23841 6582 Acked Write 9 Done Done Sent false
23842 6583 Acked Read 9 Done Done Sent false
23843 6584 Acked Write 4 Done Done Sent false
23844 6585 Acked Write 10 Done Done Sent false
23845 6586 Acked Read 3 Done Done Sent false
23846 6587 Acked Read 1 Done Done Sent false
23847 6588 Acked Write 2 Done Done Sent false
23848 6589 Acked Flush 0 Done Done Sent false
23849 6590 Acked Write 2 Done Done Sent false
23850 6591 Acked Write 1 Done Done Sent false
23851 6592 Acked Flush 0 Done Done Sent false
23852 6593 Acked Write 9 Done Done Sent false
23853 6594 Acked Read 8 Done Done Sent false
23854 6595 Acked Read 3 Done Done Sent false
23855 6596 Acked Read 10 Done Done Sent false
23856 6597 Acked Read 7 Done Done Sent false
23857 6598 Acked Write 5 Done Done Sent false
23858 6599 Acked Write 9 Done Done Sent false
23859 6600 Acked Write 2 Done Done Sent false
23860 6601 Acked Read 4 Done Done Sent false
23861 6602 Acked Read 3 Done Done Sent false
23862 6603 Acked Read 5 Done Done Sent false
23863 6604 Acked Write 5 Done Done Sent false
23864 6605 Acked Write 9 Done Done Sent false
23865 6606 Acked Read 2 Done Done Sent false
23866 6607 Acked Read 5 Done Done Sent false
23867 6608 Acked Write 10 Done Done Sent false
23868 6609 Acked Read 9 Done Done Sent false
23869 6610 Acked Flush 0 Done Done Sent false
23870 6611 Acked Write 5 Done Done Sent false
23871 6612 Acked Read 9 Done Done Sent false
23872 6613 Acked Read 2 Done Done Sent false
23873 6614 Acked Read 3 Done Done Sent false
23874 6615 Acked Read 1 Done Done Sent false
23875 6616 Acked Write 3 Done Done Sent false
23876 6617 Acked Write 5 Done Done Sent false
23877 6618 Acked Read 1 Done Done Sent false
23878 6619 Acked Flush 0 Done Done Sent false
23879 6620 Acked Flush 0 Done Done Sent false
23880 6621 Acked Read 3 Done Done Sent false
23881 6622 Acked Read 10 Done Done Sent false
23882 6623 Acked Write 8 Done Done Sent false
23883 6624 Acked Read 6 Done Done Sent false
23884 6625 Acked Flush 0 Done Done Sent false
23885 6626 Acked Read 4 Done Done Sent false
23886 6627 Acked Write 10 Done Done Sent false
23887 6628 Acked Write 1 Done Done Sent false
23888 6629 Acked Flush 0 Done Done Sent false
23889 6630 Acked Write 1 Done Done Sent false
23890 6631 Acked Read 1 Done Done Sent false
23891 6632 Acked Write 1 Done Done Sent false
23892 6633 Acked Write 10 Done Done Sent false
23893 6634 Acked Read 2 Done Done Sent false
23894 6635 Acked Read 4 Done Done Sent false
23895 6636 Acked Write 2 Done Done Sent false
23896 6637 Acked Read 8 Done Done Sent false
23897 6638 Acked Read 10 Done Done Sent false
23898 6639 Acked Read 7 Done Done Sent false
23899 6640 Acked Read 6 Done Done Sent false
23900 6641 Acked Write 1 Done Done Sent false
23901 6642 Acked Read 7 Done Done Sent false
23902 6643 Acked Read 3 Done Done Sent false
23903 6644 Acked Read 6 Done Done Sent false
23904 6645 Acked Write 5 Done Done Sent false
23905 6646 Acked Write 9 Done Done Sent false
23906 6647 Acked Write 8 Done Done Sent false
23907 6648 Acked Read 7 Done Done Sent false
23908 6649 Acked Write 9 Done Done Sent false
23909 6650 Acked Write 4 Done Done Sent false
23910 6651 Acked Write 6 Done Done Sent false
23911 6652 Acked Write 8 Done Done Sent false
23912 6653 Acked Write 8 Done Done Sent false
23913 6654 Acked Flush 0 Done Done Sent false
23914 6655 Acked Read 10 Done Done Sent false
23915 6656 Acked Write 5 Done Done Sent false
23916 6657 Acked Read 9 Done Done Sent false
23917 6658 Acked Write 9 Done Done Sent false
23918 6659 Acked Read 4 Done Done Sent false
23919 6660 Acked Write 4 Done Done Sent false
23920 6661 Acked Read 4 Done Done Sent false
23921 6662 Acked Read 10 Done Done Sent false
23922 6663 Acked Write 5 Done Done Sent false
23923 6664 Acked Write 6 Done Done Sent false
23924 6665 Acked Write 7 Done Done Sent false
23925 6666 Acked Write 8 Done Done Sent false
23926 6667 Acked Read 8 Done Done Sent false
23927 6668 Acked Write 7 Done Done Sent false
23928 6669 Acked Read 8 Done Done Sent false
23929 6670 Acked Write 1 Done Done Sent false
23930 6671 Acked Write 10 Done Done Sent false
23931 6672 Acked Read 2 Done Done Sent false
23932 6673 Acked Write 4 Done Done Sent false
23933 6674 Acked Read 10 Done Done Sent false
23934 6675 Acked Read 4 Done Done Sent false
23935 6676 Acked Write 8 Done Done Sent false
23936 6677 Acked Read 5 Done Done Sent false
23937 6678 Acked Write 3 Done Done Sent false
23938 6679 Acked Read 9 Done Done Sent false
23939 6680 Acked Write 8 Done Done Sent false
23940 6681 Acked Read 3 Done Done Sent false
23941 6682 Acked Write 6 Done Done Sent false
23942 6683 Acked Write 7 Done Done Sent false
23943 6684 Acked Write 9 Done Done Sent false
23944 6685 Acked Read 4 Done Done Sent false
23945 6686 Acked Read 7 Done Done Sent false
23946 6687 Acked Read 1 Done Done Sent false
23947 6688 Acked Read 2 Done Done Sent false
23948 6689 Acked Read 10 Done Done Sent false
23949 6690 Acked Read 6 Done Done Sent false
23950 6691 Acked Read 5 Done Done Sent false
23951 6692 Acked Write 10 Done Done Sent false
23952 6693 Acked Write 1 Done Done Sent false
23953 6694 Acked Write 7 Done Done Sent false
23954 6695 Acked Read 5 Done Done Sent false
23955 6696 Acked Write 6 Done Done Sent false
23956 6697 Acked Flush 0 Done Done Sent false
23957 6698 Acked Read 9 Done Done Sent false
23958 6699 Acked Read 9 Done Done Sent false
23959 6700 Acked Flush 0 Done Done Sent false
23960 6701 Acked Read 10 Done Done Sent false
23961 6702 Acked Read 8 Done Done Sent false
23962 6703 Acked Write 7 Done Done Sent false
23963 6704 Acked Read 9 Done Done Sent false
23964 6705 Acked Write 10 Done Done Sent false
23965 6706 Acked Write 1 Done Done Sent false
23966 6707 Acked Read 5 Done Done Sent false
23967 6708 Acked Read 5 Done Done Sent false
23968 6709 Acked Write 9 Done Done Sent false
23969 6710 Acked Write 4 Done Done Sent false
23970 6711 Acked Read 2 Done Done Sent false
23971 6712 Acked Read 2 Done Done Sent false
23972 6713 Acked Read 3 Done Done Sent false
23973 6714 Acked Write 1 Done Done Sent false
23974 6715 Acked Read 10 Done Done Sent false
23975 6716 Acked Read 3 Done Done Sent false
23976 6717 Acked Write 9 Done Done Sent false
23977 6718 Acked Read 7 Done Done Sent false
23978 6719 Acked Read 6 Done Done Sent false
23979 6720 Acked Write 9 Done Done Sent false
23980 6721 Acked Write 9 Done Done Sent false
23981 6722 Acked Write 5 Done Done Sent false
23982 6723 Acked Read 3 Done Done Sent false
23983 6724 Acked Write 7 Done Done Sent false
23984 6725 Acked Flush 0 Done Done Sent false
23985 6726 Acked Write 4 Done Done Sent false
23986 6727 Acked Read 9 Done Done Sent false
23987 6728 Acked Read 3 Done Done Sent false
23988 6729 Acked Flush 0 Done Done Sent false
23989 6730 Acked Read 5 Done Done Sent false
23990 6731 Acked Write 6 Done Done Sent false
23991 6732 Acked Write 8 Done Done Sent false
23992 6733 Acked Read 4 Done Done Sent false
23993 6734 Acked Read 7 Done Done Sent false
23994 STATES DS:0 DS:1 DS:2 TOTAL
23995 Sent 0 0 301 301
23996 Done 301 301 0 602
23997 Skipped 0 0 0 0
23998 Error 0 0 0 0
23999 Last Flush: Some(JobId(6729)) Some(JobId(6729)) Some(JobId(6433))
24000 Downstairs last five completed: 6433 6432 6431 6430 6429
24001 Upstairs last five completed: 6734 6733 6732 6731 6730
24002 CLIENT: Up:0 ds:301 act:2
240032025-03-17T22:24:15.418ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
240042025-03-17T22:24:16.404ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
240052025-03-17T22:24:16.664ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
240062025-03-17T22:24:16.665ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:46077) = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
240072025-03-17T22:24:16.666ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
240082025-03-17T22:24:16.666ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
240092025-03-17T22:24:16.666ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
240102025-03-17T22:24:16.666ZINFOcrucible: send last flush ID to this DS: Some(JobId(6433)) = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
240112025-03-17T22:24:16.666ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
240122025-03-17T22:24:16.666ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(6433)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
240132025-03-17T22:24:16.667ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(6433)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
24014 ----------------------------------------------------------------
24015 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
24016 Downstairs last five completed: 6735 6734 6733 6732 6731
24017 Upstairs last five completed: 6735 6734 6733 6732 6731
24018 CLIENT: Up:0 ds:0 act:3
24019 Replay: All jobs finished
24020 Main task has work to do, go find it
24021 got dsc Stop(0)
24022 stop 0
24023 Mar 17 22:24:18.919 [8810] Got stop action so:false kr:false
24024 INFO request completed, latency_us: 358, response_code: 204, uri: /stop/cid/0, method: POST, req_id: d651046c-610f-4a3c-81fa-fc6a81aebb09, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
24025 Mar 17 22:24:18.920 INFO request completed, latency_us: 150, response_code: 200, uri: /state/cid/0, method: GET, req_id: 2f004bda-4fac-4f99-84cd-51ea41c77d35, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
240262025-03-17T22:24:18.920ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
240272025-03-17T22:24:18.920ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
240282025-03-17T22:24:18.920ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
240292025-03-17T22:24:18.920ZINFOcrucible: client task is exiting = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
24030 [8810] Exited with: ExitStatus(unix_wait_status(9))
24031 [8810][0] reports Exit
24032 Mar 17 22:24:22.922 INFO request completed, latency_us: 241, response_code: 200, uri: /state/cid/0, method: GET, req_id: cfa0d947-c5bf-4c46-8cb0-dfffbac74c21, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
24033 001/300 Read block 767 len 32768
24034 002/300 Write block 752 len 12288 data: 3 4 4
24035 003/300 Write block 4625 len 4096 data: 3
24036 004/300 Read block 2230 len 24576
24037 005/300 Read block 4230 len 36864
24038 006/300 Write block 4705 len 40960 data: 3 4 4 4 5 6 5 5 6 8
24039 007/300 Read block 3341 len 16384
24040 008/300 Read block 1039 len 24576
24041 009/300 Write block 928 len 36864 data: 3 4 5 5 4 5 5 5 4
24042 010/300 Write block 1582 len 24576 data: 4 6 6 6 5 8
24043 011/300 Read block 3990 len 20480
24044 012/300 Read block 238 len 28672
24045 013/300 Write block 4665 len 16384 data: 5 6 6 5
24046 014/300 Write block 3614 len 20480 data: 3 2 2 2 4
24047 015/300 Write block 1581 len 40960 data: 3 5 7 7 7 6 9 10 10 8
24048 016/300 Write block 1559 len 12288 data: 6 6 6
24049 017/300 Write block 4990 len 20480 data: 5 6 5 5 2
24050 018/300 Write block 997 len 16384 data: 2 3 3 4
24051 019/300 Flush
24052 020/300 Read block 2784 len 4096
24053 021/300 Read block 4944 len 40960
24054 022/300 Read block 130 len 36864
24055 023/300 Read block 330 len 16384
24056 024/300 Read block 1058 len 4096
24057 025/300 Write block 701 len 36864 data: 6 6 6 5 5 3 3 4 4
24058 026/300 Read block 748 len 16384
24059 027/300 Write block 3102 len 24576 data: 4 4 3 3 3 4
24060 028/300 Read block 4300 len 20480
24061 029/300 Write block 134 len 28672 data: 5 4 2 3 3 3 3
24062 030/300 Read block 4110 len 36864
24063 031/300 Write block 3999 len 16384 data: 4 4 4 3
24064 032/300 Read block 1231 len 4096
24065 033/300 Write block 3425 len 36864 data: 4 6 5 7 7 7 7 6 6
24066 034/300 Write block 1160 len 20480 data: 3 3 3 3 3
24067 035/300 Write block 4208 len 20480 data: 6 6 8 6 6
24068 036/300 Read block 4640 len 24576
24069 037/300 Write block 2173 len 40960 data: 6 5 6 5 5 4 4 4 4 3
24070 038/300 Write block 2486 len 40960 data: 4 3 3 2 3 4 4 4 4 3
24071 039/300 Read block 3681 len 12288
24072 040/300 Write block 1757 len 16384 data: 5 6 6 4
24073 041/300 Write block 2316 len 32768 data: 6 6 6 5 4 3 3 3
24074 042/300 Read block 2006 len 20480
24075 043/300 Write block 3185 len 32768 data: 6 6 6 5 5 4 4 5
24076 044/300 Write block 3753 len 16384 data: 6 6 6 7
24077 045/300 Write block 4955 len 36864 data: 3 3 3 3 3 4 6 7 7
24078 046/300 Write block 968 len 20480 data: 4 4 4 3 4
24079 047/300 Flush
24080 048/300 Read block 215 len 4096
24081 049/300 Write block 2412 len 8192 data: 7 8
24082 050/300 Read block 4598 len 20480
24083 051/300 Flush
24084 052/300 Write block 4106 len 20480 data: 3 2 2 2 3
24085 053/300 Flush
24086 054/300 Write block 4688 len 24576 data: 2 2 2 3 4 4
24087 055/300 Write block 3877 len 24576 data: 6 5 5 4 4 3
24088 056/300 Write block 28 len 16384 data: 5 6 7 6
24089 057/300 Write block 72 len 40960 data: 5 7 6 5 4 4 5 4 3 4
24090 058/300 Write block 738 len 40960 data: 6 6 10 10 10 10 10 7 6 4
24091 059/300 Write block 1976 len 4096 data: 3
24092 060/300 Read block 2218 len 4096
24093 061/300 Read block 2674 len 40960
24094 062/300 Write block 123 len 24576 data: 7 7 5 6 6 6
24095 063/300 Read block 2615 len 36864
24096 064/300 Read block 2683 len 36864
24097 065/300 Flush
24098 066/300 Read block 140 len 8192
24099 067/300 Read block 1170 len 12288
24100 068/300 Write block 2746 len 24576 data: 5 4 4 4 3 2
24101 069/300 Read block 1332 len 36864
24102 070/300 Write block 471 len 8192 data: 5 6
24103 071/300 Write block 1100 len 12288 data: 4 4 4
24104 072/300 Read block 2052 len 32768
24105 073/300 Read block 3233 len 24576
24106 074/300 Read block 3485 len 40960
24107 075/300 Write block 2703 len 12288 data: 5 6 7
24108 076/300 Write block 1632 len 40960 data: 4 5 5 7 6 5 5 6 6 5
24109 077/300 Read block 3826 len 12288
24110 078/300 Flush
24111 079/300 Read block 1664 len 8192
24112 080/300 Read block 4941 len 32768
24113 081/300 Flush
24114 082/300 Read block 4603 len 20480
24115 083/300 Flush
24116 084/300 Write block 1876 len 24576 data: 7 6 7 9 7 6
24117 085/300 Write block 3109 len 40960 data: 4 5 6 6 7 8 7 7 7 6
24118 086/300 Read block 352 len 12288
24119 087/300 Read block 484 len 4096
24120 088/300 Flush
24121 089/300 Write block 1379 len 20480 data: 4 4 4 4 4
24122 090/300 Read block 3145 len 16384
24123 091/300 Read block 1098 len 16384
24124 092/300 Read block 2003 len 8192
24125 093/300 Read block 3207 len 20480
24126 094/300 Read block 981 len 28672
24127 095/300 Read block 2350 len 4096
24128 096/300 Read block 3801 len 36864
24129 097/300 Read block 2321 len 32768
24130 098/300 Read block 4420 len 28672
24131 099/300 Read block 2363 len 32768
24132 100/300 Read block 2311 len 4096
24133 101/300 Read block 4369 len 40960
24134 102/300 Read block 64 len 20480
24135 103/300 Flush
24136 104/300 Write block 4444 len 36864 data: 4 4 4 5 6 7 7 6 5
24137 105/300 Flush
24138 106/300 Read block 1488 len 28672
24139 107/300 Write block 580 len 32768 data: 4 3 4 4 3 3 3 4
24140 108/300 Read block 3426 len 4096
24141 109/300 Write block 3399 len 20480 data: 5 6 5 5 6
24142 110/300 Write block 1962 len 8192 data: 7 6
24143 111/300 Write block 1352 len 36864 data: 3 3 4 5 5 4 6 5 4
24144 112/300 Read block 4699 len 16384
24145 113/300 Write block 1250 len 24576 data: 5 7 7 6 6 6
24146 114/300 Read block 2987 len 8192
24147 115/300 Read block 2718 len 20480
24148 116/300 Write block 221 len 20480 data: 5 5 5 5 5
24149 117/300 Read block 3253 len 24576
24150 118/300 Read block 3285 len 16384
24151 119/300 Write block 4981 len 24576 data: 4 4 3 2 3 3
24152 120/300 Write block 4759 len 20480 data: 3 5 6 7 6
24153 121/300 Read block 3699 len 20480
24154 122/300 Read block 2398 len 4096
24155 123/300 Read block 2051 len 8192
24156 124/300 Read block 271 len 32768
24157 125/300 Read block 265 len 8192
24158 126/300 Read block 2746 len 40960
24159 127/300 Read block 1533 len 28672
24160 128/300 Read block 4046 len 16384
24161 129/300 Write block 54 len 36864 data: 3 3 4 4 4 4 4 5 5
24162 130/300 Write block 1603 len 36864 data: 5 5 5 5 4 6 7 7 6
24163 131/300 Read block 3374 len 32768
24164 132/300 Read block 723 len 40960
24165 133/300 Read block 4039 len 28672
24166 134/300 Write block 98 len 24576 data: 5 5 5 5 5 5
24167 135/300 Read block 4365 len 16384
24168 136/300 Write block 937 len 12288 data: 3 2 2
24169 137/300 Read block 3810 len 8192
24170 138/300 Write block 3082 len 8192 data: 9 6
24171 139/300 Read block 4648 len 4096
24172 140/300 Write block 3476 len 36864 data: 5 6 5 6 5 4 3 3 3
24173 141/300 Write block 958 len 20480 data: 5 5 6 4 4
24174 142/300 Read block 820 len 12288
24175 143/300 Flush
24176 144/300 Write block 369 len 4096 data: 7
24177 145/300 Write block 3207 len 36864 data: 4 5 4 4 4 4 4 5 4
24178 146/300 Write block 2540 len 24576 data: 5 6 5 5 5 6
24179 147/300 Read block 2527 len 20480
24180 148/300 Read block 1901 len 16384
24181 149/300 Write block 2338 len 4096 data: 6
24182 150/300 Write block 1214 len 12288 data: 4 5 2
24183 151/300 Write block 406 len 16384 data: 4 5 5 3
24184 152/300 Write block 4880 len 8192 data: 3 3
24185 153/300 Read block 925 len 24576
24186 154/300 Read block 1501 len 28672
24187 155/300 Read block 4697 len 28672
24188 156/300 Flush
24189 157/300 Write block 4887 len 32768 data: 3 4 4 4 3 3 3 4
24190 158/300 Write block 535 len 4096 data: 3
24191 159/300 Read block 4245 len 20480
24192 160/300 Read block 1907 len 20480
24193 161/300 Write block 1400 len 24576 data: 3 4 4 4 3 3
24194 162/300 Write block 1949 len 20480 data: 4 4 4 4 3
24195 163/300 Read block 3895 len 4096
24196 164/300 Write block 2243 len 28672 data: 7 6 6 6 6 7 8
24197 165/300 Flush
24198 166/300 Read block 3065 len 24576
24199 167/300 Write block 442 len 32768 data: 2 4 4 4 7 8 7 8
24200 168/300 Read block 4740 len 24576
24201 169/300 Read block 531 len 20480
24202 170/300 Write block 3450 len 20480 data: 4 4 4 4 3
24203 171/300 Read block 2183 len 24576
24204 172/300 Read block 26 len 4096
24205 173/300 Read block 3077 len 12288
24206 174/300 Read block 3603 len 40960
24207 175/300 Read block 2306 len 12288
24208 176/300 Write block 4565 len 24576 data: 4 3 3 5 6 5
24209 177/300 Read block 1550 len 32768
24210 178/300 Read block 1949 len 20480
24211 179/300 Write block 498 len 8192 data: 5 3
24212 180/300 Read block 3873 len 40960
24213 181/300 Read block 996 len 4096
24214 182/300 Flush
24215 183/300 Read block 2053 len 36864
24216 184/300 Write block 2965 len 36864 data: 6 5 5 5 7 6 6 4 4
24217 185/300 Write block 3694 len 40960 data: 7 7 7 5 6 5 4 4 4 3
24218 186/300 Read block 2495 len 40960
24219 187/300 Write block 1330 len 36864 data: 6 7 6 6 6 6 6 3 3
24220 188/300 Read block 2677 len 20480
24221 189/300 Write block 2146 len 8192 data: 6 6
24222 190/300 Read block 4478 len 8192
24223 191/300 Read block 2914 len 24576
24224 192/300 Read block 1805 len 4096
24225 193/300 Read block 87 len 8192
24226 194/300 Write block 1363 len 24576 data: 4 2 3 3 3 3
24227 195/300 Write block 1658 len 36864 data: 5 5 5 4 4 4 4 4 4
24228 196/300 Write block 1837 len 20480 data: 6 5 7 6 5
24229 197/300 Read block 3562 len 4096
24230 198/300 Write block 1549 len 28672 data: 7 7 7 7 8 8 7
24231 199/300 Flush
24232 200/300 Flush
24233 201/300 Read block 3228 len 28672
24234 202/300 Read block 2098 len 28672
24235 203/300 Write block 68 len 28672 data: 5 4 4 5 6 8 7
24236 204/300 Write block 2908 len 8192 data: 5 4
24237 205/300 Flush
24238 206/300 Write block 600 len 20480 data: 4 3 2 3 3
24239 207/300 Write block 3847 len 20480 data: 5 7 8 9 8
24240 208/300 Write block 563 len 8192 data: 3 3
24241 209/300 Write block 1481 len 20480 data: 5 4 4 3 3
24242 210/300 Write block 2456 len 36864 data: 3 3 3 3 4 3 2 2 2
24243 211/300 Read block 4587 len 20480
24244 212/300 Read block 192 len 40960
24245 213/300 Write block 1319 len 36864 data: 4 4 4 4 3 2 3 4 6
24246 214/300 Read block 3482 len 16384
24247 215/300 Read block 2528 len 8192
24248 216/300 Write block 789 len 36864 data: 5 4 4 3 3 3 4 5 7
24249 217/300 Read block 193 len 8192
24250 218/300 Read block 4867 len 20480
24251 219/300 Read block 1199 len 8192
24252 220/300 Read block 3115 len 24576
24253 221/300 Read block 3315 len 8192
24254 222/300 Write block 2622 len 8192 data: 6 6
24255 223/300 Write block 2574 len 8192 data: 4 4
24256 224/300 Read block 4055 len 4096
24257 225/300 Read block 2807 len 24576
24258 226/300 Read block 2244 len 12288
24259 227/300 Write block 282 len 12288 data: 5 4 4
24260 228/300 Write block 2079 len 24576 data: 4 4 3 4 4 4
24261 229/300 Read block 1938 len 16384
24262 230/300 Read block 2434 len 16384
24263 231/300 Read block 1187 len 40960
24264 232/300 Read block 1302 len 40960
24265 233/300 Read block 3605 len 32768
24266 234/300 Read block 4368 len 20480
24267 235/300 Read block 1178 len 4096
24268 236/300 Write block 4300 len 16384 data: 3 4 3 3
24269 237/300 Write block 1211 len 40960 data: 4 5 5 5 6 3 2 2 2 2
24270 238/300 Read block 2100 len 40960
24271 239/300 Read block 80 len 12288
24272 240/300 Write block 4155 len 24576 data: 5 6 7 3 4 5
24273 241/300 Write block 2766 len 4096 data: 5
24274 242/300 Write block 2549 len 16384 data: 5 5 4 5
24275 243/300 Write block 300 len 20480 data: 4 5 5 3 3
24276 244/300 Read block 3367 len 36864
24277 245/300 Flush
24278 246/300 Read block 1349 len 16384
24279 247/300 Read block 2402 len 16384
24280 248/300 Write block 331 len 40960 data: 3 2 2 2 4 4 3 3 3 4
24281 249/300 Read block 150 len 28672
24282 250/300 Flush
24283 251/300 Read block 4269 len 36864
24284 252/300 Write block 3574 len 12288 data: 4 6 6
24285 253/300 Read block 2461 len 12288
24286 254/300 Write block 2905 len 28672 data: 5 4 4 6 5 3 3
24287 255/300 Write block 3247 len 8192 data: 6 8
24288 256/300 Read block 1824 len 40960
24289 257/300 Read block 4585 len 12288
24290 258/300 Read block 4324 len 20480
24291 259/300 Flush
24292 260/300 Read block 569 len 36864
24293 261/300 Read block 2565 len 24576
24294 262/300 Write block 4297 len 20480 data: 3 2 3 4 5
24295 263/300 Write block 2418 len 24576 data: 7 8 10 7 7 6
24296 264/300 Write block 3632 len 40960 data: 3 3 3 4 5 6 6 6 6 4
24297 265/300 Write block 105 len 20480 data: 4 5 3 4 4
24298 266/300 Write block 2686 len 28672 data: 5 6 7 7 5 3 3
24299 267/300 Flush
24300 268/300 Read block 1082 len 32768
24301 269/300 Read block 3426 len 40960
24302 270/300 Flush
24303 271/300 Read block 1013 len 40960
24304 272/300 Write block 3503 len 36864 data: 6 7 7 5 5 6 5 6 5
24305 273/300 Read block 76 len 24576
24306 274/300 Flush
24307 275/300 Write block 4624 len 20480 data: 3 4 3 3 3
24308 276/300 Read block 2589 len 24576
24309 277/300 Write block 2888 len 40960 data: 4 4 3 3 2 2 4 4 5 6
24310 278/300 Write block 4932 len 28672 data: 7 7 4 5 5 5 4
24311 279/300 Write block 742 len 8192 data: 11 11
24312 280/300 Read block 4972 len 28672
24313 281/300 Flush
24314 282/300 Read block 2601 len 36864
24315 283/300 Write block 2742 len 36864 data: 7 5 5 4 6 5 5 5 4
24316 284/300 Read block 1449 len 28672
24317 285/300 Flush
24318 286/300 Flush
24319 287/300 Read block 4396 len 32768
24320 288/300 Flush
24321 289/300 Read block 3154 len 36864
24322 290/300 Read block 2425 len 24576
24323 291/300 Write block 4351 len 20480 data: 4 4 4 5 5
24324 292/300 Write block 2049 len 20480 data: 4 2 3 4 5
24325 293/300 Read block 3416 len 12288
24326 294/300 Flush
24327 295/300 Write block 4552 len 40960 data: 6 5 6 6 8 9 7 7 6 7
24328 296/300 Write block 4307 len 24576 data: 2 2 3 4 5 6
24329 297/300 Flush
24330 298/300 Write block 3577 len 32768 data: 5 7 7 6 5 4 4 4
24331 299/300 Write block 2035 len 16384 data: 3 3 4 5
24332 300/300 Write block 3892 len 32768 data: 4 3 3 3 3 2 2 2
24333 301/300 Read block 3885 len 4096
24334 Main task has work to do, go find it
24335 got dsc Start(0)
24336 start 0
24337 Mar 17 22:24:23.166 INFO request completed, latency_us: 274, response_code: 204, uri: /start/cid/0, method: POST, req_id: 7cb6689f-6af8-4f9f-af27-742d6e5ef32d, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
24338 Replay: started 0, returned:Ok(())
24339 ----------------------------------------------------------------
24340 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24341 States: Offline Active Active
24342 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24343 6736 Acked Read 8 Sent Done Done false
24344 6737 Acked Write 3 Sent Done Done false
24345 6738 Acked Write 1 Sent Done Done false
24346 6739 Acked Read 6 Sent Done Done false
24347 6740 Acked Read 9 Sent Done Done false
24348 6741 Acked Write 10 Sent Done Done false
24349 6742 Acked Read 4 Sent Done Done false
24350 6743 Acked Read 6 Sent Done Done false
24351 6744 Acked Write 9 Sent Done Done false
24352 6745 Acked Write 6 Sent Done Done false
24353 6746 Acked Read 5 Sent Done Done false
24354 6747 Acked Read 7 Sent Done Done false
24355 6748 Acked Write 4 Sent Done Done false
24356 6749 Acked Write 5 Sent Done Done false
24357 6750 Acked Write 10 Sent Done Done false
24358 6751 Acked Write 3 Sent Done Done false
24359 6752 Acked Write 5 Sent Done Done false
24360 6753 Acked Write 4 Sent Done Done false
24361 6754 Acked Flush 0 Sent Done Done false
24362 6755 Acked Read 1 Sent Done Done false
24363 6756 Acked Read 10 Sent Done Done false
24364 6757 Acked Read 9 Sent Done Done false
24365 6758 Acked Read 4 Sent Done Done false
24366 6759 Acked Read 1 Sent Done Done false
24367 6760 Acked Write 9 Sent Done Done false
24368 6761 Acked Read 4 Sent Done Done false
24369 6762 Acked Write 6 Sent Done Done false
24370 6763 Acked Read 5 Sent Done Done false
24371 6764 Acked Write 7 Sent Done Done false
24372 6765 Acked Read 9 Sent Done Done false
24373 6766 Acked Write 4 Sent Done Done false
24374 6767 Acked Read 1 Sent Done Done false
24375 6768 Acked Write 9 Sent Done Done false
24376 6769 Acked Write 5 Sent Done Done false
24377 6770 Acked Write 5 Sent Done Done false
24378 6771 Acked Read 6 Sent Done Done false
24379 6772 Acked Write 10 Sent Done Done false
24380 6773 Acked Write 10 Sent Done Done false
24381 6774 Acked Read 3 Sent Done Done false
24382 6775 Acked Write 4 Sent Done Done false
24383 6776 Acked Write 8 Sent Done Done false
24384 6777 Acked Read 5 Sent Done Done false
24385 6778 Acked Write 8 Sent Done Done false
24386 6779 Acked Write 4 Sent Done Done false
24387 6780 Acked Write 9 Sent Done Done false
24388 6781 Acked Write 5 Sent Done Done false
24389 6782 Acked Flush 0 Sent Done Done false
24390 6783 Acked Read 1 Sent Done Done false
24391 6784 Acked Write 2 Sent Done Done false
24392 6785 Acked Read 5 Sent Done Done false
24393 6786 Acked Flush 0 Sent Done Done false
24394 6787 Acked Write 5 Sent Done Done false
24395 6788 Acked Flush 0 Sent Done Done false
24396 6789 Acked Write 6 Sent Done Done false
24397 6790 Acked Write 6 Sent Done Done false
24398 6791 Acked Write 4 Sent Done Done false
24399 6792 Acked Write 10 Sent Done Done false
24400 6793 Acked Write 10 Sent Done Done false
24401 6794 Acked Write 1 Sent Done Done false
24402 6795 Acked Read 1 Sent Done Done false
24403 6796 Acked Read 10 Sent Done Done false
24404 6797 Acked Write 6 Sent Done Done false
24405 6798 Acked Read 9 Sent Done Done false
24406 6799 Acked Read 9 Sent Done Done false
24407 6800 Acked Flush 0 Sent Done Done false
24408 6801 Acked Read 2 Sent Done Done false
24409 6802 Acked Read 3 Sent Done Done false
24410 6803 Acked Write 6 Sent Done Done false
24411 6804 Acked Read 9 Sent Done Done false
24412 6805 Acked Write 2 Sent Done Done false
24413 6806 Acked Write 3 Sent Done Done false
24414 6807 Acked Read 8 Sent Done Done false
24415 6808 Acked Read 6 Sent Done Done false
24416 6809 Acked Read 10 Sent Done Done false
24417 6810 Acked Write 3 Sent Done Done false
24418 6811 Acked Write 10 Sent Done Done false
24419 6812 Acked Read 3 Sent Done Done false
24420 6813 Acked Flush 0 Sent Done Done false
24421 6814 Acked Read 2 Sent Done Done false
24422 6815 Acked Read 8 Sent Done Done false
24423 6816 Acked Flush 0 Sent Done Done false
24424 6817 Acked Read 5 Sent Done Done false
24425 6818 Acked Flush 0 Sent Done Done false
24426 6819 Acked Write 6 Sent Done Done false
24427 6820 Acked Write 10 Sent Done Done false
24428 6821 Acked Read 3 Sent Done Done false
24429 6822 Acked Read 1 Sent Done Done false
24430 6823 Acked Flush 0 Sent Done Done false
24431 6824 Acked Write 5 Sent Done Done false
24432 6825 Acked Read 4 Sent Done Done false
24433 6826 Acked Read 4 Sent Done Done false
24434 6827 Acked Read 2 Sent Done Done false
24435 6828 Acked Read 5 Sent Done Done false
24436 6829 Acked Read 7 Sent Done Done false
24437 6830 Acked Read 1 Sent Done Done false
24438 6831 Acked Read 9 Sent Done Done false
24439 6832 Acked Read 8 Sent Done Done false
24440 6833 Acked Read 7 Sent Done Done false
24441 6834 Acked Read 8 Sent Done Done false
24442 6835 Acked Read 1 Sent Done Done false
24443 6836 Acked Read 10 Sent Done Done false
24444 6837 Acked Read 5 Sent Done Done false
24445 6838 Acked Flush 0 Sent Done Done false
24446 6839 Acked Write 9 Sent Done Done false
24447 6840 Acked Flush 0 Sent Done Done false
24448 6841 Acked Read 7 Sent Done Done false
24449 6842 Acked Write 8 Sent Done Done false
24450 6843 Acked Read 1 Sent Done Done false
24451 6844 Acked Write 5 Sent Done Done false
24452 6845 Acked Write 2 Sent Done Done false
24453 6846 Acked Write 9 Sent Done Done false
24454 6847 Acked Read 4 Sent Done Done false
24455 6848 Acked Write 6 Sent Done Done false
24456 6849 Acked Read 2 Sent Done Done false
24457 6850 Acked Read 5 Sent Done Done false
24458 6851 Acked Write 5 Sent Done Done false
24459 6852 Acked Read 6 Sent Done Done false
24460 6853 Acked Read 4 Sent Done Done false
24461 6854 Acked Write 6 Sent Done Done false
24462 6855 Acked Write 5 Sent Done Done false
24463 6856 Acked Read 5 Sent Done Done false
24464 6857 Acked Read 1 Sent Done Done false
24465 6858 Acked Read 2 Sent Done Done false
24466 6859 Acked Read 8 Sent Done Done false
24467 6860 Acked Read 2 Sent Done Done false
24468 6861 Acked Read 10 Sent Done Done false
24469 6862 Acked Read 7 Sent Done Done false
24470 6863 Acked Read 4 Sent Done Done false
24471 6864 Acked Write 9 Sent Done Done false
24472 6865 Acked Write 9 Sent Done Done false
24473 6866 Acked Read 8 Sent Done Done false
24474 6867 Acked Read 10 Sent Done Done false
24475 6868 Acked Read 7 Sent Done Done false
24476 6869 Acked Write 6 Sent Done Done false
24477 6870 Acked Read 4 Sent Done Done false
24478 6871 Acked Write 3 Sent Done Done false
24479 6872 Acked Read 2 Sent Done Done false
24480 6873 Acked Write 2 Sent Done Done false
24481 6874 Acked Read 1 Sent Done Done false
24482 6875 Acked Write 9 Sent Done Done false
24483 6876 Acked Write 5 Sent Done Done false
24484 6877 Acked Read 3 Sent Done Done false
24485 6878 Acked Flush 0 Sent Done Done false
24486 6879 Acked Write 1 Sent Done Done false
24487 6880 Acked Write 9 Sent Done Done false
24488 6881 Acked Write 6 Sent Done Done false
24489 6882 Acked Read 5 Sent Done Done false
24490 6883 Acked Read 4 Sent Done Done false
24491 6884 Acked Write 1 Sent Done Done false
24492 6885 Acked Write 3 Sent Done Done false
24493 6886 Acked Write 4 Sent Done Done false
24494 6887 Acked Write 2 Sent Done Done false
24495 6888 Acked Read 6 Sent Done Done false
24496 6889 Acked Read 7 Sent Done Done false
24497 6890 Acked Read 7 Sent Done Done false
24498 6891 Acked Flush 0 Sent Done Done false
24499 6892 Acked Write 8 Sent Done Done false
24500 6893 Acked Write 1 Sent Done Done false
24501 6894 Acked Read 5 Sent Done Done false
24502 6895 Acked Read 5 Sent Done Done false
24503 6896 Acked Write 6 Sent Done Done false
24504 6897 Acked Write 5 Sent Done Done false
24505 6898 Acked Read 1 Sent Done Done false
24506 6899 Acked Write 7 Sent Done Done false
24507 6900 Acked Flush 0 Sent Done Done false
24508 6901 Acked Read 6 Sent Done Done false
24509 6902 Acked Write 8 Sent Done Done false
24510 6903 Acked Read 6 Sent Done Done false
24511 6904 Acked Read 5 Sent Done Done false
24512 6905 Acked Write 5 Sent Done Done false
24513 6906 Acked Read 6 Sent Done Done false
24514 6907 Acked Read 1 Sent Done Done false
24515 6908 Acked Read 3 Sent Done Done false
24516 6909 Acked Read 10 Sent Done Done false
24517 6910 Acked Read 3 Sent Done Done false
24518 6911 Acked Write 6 Sent Done Done false
24519 6912 Acked Read 8 Sent Done Done false
24520 6913 Acked Read 5 Sent Done Done false
24521 6914 Acked Write 2 Sent Done Done false
24522 6915 Acked Read 10 Sent Done Done false
24523 6916 Acked Read 1 Sent Done Done false
24524 6917 Acked Flush 0 Sent Done Done false
24525 6918 Acked Read 9 Sent Done Done false
24526 6919 Acked Write 9 Sent Done Done false
24527 6920 Acked Write 10 Sent Done Done false
24528 6921 Acked Read 10 Sent Done Done false
24529 6922 Acked Write 9 Sent Done Done false
24530 6923 Acked Read 5 Sent Done Done false
24531 6924 Acked Write 2 Sent Done Done false
24532 6925 Acked Read 2 Sent Done Done false
24533 6926 Acked Read 6 Sent Done Done false
24534 6927 Acked Read 1 Sent Done Done false
24535 6928 Acked Read 2 Sent Done Done false
24536 6929 Acked Write 6 Sent Done Done false
24537 6930 Acked Write 9 Sent Done Done false
24538 6931 Acked Write 5 Sent Done Done false
24539 6932 Acked Read 1 Sent Done Done false
24540 6933 Acked Write 7 Sent Done Done false
24541 6934 Acked Flush 0 Sent Done Done false
24542 6935 Acked Flush 0 Sent Done Done false
24543 6936 Acked Read 7 Sent Done Done false
24544 6937 Acked Read 7 Sent Done Done false
24545 6938 Acked Write 7 Sent Done Done false
24546 6939 Acked Write 2 Sent Done Done false
24547 6940 Acked Flush 0 Sent Done Done false
24548 6941 Acked Write 5 Sent Done Done false
24549 6942 Acked Write 5 Sent Done Done false
24550 6943 Acked Write 2 Sent Done Done false
24551 6944 Acked Write 5 Sent Done Done false
24552 6945 Acked Write 9 Sent Done Done false
24553 6946 Acked Read 5 Sent Done Done false
24554 6947 Acked Read 10 Sent Done Done false
24555 6948 Acked Write 9 Sent Done Done false
24556 6949 Acked Read 4 Sent Done Done false
24557 6950 Acked Read 2 Sent Done Done false
24558 6951 Acked Write 9 Sent Done Done false
24559 6952 Acked Read 2 Sent Done Done false
24560 6953 Acked Read 5 Sent Done Done false
24561 6954 Acked Read 2 Sent Done Done false
24562 6955 Acked Read 6 Sent Done Done false
24563 6956 Acked Read 2 Sent Done Done false
24564 6957 Acked Write 2 Sent Done Done false
24565 6958 Acked Write 2 Sent Done Done false
24566 6959 Acked Read 1 Sent Done Done false
24567 6960 Acked Read 6 Sent Done Done false
24568 6961 Acked Read 3 Sent Done Done false
24569 6962 Acked Write 3 Sent Done Done false
24570 6963 Acked Write 6 Sent Done Done false
24571 6964 Acked Read 4 Sent Done Done false
24572 6965 Acked Read 4 Sent Done Done false
24573 6966 Acked Read 10 Sent Done Done false
24574 6967 Acked Read 10 Sent Done Done false
24575 6968 Acked Read 8 Sent Done Done false
24576 6969 Acked Read 5 Sent Done Done false
24577 6970 Acked Read 1 Sent Done Done false
24578 6971 Acked Write 4 Sent Done Done false
24579 6972 Acked Write 10 Sent Done Done false
24580 6973 Acked Read 10 Sent Done Done false
24581 6974 Acked Read 3 Sent Done Done false
24582 6975 Acked Write 6 Sent Done Done false
24583 6976 Acked Write 1 Sent Done Done false
24584 6977 Acked Write 4 Sent Done Done false
24585 6978 Acked Write 5 Sent Done Done false
24586 6979 Acked Read 9 Sent Done Done false
24587 6980 Acked Flush 0 Sent Done Done false
24588 6981 Acked Read 4 Sent Done Done false
24589 6982 Acked Read 4 Sent Done Done false
24590 6983 Acked Write 10 Sent Done Done false
24591 6984 Acked Read 7 Sent Done Done false
24592 6985 Acked Flush 0 Sent Done Done false
24593 6986 Acked Read 9 Sent Done Done false
24594 6987 Acked Write 3 Sent Done Done false
24595 6988 Acked Read 3 Sent Done Done false
24596 6989 Acked Write 7 Sent Done Done false
24597 6990 Acked Write 2 Sent Done Done false
24598 6991 Acked Read 10 Sent Done Done false
24599 6992 Acked Read 3 Sent Done Done false
24600 6993 Acked Read 5 Sent Done Done false
24601 6994 Acked Flush 0 Sent Done Done false
24602 6995 Acked Read 9 Sent Done Done false
24603 6996 Acked Read 6 Sent Done Done false
24604 6997 Acked Write 5 Sent Done Done false
24605 6998 Acked Write 6 Sent Done Done false
24606 6999 Acked Write 10 Sent Done Done false
24607 7000 Acked Write 5 Sent Done Done false
24608 7001 Acked Write 7 Sent Done Done false
24609 7002 Acked Flush 0 Sent Done Done false
24610 7003 Acked Read 8 Sent Done Done false
24611 7004 Acked Read 10 Sent Done Done false
24612 7005 Acked Flush 0 Sent Done Done false
24613 7006 Acked Read 10 Sent Done Done false
24614 7007 Acked Write 9 Sent Done Done false
24615 7008 Acked Read 6 Sent Done Done false
24616 7009 Acked Flush 0 Sent Done Done false
24617 7010 Acked Write 5 Sent Done Done false
24618 7011 Acked Read 6 Sent Done Done false
24619 7012 Acked Write 10 Sent Done Done false
24620 7013 Acked Write 7 Sent Done Done false
24621 7014 Acked Write 2 Sent Done Done false
24622 7015 Acked Read 7 Sent Done Done false
24623 7016 Acked Flush 0 Sent Done Done false
24624 7017 Acked Read 9 Sent Done Done false
24625 7018 Acked Write 9 Sent Done Done false
24626 7019 Acked Read 7 Sent Done Done false
24627 7020 Acked Flush 0 Sent Done Done false
24628 7021 Acked Flush 0 Sent Done Done false
24629 7022 Acked Read 8 Sent Done Done false
24630 7023 Acked Flush 0 Sent Done Done false
24631 7024 Acked Read 9 Sent Done Done false
24632 7025 Acked Read 6 Sent Done Done false
24633 7026 Acked Write 5 Sent Done Done false
24634 7027 Acked Write 5 Sent Done Done false
24635 7028 Acked Read 3 Sent Done Done false
24636 7029 Acked Flush 0 Sent Done Done false
24637 7030 Acked Write 10 Sent Done Done false
24638 7031 Acked Write 6 Sent Done Done false
24639 7032 Acked Flush 0 Sent Done Done false
24640 7033 Acked Write 8 Sent Done Done false
24641 7034 Acked Write 4 Sent Done Done false
24642 7035 Acked Write 8 Sent Done Done false
24643 7036 Acked Read 1 Sent Done Done false
24644 STATES DS:0 DS:1 DS:2 TOTAL
24645 Sent 301 0 0 301
24646 Done 0 301 301 602
24647 Skipped 0 0 0 0
24648 Error 0 0 0 0
24649 Last Flush: Some(JobId(6735)) Some(JobId(7032)) Some(JobId(7032))
24650 Downstairs last five completed: 6735 6734 6733 6732 6731
24651 Upstairs last five completed: 7036 7035 7034 7033 7032
24652 CLIENT: Up:0 ds:301 act:2
246532025-03-17T22:24:24.093ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
246542025-03-17T22:24:25.095ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
246552025-03-17T22:24:26.098ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
24656 ----------------------------------------------------------------
24657 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24658 States: Offline Active Active
24659 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24660 6736 Acked Read 8 Sent Done Done false
24661 6737 Acked Write 3 Sent Done Done false
24662 6738 Acked Write 1 Sent Done Done false
24663 6739 Acked Read 6 Sent Done Done false
24664 6740 Acked Read 9 Sent Done Done false
24665 6741 Acked Write 10 Sent Done Done false
24666 6742 Acked Read 4 Sent Done Done false
24667 6743 Acked Read 6 Sent Done Done false
24668 6744 Acked Write 9 Sent Done Done false
24669 6745 Acked Write 6 Sent Done Done false
24670 6746 Acked Read 5 Sent Done Done false
24671 6747 Acked Read 7 Sent Done Done false
24672 6748 Acked Write 4 Sent Done Done false
24673 6749 Acked Write 5 Sent Done Done false
24674 6750 Acked Write 10 Sent Done Done false
24675 6751 Acked Write 3 Sent Done Done false
24676 6752 Acked Write 5 Sent Done Done false
24677 6753 Acked Write 4 Sent Done Done false
24678 6754 Acked Flush 0 Sent Done Done false
24679 6755 Acked Read 1 Sent Done Done false
24680 6756 Acked Read 10 Sent Done Done false
24681 6757 Acked Read 9 Sent Done Done false
24682 6758 Acked Read 4 Sent Done Done false
24683 6759 Acked Read 1 Sent Done Done false
24684 6760 Acked Write 9 Sent Done Done false
24685 6761 Acked Read 4 Sent Done Done false
24686 6762 Acked Write 6 Sent Done Done false
24687 6763 Acked Read 5 Sent Done Done false
24688 6764 Acked Write 7 Sent Done Done false
24689 6765 Acked Read 9 Sent Done Done false
24690 6766 Acked Write 4 Sent Done Done false
24691 6767 Acked Read 1 Sent Done Done false
24692 6768 Acked Write 9 Sent Done Done false
24693 6769 Acked Write 5 Sent Done Done false
24694 6770 Acked Write 5 Sent Done Done false
24695 6771 Acked Read 6 Sent Done Done false
24696 6772 Acked Write 10 Sent Done Done false
24697 6773 Acked Write 10 Sent Done Done false
24698 6774 Acked Read 3 Sent Done Done false
24699 6775 Acked Write 4 Sent Done Done false
24700 6776 Acked Write 8 Sent Done Done false
24701 6777 Acked Read 5 Sent Done Done false
24702 6778 Acked Write 8 Sent Done Done false
24703 6779 Acked Write 4 Sent Done Done false
24704 6780 Acked Write 9 Sent Done Done false
24705 6781 Acked Write 5 Sent Done Done false
24706 6782 Acked Flush 0 Sent Done Done false
24707 6783 Acked Read 1 Sent Done Done false
24708 6784 Acked Write 2 Sent Done Done false
24709 6785 Acked Read 5 Sent Done Done false
24710 6786 Acked Flush 0 Sent Done Done false
24711 6787 Acked Write 5 Sent Done Done false
24712 6788 Acked Flush 0 Sent Done Done false
24713 6789 Acked Write 6 Sent Done Done false
24714 6790 Acked Write 6 Sent Done Done false
24715 6791 Acked Write 4 Sent Done Done false
24716 6792 Acked Write 10 Sent Done Done false
24717 6793 Acked Write 10 Sent Done Done false
24718 6794 Acked Write 1 Sent Done Done false
24719 6795 Acked Read 1 Sent Done Done false
24720 6796 Acked Read 10 Sent Done Done false
24721 6797 Acked Write 6 Sent Done Done false
24722 6798 Acked Read 9 Sent Done Done false
24723 6799 Acked Read 9 Sent Done Done false
24724 6800 Acked Flush 0 Sent Done Done false
24725 6801 Acked Read 2 Sent Done Done false
24726 6802 Acked Read 3 Sent Done Done false
24727 6803 Acked Write 6 Sent Done Done false
24728 6804 Acked Read 9 Sent Done Done false
24729 6805 Acked Write 2 Sent Done Done false
24730 6806 Acked Write 3 Sent Done Done false
24731 6807 Acked Read 8 Sent Done Done false
24732 6808 Acked Read 6 Sent Done Done false
24733 6809 Acked Read 10 Sent Done Done false
24734 6810 Acked Write 3 Sent Done Done false
24735 6811 Acked Write 10 Sent Done Done false
24736 6812 Acked Read 3 Sent Done Done false
24737 6813 Acked Flush 0 Sent Done Done false
24738 6814 Acked Read 2 Sent Done Done false
24739 6815 Acked Read 8 Sent Done Done false
24740 6816 Acked Flush 0 Sent Done Done false
24741 6817 Acked Read 5 Sent Done Done false
24742 6818 Acked Flush 0 Sent Done Done false
24743 6819 Acked Write 6 Sent Done Done false
24744 6820 Acked Write 10 Sent Done Done false
24745 6821 Acked Read 3 Sent Done Done false
24746 6822 Acked Read 1 Sent Done Done false
24747 6823 Acked Flush 0 Sent Done Done false
24748 6824 Acked Write 5 Sent Done Done false
24749 6825 Acked Read 4 Sent Done Done false
24750 6826 Acked Read 4 Sent Done Done false
24751 6827 Acked Read 2 Sent Done Done false
24752 6828 Acked Read 5 Sent Done Done false
24753 6829 Acked Read 7 Sent Done Done false
24754 6830 Acked Read 1 Sent Done Done false
24755 6831 Acked Read 9 Sent Done Done false
24756 6832 Acked Read 8 Sent Done Done false
24757 6833 Acked Read 7 Sent Done Done false
24758 6834 Acked Read 8 Sent Done Done false
24759 6835 Acked Read 1 Sent Done Done false
24760 6836 Acked Read 10 Sent Done Done false
24761 6837 Acked Read 5 Sent Done Done false
24762 6838 Acked Flush 0 Sent Done Done false
24763 6839 Acked Write 9 Sent Done Done false
24764 6840 Acked Flush 0 Sent Done Done false
24765 6841 Acked Read 7 Sent Done Done false
24766 6842 Acked Write 8 Sent Done Done false
24767 6843 Acked Read 1 Sent Done Done false
24768 6844 Acked Write 5 Sent Done Done false
24769 6845 Acked Write 2 Sent Done Done false
24770 6846 Acked Write 9 Sent Done Done false
24771 6847 Acked Read 4 Sent Done Done false
24772 6848 Acked Write 6 Sent Done Done false
24773 6849 Acked Read 2 Sent Done Done false
24774 6850 Acked Read 5 Sent Done Done false
24775 6851 Acked Write 5 Sent Done Done false
24776 6852 Acked Read 6 Sent Done Done false
24777 6853 Acked Read 4 Sent Done Done false
24778 6854 Acked Write 6 Sent Done Done false
24779 6855 Acked Write 5 Sent Done Done false
24780 6856 Acked Read 5 Sent Done Done false
24781 6857 Acked Read 1 Sent Done Done false
24782 6858 Acked Read 2 Sent Done Done false
24783 6859 Acked Read 8 Sent Done Done false
24784 6860 Acked Read 2 Sent Done Done false
24785 6861 Acked Read 10 Sent Done Done false
24786 6862 Acked Read 7 Sent Done Done false
24787 6863 Acked Read 4 Sent Done Done false
24788 6864 Acked Write 9 Sent Done Done false
24789 6865 Acked Write 9 Sent Done Done false
24790 6866 Acked Read 8 Sent Done Done false
24791 6867 Acked Read 10 Sent Done Done false
24792 6868 Acked Read 7 Sent Done Done false
24793 6869 Acked Write 6 Sent Done Done false
24794 6870 Acked Read 4 Sent Done Done false
24795 6871 Acked Write 3 Sent Done Done false
24796 6872 Acked Read 2 Sent Done Done false
24797 6873 Acked Write 2 Sent Done Done false
24798 6874 Acked Read 1 Sent Done Done false
24799 6875 Acked Write 9 Sent Done Done false
24800 6876 Acked Write 5 Sent Done Done false
24801 6877 Acked Read 3 Sent Done Done false
24802 6878 Acked Flush 0 Sent Done Done false
24803 6879 Acked Write 1 Sent Done Done false
24804 6880 Acked Write 9 Sent Done Done false
24805 6881 Acked Write 6 Sent Done Done false
24806 6882 Acked Read 5 Sent Done Done false
24807 6883 Acked Read 4 Sent Done Done false
24808 6884 Acked Write 1 Sent Done Done false
24809 6885 Acked Write 3 Sent Done Done false
24810 6886 Acked Write 4 Sent Done Done false
24811 6887 Acked Write 2 Sent Done Done false
24812 6888 Acked Read 6 Sent Done Done false
24813 6889 Acked Read 7 Sent Done Done false
24814 6890 Acked Read 7 Sent Done Done false
24815 6891 Acked Flush 0 Sent Done Done false
24816 6892 Acked Write 8 Sent Done Done false
24817 6893 Acked Write 1 Sent Done Done false
24818 6894 Acked Read 5 Sent Done Done false
24819 6895 Acked Read 5 Sent Done Done false
24820 6896 Acked Write 6 Sent Done Done false
24821 6897 Acked Write 5 Sent Done Done false
24822 6898 Acked Read 1 Sent Done Done false
24823 6899 Acked Write 7 Sent Done Done false
24824 6900 Acked Flush 0 Sent Done Done false
24825 6901 Acked Read 6 Sent Done Done false
24826 6902 Acked Write 8 Sent Done Done false
24827 6903 Acked Read 6 Sent Done Done false
24828 6904 Acked Read 5 Sent Done Done false
24829 6905 Acked Write 5 Sent Done Done false
24830 6906 Acked Read 6 Sent Done Done false
24831 6907 Acked Read 1 Sent Done Done false
24832 6908 Acked Read 3 Sent Done Done false
24833 6909 Acked Read 10 Sent Done Done false
24834 6910 Acked Read 3 Sent Done Done false
24835 6911 Acked Write 6 Sent Done Done false
24836 6912 Acked Read 8 Sent Done Done false
24837 6913 Acked Read 5 Sent Done Done false
24838 6914 Acked Write 2 Sent Done Done false
24839 6915 Acked Read 10 Sent Done Done false
24840 6916 Acked Read 1 Sent Done Done false
24841 6917 Acked Flush 0 Sent Done Done false
24842 6918 Acked Read 9 Sent Done Done false
24843 6919 Acked Write 9 Sent Done Done false
24844 6920 Acked Write 10 Sent Done Done false
24845 6921 Acked Read 10 Sent Done Done false
24846 6922 Acked Write 9 Sent Done Done false
24847 6923 Acked Read 5 Sent Done Done false
24848 6924 Acked Write 2 Sent Done Done false
24849 6925 Acked Read 2 Sent Done Done false
24850 6926 Acked Read 6 Sent Done Done false
24851 6927 Acked Read 1 Sent Done Done false
24852 6928 Acked Read 2 Sent Done Done false
24853 6929 Acked Write 6 Sent Done Done false
24854 6930 Acked Write 9 Sent Done Done false
24855 6931 Acked Write 5 Sent Done Done false
24856 6932 Acked Read 1 Sent Done Done false
24857 6933 Acked Write 7 Sent Done Done false
24858 6934 Acked Flush 0 Sent Done Done false
24859 6935 Acked Flush 0 Sent Done Done false
24860 6936 Acked Read 7 Sent Done Done false
24861 6937 Acked Read 7 Sent Done Done false
24862 6938 Acked Write 7 Sent Done Done false
24863 6939 Acked Write 2 Sent Done Done false
24864 6940 Acked Flush 0 Sent Done Done false
24865 6941 Acked Write 5 Sent Done Done false
24866 6942 Acked Write 5 Sent Done Done false
24867 6943 Acked Write 2 Sent Done Done false
24868 6944 Acked Write 5 Sent Done Done false
24869 6945 Acked Write 9 Sent Done Done false
24870 6946 Acked Read 5 Sent Done Done false
24871 6947 Acked Read 10 Sent Done Done false
24872 6948 Acked Write 9 Sent Done Done false
24873 6949 Acked Read 4 Sent Done Done false
24874 6950 Acked Read 2 Sent Done Done false
24875 6951 Acked Write 9 Sent Done Done false
24876 6952 Acked Read 2 Sent Done Done false
24877 6953 Acked Read 5 Sent Done Done false
24878 6954 Acked Read 2 Sent Done Done false
24879 6955 Acked Read 6 Sent Done Done false
24880 6956 Acked Read 2 Sent Done Done false
24881 6957 Acked Write 2 Sent Done Done false
24882 6958 Acked Write 2 Sent Done Done false
24883 6959 Acked Read 1 Sent Done Done false
24884 6960 Acked Read 6 Sent Done Done false
24885 6961 Acked Read 3 Sent Done Done false
24886 6962 Acked Write 3 Sent Done Done false
24887 6963 Acked Write 6 Sent Done Done false
24888 6964 Acked Read 4 Sent Done Done false
24889 6965 Acked Read 4 Sent Done Done false
24890 6966 Acked Read 10 Sent Done Done false
24891 6967 Acked Read 10 Sent Done Done false
24892 6968 Acked Read 8 Sent Done Done false
24893 6969 Acked Read 5 Sent Done Done false
24894 6970 Acked Read 1 Sent Done Done false
24895 6971 Acked Write 4 Sent Done Done false
24896 6972 Acked Write 10 Sent Done Done false
24897 6973 Acked Read 10 Sent Done Done false
24898 6974 Acked Read 3 Sent Done Done false
24899 6975 Acked Write 6 Sent Done Done false
24900 6976 Acked Write 1 Sent Done Done false
24901 6977 Acked Write 4 Sent Done Done false
24902 6978 Acked Write 5 Sent Done Done false
24903 6979 Acked Read 9 Sent Done Done false
24904 6980 Acked Flush 0 Sent Done Done false
24905 6981 Acked Read 4 Sent Done Done false
24906 6982 Acked Read 4 Sent Done Done false
24907 6983 Acked Write 10 Sent Done Done false
24908 6984 Acked Read 7 Sent Done Done false
24909 6985 Acked Flush 0 Sent Done Done false
24910 6986 Acked Read 9 Sent Done Done false
24911 6987 Acked Write 3 Sent Done Done false
24912 6988 Acked Read 3 Sent Done Done false
24913 6989 Acked Write 7 Sent Done Done false
24914 6990 Acked Write 2 Sent Done Done false
24915 6991 Acked Read 10 Sent Done Done false
24916 6992 Acked Read 3 Sent Done Done false
24917 6993 Acked Read 5 Sent Done Done false
24918 6994 Acked Flush 0 Sent Done Done false
24919 6995 Acked Read 9 Sent Done Done false
24920 6996 Acked Read 6 Sent Done Done false
24921 6997 Acked Write 5 Sent Done Done false
24922 6998 Acked Write 6 Sent Done Done false
24923 6999 Acked Write 10 Sent Done Done false
24924 7000 Acked Write 5 Sent Done Done false
24925 7001 Acked Write 7 Sent Done Done false
24926 7002 Acked Flush 0 Sent Done Done false
24927 7003 Acked Read 8 Sent Done Done false
24928 7004 Acked Read 10 Sent Done Done false
24929 7005 Acked Flush 0 Sent Done Done false
24930 7006 Acked Read 10 Sent Done Done false
24931 7007 Acked Write 9 Sent Done Done false
24932 7008 Acked Read 6 Sent Done Done false
24933 7009 Acked Flush 0 Sent Done Done false
24934 7010 Acked Write 5 Sent Done Done false
24935 7011 Acked Read 6 Sent Done Done false
24936 7012 Acked Write 10 Sent Done Done false
24937 7013 Acked Write 7 Sent Done Done false
24938 7014 Acked Write 2 Sent Done Done false
24939 7015 Acked Read 7 Sent Done Done false
24940 7016 Acked Flush 0 Sent Done Done false
24941 7017 Acked Read 9 Sent Done Done false
24942 7018 Acked Write 9 Sent Done Done false
24943 7019 Acked Read 7 Sent Done Done false
24944 7020 Acked Flush 0 Sent Done Done false
24945 7021 Acked Flush 0 Sent Done Done false
24946 7022 Acked Read 8 Sent Done Done false
24947 7023 Acked Flush 0 Sent Done Done false
24948 7024 Acked Read 9 Sent Done Done false
24949 7025 Acked Read 6 Sent Done Done false
24950 7026 Acked Write 5 Sent Done Done false
24951 7027 Acked Write 5 Sent Done Done false
24952 7028 Acked Read 3 Sent Done Done false
24953 7029 Acked Flush 0 Sent Done Done false
24954 7030 Acked Write 10 Sent Done Done false
24955 7031 Acked Write 6 Sent Done Done false
24956 7032 Acked Flush 0 Sent Done Done false
24957 7033 Acked Write 8 Sent Done Done false
24958 7034 Acked Write 4 Sent Done Done false
24959 7035 Acked Write 8 Sent Done Done false
24960 7036 Acked Read 1 Sent Done Done false
24961 STATES DS:0 DS:1 DS:2 TOTAL
24962 Sent 301 0 0 301
24963 Done 0 301 301 602
24964 Skipped 0 0 0 0
24965 Error 0 0 0 0
24966 Last Flush: Some(JobId(6735)) Some(JobId(7032)) Some(JobId(7032))
24967 Downstairs last five completed: 6735 6734 6733 6732 6731
24968 Upstairs last five completed: 7036 7035 7034 7033 7032
24969 CLIENT: Up:0 ds:301 act:2
249702025-03-17T22:24:28.101ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
249712025-03-17T22:24:28.921ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
249722025-03-17T22:24:28.922ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
249732025-03-17T22:24:28.922ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
249742025-03-17T22:24:28.922ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
249752025-03-17T22:24:28.922ZINFOcrucible: client task is exiting = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
24976 [8810] got start action
24977 [8810] I am going to restart
24978 Starting downstairs at port 8810
24979 Make output file at "/tmp/dsc/downstairs-8810.txt"
24980 [8810][0] reports Starting
24981 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1231)
24982 [8810][0] reports Running
249832025-03-17T22:24:29.423ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
249842025-03-17T22:24:30.105ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
249852025-03-17T22:24:31.106ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
24986 ----------------------------------------------------------------
24987 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24988 States: Offline Active Active
24989 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24990 6736 Acked Read 8 Sent Done Done false
24991 6737 Acked Write 3 Sent Done Done false
24992 6738 Acked Write 1 Sent Done Done false
24993 6739 Acked Read 6 Sent Done Done false
24994 6740 Acked Read 9 Sent Done Done false
24995 6741 Acked Write 10 Sent Done Done false
24996 6742 Acked Read 4 Sent Done Done false
24997 6743 Acked Read 6 Sent Done Done false
24998 6744 Acked Write 9 Sent Done Done false
24999 6745 Acked Write 6 Sent Done Done false
25000 6746 Acked Read 5 Sent Done Done false
25001 6747 Acked Read 7 Sent Done Done false
25002 6748 Acked Write 4 Sent Done Done false
25003 6749 Acked Write 5 Sent Done Done false
25004 6750 Acked Write 10 Sent Done Done false
25005 6751 Acked Write 3 Sent Done Done false
25006 6752 Acked Write 5 Sent Done Done false
25007 6753 Acked Write 4 Sent Done Done false
25008 6754 Acked Flush 0 Sent Done Done false
25009 6755 Acked Read 1 Sent Done Done false
25010 6756 Acked Read 10 Sent Done Done false
25011 6757 Acked Read 9 Sent Done Done false
25012 6758 Acked Read 4 Sent Done Done false
25013 6759 Acked Read 1 Sent Done Done false
25014 6760 Acked Write 9 Sent Done Done false
25015 6761 Acked Read 4 Sent Done Done false
25016 6762 Acked Write 6 Sent Done Done false
25017 6763 Acked Read 5 Sent Done Done false
25018 6764 Acked Write 7 Sent Done Done false
25019 6765 Acked Read 9 Sent Done Done false
25020 6766 Acked Write 4 Sent Done Done false
25021 6767 Acked Read 1 Sent Done Done false
25022 6768 Acked Write 9 Sent Done Done false
25023 6769 Acked Write 5 Sent Done Done false
25024 6770 Acked Write 5 Sent Done Done false
25025 6771 Acked Read 6 Sent Done Done false
25026 6772 Acked Write 10 Sent Done Done false
25027 6773 Acked Write 10 Sent Done Done false
25028 6774 Acked Read 3 Sent Done Done false
25029 6775 Acked Write 4 Sent Done Done false
25030 6776 Acked Write 8 Sent Done Done false
25031 6777 Acked Read 5 Sent Done Done false
25032 6778 Acked Write 8 Sent Done Done false
25033 6779 Acked Write 4 Sent Done Done false
25034 6780 Acked Write 9 Sent Done Done false
25035 6781 Acked Write 5 Sent Done Done false
25036 6782 Acked Flush 0 Sent Done Done false
25037 6783 Acked Read 1 Sent Done Done false
25038 6784 Acked Write 2 Sent Done Done false
25039 6785 Acked Read 5 Sent Done Done false
25040 6786 Acked Flush 0 Sent Done Done false
25041 6787 Acked Write 5 Sent Done Done false
25042 6788 Acked Flush 0 Sent Done Done false
25043 6789 Acked Write 6 Sent Done Done false
25044 6790 Acked Write 6 Sent Done Done false
25045 6791 Acked Write 4 Sent Done Done false
25046 6792 Acked Write 10 Sent Done Done false
25047 6793 Acked Write 10 Sent Done Done false
25048 6794 Acked Write 1 Sent Done Done false
25049 6795 Acked Read 1 Sent Done Done false
25050 6796 Acked Read 10 Sent Done Done false
25051 6797 Acked Write 6 Sent Done Done false
25052 6798 Acked Read 9 Sent Done Done false
25053 6799 Acked Read 9 Sent Done Done false
25054 6800 Acked Flush 0 Sent Done Done false
25055 6801 Acked Read 2 Sent Done Done false
25056 6802 Acked Read 3 Sent Done Done false
25057 6803 Acked Write 6 Sent Done Done false
25058 6804 Acked Read 9 Sent Done Done false
25059 6805 Acked Write 2 Sent Done Done false
25060 6806 Acked Write 3 Sent Done Done false
25061 6807 Acked Read 8 Sent Done Done false
25062 6808 Acked Read 6 Sent Done Done false
25063 6809 Acked Read 10 Sent Done Done false
25064 6810 Acked Write 3 Sent Done Done false
25065 6811 Acked Write 10 Sent Done Done false
25066 6812 Acked Read 3 Sent Done Done false
25067 6813 Acked Flush 0 Sent Done Done false
25068 6814 Acked Read 2 Sent Done Done false
25069 6815 Acked Read 8 Sent Done Done false
25070 6816 Acked Flush 0 Sent Done Done false
25071 6817 Acked Read 5 Sent Done Done false
25072 6818 Acked Flush 0 Sent Done Done false
25073 6819 Acked Write 6 Sent Done Done false
25074 6820 Acked Write 10 Sent Done Done false
25075 6821 Acked Read 3 Sent Done Done false
25076 6822 Acked Read 1 Sent Done Done false
25077 6823 Acked Flush 0 Sent Done Done false
25078 6824 Acked Write 5 Sent Done Done false
25079 6825 Acked Read 4 Sent Done Done false
25080 6826 Acked Read 4 Sent Done Done false
25081 6827 Acked Read 2 Sent Done Done false
25082 6828 Acked Read 5 Sent Done Done false
25083 6829 Acked Read 7 Sent Done Done false
25084 6830 Acked Read 1 Sent Done Done false
25085 6831 Acked Read 9 Sent Done Done false
25086 6832 Acked Read 8 Sent Done Done false
25087 6833 Acked Read 7 Sent Done Done false
25088 6834 Acked Read 8 Sent Done Done false
25089 6835 Acked Read 1 Sent Done Done false
25090 6836 Acked Read 10 Sent Done Done false
25091 6837 Acked Read 5 Sent Done Done false
25092 6838 Acked Flush 0 Sent Done Done false
25093 6839 Acked Write 9 Sent Done Done false
25094 6840 Acked Flush 0 Sent Done Done false
25095 6841 Acked Read 7 Sent Done Done false
25096 6842 Acked Write 8 Sent Done Done false
25097 6843 Acked Read 1 Sent Done Done false
25098 6844 Acked Write 5 Sent Done Done false
25099 6845 Acked Write 2 Sent Done Done false
25100 6846 Acked Write 9 Sent Done Done false
25101 6847 Acked Read 4 Sent Done Done false
25102 6848 Acked Write 6 Sent Done Done false
25103 6849 Acked Read 2 Sent Done Done false
25104 6850 Acked Read 5 Sent Done Done false
25105 6851 Acked Write 5 Sent Done Done false
25106 6852 Acked Read 6 Sent Done Done false
25107 6853 Acked Read 4 Sent Done Done false
25108 6854 Acked Write 6 Sent Done Done false
25109 6855 Acked Write 5 Sent Done Done false
25110 6856 Acked Read 5 Sent Done Done false
25111 6857 Acked Read 1 Sent Done Done false
25112 6858 Acked Read 2 Sent Done Done false
25113 6859 Acked Read 8 Sent Done Done false
25114 6860 Acked Read 2 Sent Done Done false
25115 6861 Acked Read 10 Sent Done Done false
25116 6862 Acked Read 7 Sent Done Done false
25117 6863 Acked Read 4 Sent Done Done false
25118 6864 Acked Write 9 Sent Done Done false
25119 6865 Acked Write 9 Sent Done Done false
25120 6866 Acked Read 8 Sent Done Done false
25121 6867 Acked Read 10 Sent Done Done false
25122 6868 Acked Read 7 Sent Done Done false
25123 6869 Acked Write 6 Sent Done Done false
25124 6870 Acked Read 4 Sent Done Done false
25125 6871 Acked Write 3 Sent Done Done false
25126 6872 Acked Read 2 Sent Done Done false
25127 6873 Acked Write 2 Sent Done Done false
25128 6874 Acked Read 1 Sent Done Done false
25129 6875 Acked Write 9 Sent Done Done false
25130 6876 Acked Write 5 Sent Done Done false
25131 6877 Acked Read 3 Sent Done Done false
25132 6878 Acked Flush 0 Sent Done Done false
25133 6879 Acked Write 1 Sent Done Done false
25134 6880 Acked Write 9 Sent Done Done false
25135 6881 Acked Write 6 Sent Done Done false
25136 6882 Acked Read 5 Sent Done Done false
25137 6883 Acked Read 4 Sent Done Done false
25138 6884 Acked Write 1 Sent Done Done false
25139 6885 Acked Write 3 Sent Done Done false
25140 6886 Acked Write 4 Sent Done Done false
25141 6887 Acked Write 2 Sent Done Done false
25142 6888 Acked Read 6 Sent Done Done false
25143 6889 Acked Read 7 Sent Done Done false
25144 6890 Acked Read 7 Sent Done Done false
25145 6891 Acked Flush 0 Sent Done Done false
25146 6892 Acked Write 8 Sent Done Done false
25147 6893 Acked Write 1 Sent Done Done false
25148 6894 Acked Read 5 Sent Done Done false
25149 6895 Acked Read 5 Sent Done Done false
25150 6896 Acked Write 6 Sent Done Done false
25151 6897 Acked Write 5 Sent Done Done false
25152 6898 Acked Read 1 Sent Done Done false
25153 6899 Acked Write 7 Sent Done Done false
25154 6900 Acked Flush 0 Sent Done Done false
25155 6901 Acked Read 6 Sent Done Done false
25156 6902 Acked Write 8 Sent Done Done false
25157 6903 Acked Read 6 Sent Done Done false
25158 6904 Acked Read 5 Sent Done Done false
25159 6905 Acked Write 5 Sent Done Done false
25160 6906 Acked Read 6 Sent Done Done false
25161 6907 Acked Read 1 Sent Done Done false
25162 6908 Acked Read 3 Sent Done Done false
25163 6909 Acked Read 10 Sent Done Done false
25164 6910 Acked Read 3 Sent Done Done false
25165 6911 Acked Write 6 Sent Done Done false
25166 6912 Acked Read 8 Sent Done Done false
25167 6913 Acked Read 5 Sent Done Done false
25168 6914 Acked Write 2 Sent Done Done false
25169 6915 Acked Read 10 Sent Done Done false
25170 6916 Acked Read 1 Sent Done Done false
25171 6917 Acked Flush 0 Sent Done Done false
25172 6918 Acked Read 9 Sent Done Done false
25173 6919 Acked Write 9 Sent Done Done false
25174 6920 Acked Write 10 Sent Done Done false
25175 6921 Acked Read 10 Sent Done Done false
25176 6922 Acked Write 9 Sent Done Done false
25177 6923 Acked Read 5 Sent Done Done false
25178 6924 Acked Write 2 Sent Done Done false
25179 6925 Acked Read 2 Sent Done Done false
25180 6926 Acked Read 6 Sent Done Done false
25181 6927 Acked Read 1 Sent Done Done false
25182 6928 Acked Read 2 Sent Done Done false
25183 6929 Acked Write 6 Sent Done Done false
25184 6930 Acked Write 9 Sent Done Done false
25185 6931 Acked Write 5 Sent Done Done false
25186 6932 Acked Read 1 Sent Done Done false
25187 6933 Acked Write 7 Sent Done Done false
25188 6934 Acked Flush 0 Sent Done Done false
25189 6935 Acked Flush 0 Sent Done Done false
25190 6936 Acked Read 7 Sent Done Done false
25191 6937 Acked Read 7 Sent Done Done false
25192 6938 Acked Write 7 Sent Done Done false
25193 6939 Acked Write 2 Sent Done Done false
25194 6940 Acked Flush 0 Sent Done Done false
25195 6941 Acked Write 5 Sent Done Done false
25196 6942 Acked Write 5 Sent Done Done false
25197 6943 Acked Write 2 Sent Done Done false
25198 6944 Acked Write 5 Sent Done Done false
25199 6945 Acked Write 9 Sent Done Done false
25200 6946 Acked Read 5 Sent Done Done false
25201 6947 Acked Read 10 Sent Done Done false
25202 6948 Acked Write 9 Sent Done Done false
25203 6949 Acked Read 4 Sent Done Done false
25204 6950 Acked Read 2 Sent Done Done false
25205 6951 Acked Write 9 Sent Done Done false
25206 6952 Acked Read 2 Sent Done Done false
25207 6953 Acked Read 5 Sent Done Done false
25208 6954 Acked Read 2 Sent Done Done false
25209 6955 Acked Read 6 Sent Done Done false
25210 6956 Acked Read 2 Sent Done Done false
25211 6957 Acked Write 2 Sent Done Done false
25212 6958 Acked Write 2 Sent Done Done false
25213 6959 Acked Read 1 Sent Done Done false
25214 6960 Acked Read 6 Sent Done Done false
25215 6961 Acked Read 3 Sent Done Done false
25216 6962 Acked Write 3 Sent Done Done false
25217 6963 Acked Write 6 Sent Done Done false
25218 6964 Acked Read 4 Sent Done Done false
25219 6965 Acked Read 4 Sent Done Done false
25220 6966 Acked Read 10 Sent Done Done false
25221 6967 Acked Read 10 Sent Done Done false
25222 6968 Acked Read 8 Sent Done Done false
25223 6969 Acked Read 5 Sent Done Done false
25224 6970 Acked Read 1 Sent Done Done false
25225 6971 Acked Write 4 Sent Done Done false
25226 6972 Acked Write 10 Sent Done Done false
25227 6973 Acked Read 10 Sent Done Done false
25228 6974 Acked Read 3 Sent Done Done false
25229 6975 Acked Write 6 Sent Done Done false
25230 6976 Acked Write 1 Sent Done Done false
25231 6977 Acked Write 4 Sent Done Done false
25232 6978 Acked Write 5 Sent Done Done false
25233 6979 Acked Read 9 Sent Done Done false
25234 6980 Acked Flush 0 Sent Done Done false
25235 6981 Acked Read 4 Sent Done Done false
25236 6982 Acked Read 4 Sent Done Done false
25237 6983 Acked Write 10 Sent Done Done false
25238 6984 Acked Read 7 Sent Done Done false
25239 6985 Acked Flush 0 Sent Done Done false
25240 6986 Acked Read 9 Sent Done Done false
25241 6987 Acked Write 3 Sent Done Done false
25242 6988 Acked Read 3 Sent Done Done false
25243 6989 Acked Write 7 Sent Done Done false
25244 6990 Acked Write 2 Sent Done Done false
25245 6991 Acked Read 10 Sent Done Done false
25246 6992 Acked Read 3 Sent Done Done false
25247 6993 Acked Read 5 Sent Done Done false
25248 6994 Acked Flush 0 Sent Done Done false
25249 6995 Acked Read 9 Sent Done Done false
25250 6996 Acked Read 6 Sent Done Done false
25251 6997 Acked Write 5 Sent Done Done false
25252 6998 Acked Write 6 Sent Done Done false
25253 6999 Acked Write 10 Sent Done Done false
25254 7000 Acked Write 5 Sent Done Done false
25255 7001 Acked Write 7 Sent Done Done false
25256 7002 Acked Flush 0 Sent Done Done false
25257 7003 Acked Read 8 Sent Done Done false
25258 7004 Acked Read 10 Sent Done Done false
25259 7005 Acked Flush 0 Sent Done Done false
25260 7006 Acked Read 10 Sent Done Done false
25261 7007 Acked Write 9 Sent Done Done false
25262 7008 Acked Read 6 Sent Done Done false
25263 7009 Acked Flush 0 Sent Done Done false
25264 7010 Acked Write 5 Sent Done Done false
25265 7011 Acked Read 6 Sent Done Done false
25266 7012 Acked Write 10 Sent Done Done false
25267 7013 Acked Write 7 Sent Done Done false
25268 7014 Acked Write 2 Sent Done Done false
25269 7015 Acked Read 7 Sent Done Done false
25270 7016 Acked Flush 0 Sent Done Done false
25271 7017 Acked Read 9 Sent Done Done false
25272 7018 Acked Write 9 Sent Done Done false
25273 7019 Acked Read 7 Sent Done Done false
25274 7020 Acked Flush 0 Sent Done Done false
25275 7021 Acked Flush 0 Sent Done Done false
25276 7022 Acked Read 8 Sent Done Done false
25277 7023 Acked Flush 0 Sent Done Done false
25278 7024 Acked Read 9 Sent Done Done false
25279 7025 Acked Read 6 Sent Done Done false
25280 7026 Acked Write 5 Sent Done Done false
25281 7027 Acked Write 5 Sent Done Done false
25282 7028 Acked Read 3 Sent Done Done false
25283 7029 Acked Flush 0 Sent Done Done false
25284 7030 Acked Write 10 Sent Done Done false
25285 7031 Acked Write 6 Sent Done Done false
25286 7032 Acked Flush 0 Sent Done Done false
25287 7033 Acked Write 8 Sent Done Done false
25288 7034 Acked Write 4 Sent Done Done false
25289 7035 Acked Write 8 Sent Done Done false
25290 7036 Acked Read 1 Sent Done Done false
25291 STATES DS:0 DS:1 DS:2 TOTAL
25292 Sent 301 0 0 301
25293 Done 0 301 301 602
25294 Skipped 0 0 0 0
25295 Error 0 0 0 0
25296 Last Flush: Some(JobId(6735)) Some(JobId(7032)) Some(JobId(7032))
25297 Downstairs last five completed: 6735 6734 6733 6732 6731
25298 Upstairs last five completed: 7036 7035 7034 7033 7032
25299 CLIENT: Up:0 ds:301 act:2
253002025-03-17T22:24:32.423ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
253012025-03-17T22:24:33.110ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
253022025-03-17T22:24:34.112ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
253032025-03-17T22:24:35.114ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
25304 ----------------------------------------------------------------
25305 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25306 States: Offline Active Active
25307 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25308 6736 Acked Read 8 Sent Done Done false
25309 6737 Acked Write 3 Sent Done Done false
25310 6738 Acked Write 1 Sent Done Done false
25311 6739 Acked Read 6 Sent Done Done false
25312 6740 Acked Read 9 Sent Done Done false
25313 6741 Acked Write 10 Sent Done Done false
25314 6742 Acked Read 4 Sent Done Done false
25315 6743 Acked Read 6 Sent Done Done false
25316 6744 Acked Write 9 Sent Done Done false
25317 6745 Acked Write 6 Sent Done Done false
25318 6746 Acked Read 5 Sent Done Done false
25319 6747 Acked Read 7 Sent Done Done false
25320 6748 Acked Write 4 Sent Done Done false
25321 6749 Acked Write 5 Sent Done Done false
25322 6750 Acked Write 10 Sent Done Done false
25323 6751 Acked Write 3 Sent Done Done false
25324 6752 Acked Write 5 Sent Done Done false
25325 6753 Acked Write 4 Sent Done Done false
25326 6754 Acked Flush 0 Sent Done Done false
25327 6755 Acked Read 1 Sent Done Done false
25328 6756 Acked Read 10 Sent Done Done false
25329 6757 Acked Read 9 Sent Done Done false
25330 6758 Acked Read 4 Sent Done Done false
25331 6759 Acked Read 1 Sent Done Done false
25332 6760 Acked Write 9 Sent Done Done false
25333 6761 Acked Read 4 Sent Done Done false
25334 6762 Acked Write 6 Sent Done Done false
25335 6763 Acked Read 5 Sent Done Done false
25336 6764 Acked Write 7 Sent Done Done false
25337 6765 Acked Read 9 Sent Done Done false
25338 6766 Acked Write 4 Sent Done Done false
25339 6767 Acked Read 1 Sent Done Done false
25340 6768 Acked Write 9 Sent Done Done false
25341 6769 Acked Write 5 Sent Done Done false
25342 6770 Acked Write 5 Sent Done Done false
25343 6771 Acked Read 6 Sent Done Done false
25344 6772 Acked Write 10 Sent Done Done false
25345 6773 Acked Write 10 Sent Done Done false
25346 6774 Acked Read 3 Sent Done Done false
25347 6775 Acked Write 4 Sent Done Done false
25348 6776 Acked Write 8 Sent Done Done false
25349 6777 Acked Read 5 Sent Done Done false
25350 6778 Acked Write 8 Sent Done Done false
25351 6779 Acked Write 4 Sent Done Done false
25352 6780 Acked Write 9 Sent Done Done false
25353 6781 Acked Write 5 Sent Done Done false
25354 6782 Acked Flush 0 Sent Done Done false
25355 6783 Acked Read 1 Sent Done Done false
25356 6784 Acked Write 2 Sent Done Done false
25357 6785 Acked Read 5 Sent Done Done false
25358 6786 Acked Flush 0 Sent Done Done false
25359 6787 Acked Write 5 Sent Done Done false
25360 6788 Acked Flush 0 Sent Done Done false
25361 6789 Acked Write 6 Sent Done Done false
25362 6790 Acked Write 6 Sent Done Done false
25363 6791 Acked Write 4 Sent Done Done false
25364 6792 Acked Write 10 Sent Done Done false
25365 6793 Acked Write 10 Sent Done Done false
25366 6794 Acked Write 1 Sent Done Done false
25367 6795 Acked Read 1 Sent Done Done false
25368 6796 Acked Read 10 Sent Done Done false
25369 6797 Acked Write 6 Sent Done Done false
25370 6798 Acked Read 9 Sent Done Done false
25371 6799 Acked Read 9 Sent Done Done false
25372 6800 Acked Flush 0 Sent Done Done false
25373 6801 Acked Read 2 Sent Done Done false
25374 6802 Acked Read 3 Sent Done Done false
25375 6803 Acked Write 6 Sent Done Done false
25376 6804 Acked Read 9 Sent Done Done false
25377 6805 Acked Write 2 Sent Done Done false
25378 6806 Acked Write 3 Sent Done Done false
25379 6807 Acked Read 8 Sent Done Done false
25380 6808 Acked Read 6 Sent Done Done false
25381 6809 Acked Read 10 Sent Done Done false
25382 6810 Acked Write 3 Sent Done Done false
25383 6811 Acked Write 10 Sent Done Done false
25384 6812 Acked Read 3 Sent Done Done false
25385 6813 Acked Flush 0 Sent Done Done false
25386 6814 Acked Read 2 Sent Done Done false
25387 6815 Acked Read 8 Sent Done Done false
25388 6816 Acked Flush 0 Sent Done Done false
25389 6817 Acked Read 5 Sent Done Done false
25390 6818 Acked Flush 0 Sent Done Done false
25391 6819 Acked Write 6 Sent Done Done false
25392 6820 Acked Write 10 Sent Done Done false
25393 6821 Acked Read 3 Sent Done Done false
25394 6822 Acked Read 1 Sent Done Done false
25395 6823 Acked Flush 0 Sent Done Done false
25396 6824 Acked Write 5 Sent Done Done false
25397 6825 Acked Read 4 Sent Done Done false
25398 6826 Acked Read 4 Sent Done Done false
25399 6827 Acked Read 2 Sent Done Done false
25400 6828 Acked Read 5 Sent Done Done false
25401 6829 Acked Read 7 Sent Done Done false
25402 6830 Acked Read 1 Sent Done Done false
25403 6831 Acked Read 9 Sent Done Done false
25404 6832 Acked Read 8 Sent Done Done false
25405 6833 Acked Read 7 Sent Done Done false
25406 6834 Acked Read 8 Sent Done Done false
25407 6835 Acked Read 1 Sent Done Done false
25408 6836 Acked Read 10 Sent Done Done false
25409 6837 Acked Read 5 Sent Done Done false
25410 6838 Acked Flush 0 Sent Done Done false
25411 6839 Acked Write 9 Sent Done Done false
25412 6840 Acked Flush 0 Sent Done Done false
25413 6841 Acked Read 7 Sent Done Done false
25414 6842 Acked Write 8 Sent Done Done false
25415 6843 Acked Read 1 Sent Done Done false
25416 6844 Acked Write 5 Sent Done Done false
25417 6845 Acked Write 2 Sent Done Done false
25418 6846 Acked Write 9 Sent Done Done false
25419 6847 Acked Read 4 Sent Done Done false
25420 6848 Acked Write 6 Sent Done Done false
25421 6849 Acked Read 2 Sent Done Done false
25422 6850 Acked Read 5 Sent Done Done false
25423 6851 Acked Write 5 Sent Done Done false
25424 6852 Acked Read 6 Sent Done Done false
25425 6853 Acked Read 4 Sent Done Done false
25426 6854 Acked Write 6 Sent Done Done false
25427 6855 Acked Write 5 Sent Done Done false
25428 6856 Acked Read 5 Sent Done Done false
25429 6857 Acked Read 1 Sent Done Done false
25430 6858 Acked Read 2 Sent Done Done false
25431 6859 Acked Read 8 Sent Done Done false
25432 6860 Acked Read 2 Sent Done Done false
25433 6861 Acked Read 10 Sent Done Done false
25434 6862 Acked Read 7 Sent Done Done false
25435 6863 Acked Read 4 Sent Done Done false
25436 6864 Acked Write 9 Sent Done Done false
25437 6865 Acked Write 9 Sent Done Done false
25438 6866 Acked Read 8 Sent Done Done false
25439 6867 Acked Read 10 Sent Done Done false
25440 6868 Acked Read 7 Sent Done Done false
25441 6869 Acked Write 6 Sent Done Done false
25442 6870 Acked Read 4 Sent Done Done false
25443 6871 Acked Write 3 Sent Done Done false
25444 6872 Acked Read 2 Sent Done Done false
25445 6873 Acked Write 2 Sent Done Done false
25446 6874 Acked Read 1 Sent Done Done false
25447 6875 Acked Write 9 Sent Done Done false
25448 6876 Acked Write 5 Sent Done Done false
25449 6877 Acked Read 3 Sent Done Done false
25450 6878 Acked Flush 0 Sent Done Done false
25451 6879 Acked Write 1 Sent Done Done false
25452 6880 Acked Write 9 Sent Done Done false
25453 6881 Acked Write 6 Sent Done Done false
25454 6882 Acked Read 5 Sent Done Done false
25455 6883 Acked Read 4 Sent Done Done false
25456 6884 Acked Write 1 Sent Done Done false
25457 6885 Acked Write 3 Sent Done Done false
25458 6886 Acked Write 4 Sent Done Done false
25459 6887 Acked Write 2 Sent Done Done false
25460 6888 Acked Read 6 Sent Done Done false
25461 6889 Acked Read 7 Sent Done Done false
25462 6890 Acked Read 7 Sent Done Done false
25463 6891 Acked Flush 0 Sent Done Done false
25464 6892 Acked Write 8 Sent Done Done false
25465 6893 Acked Write 1 Sent Done Done false
25466 6894 Acked Read 5 Sent Done Done false
25467 6895 Acked Read 5 Sent Done Done false
25468 6896 Acked Write 6 Sent Done Done false
25469 6897 Acked Write 5 Sent Done Done false
25470 6898 Acked Read 1 Sent Done Done false
25471 6899 Acked Write 7 Sent Done Done false
25472 6900 Acked Flush 0 Sent Done Done false
25473 6901 Acked Read 6 Sent Done Done false
25474 6902 Acked Write 8 Sent Done Done false
25475 6903 Acked Read 6 Sent Done Done false
25476 6904 Acked Read 5 Sent Done Done false
25477 6905 Acked Write 5 Sent Done Done false
25478 6906 Acked Read 6 Sent Done Done false
25479 6907 Acked Read 1 Sent Done Done false
25480 6908 Acked Read 3 Sent Done Done false
25481 6909 Acked Read 10 Sent Done Done false
25482 6910 Acked Read 3 Sent Done Done false
25483 6911 Acked Write 6 Sent Done Done false
25484 6912 Acked Read 8 Sent Done Done false
25485 6913 Acked Read 5 Sent Done Done false
25486 6914 Acked Write 2 Sent Done Done false
25487 6915 Acked Read 10 Sent Done Done false
25488 6916 Acked Read 1 Sent Done Done false
25489 6917 Acked Flush 0 Sent Done Done false
25490 6918 Acked Read 9 Sent Done Done false
25491 6919 Acked Write 9 Sent Done Done false
25492 6920 Acked Write 10 Sent Done Done false
25493 6921 Acked Read 10 Sent Done Done false
25494 6922 Acked Write 9 Sent Done Done false
25495 6923 Acked Read 5 Sent Done Done false
25496 6924 Acked Write 2 Sent Done Done false
25497 6925 Acked Read 2 Sent Done Done false
25498 6926 Acked Read 6 Sent Done Done false
25499 6927 Acked Read 1 Sent Done Done false
25500 6928 Acked Read 2 Sent Done Done false
25501 6929 Acked Write 6 Sent Done Done false
25502 6930 Acked Write 9 Sent Done Done false
25503 6931 Acked Write 5 Sent Done Done false
25504 6932 Acked Read 1 Sent Done Done false
25505 6933 Acked Write 7 Sent Done Done false
25506 6934 Acked Flush 0 Sent Done Done false
25507 6935 Acked Flush 0 Sent Done Done false
25508 6936 Acked Read 7 Sent Done Done false
25509 6937 Acked Read 7 Sent Done Done false
25510 6938 Acked Write 7 Sent Done Done false
25511 6939 Acked Write 2 Sent Done Done false
25512 6940 Acked Flush 0 Sent Done Done false
25513 6941 Acked Write 5 Sent Done Done false
25514 6942 Acked Write 5 Sent Done Done false
25515 6943 Acked Write 2 Sent Done Done false
25516 6944 Acked Write 5 Sent Done Done false
25517 6945 Acked Write 9 Sent Done Done false
25518 6946 Acked Read 5 Sent Done Done false
25519 6947 Acked Read 10 Sent Done Done false
25520 6948 Acked Write 9 Sent Done Done false
25521 6949 Acked Read 4 Sent Done Done false
25522 6950 Acked Read 2 Sent Done Done false
25523 6951 Acked Write 9 Sent Done Done false
25524 6952 Acked Read 2 Sent Done Done false
25525 6953 Acked Read 5 Sent Done Done false
25526 6954 Acked Read 2 Sent Done Done false
25527 6955 Acked Read 6 Sent Done Done false
25528 6956 Acked Read 2 Sent Done Done false
25529 6957 Acked Write 2 Sent Done Done false
25530 6958 Acked Write 2 Sent Done Done false
25531 6959 Acked Read 1 Sent Done Done false
25532 6960 Acked Read 6 Sent Done Done false
25533 6961 Acked Read 3 Sent Done Done false
25534 6962 Acked Write 3 Sent Done Done false
25535 6963 Acked Write 6 Sent Done Done false
25536 6964 Acked Read 4 Sent Done Done false
25537 6965 Acked Read 4 Sent Done Done false
25538 6966 Acked Read 10 Sent Done Done false
25539 6967 Acked Read 10 Sent Done Done false
25540 6968 Acked Read 8 Sent Done Done false
25541 6969 Acked Read 5 Sent Done Done false
25542 6970 Acked Read 1 Sent Done Done false
25543 6971 Acked Write 4 Sent Done Done false
25544 6972 Acked Write 10 Sent Done Done false
25545 6973 Acked Read 10 Sent Done Done false
25546 6974 Acked Read 3 Sent Done Done false
25547 6975 Acked Write 6 Sent Done Done false
25548 6976 Acked Write 1 Sent Done Done false
25549 6977 Acked Write 4 Sent Done Done false
25550 6978 Acked Write 5 Sent Done Done false
25551 6979 Acked Read 9 Sent Done Done false
25552 6980 Acked Flush 0 Sent Done Done false
25553 6981 Acked Read 4 Sent Done Done false
25554 6982 Acked Read 4 Sent Done Done false
25555 6983 Acked Write 10 Sent Done Done false
25556 6984 Acked Read 7 Sent Done Done false
25557 6985 Acked Flush 0 Sent Done Done false
25558 6986 Acked Read 9 Sent Done Done false
25559 6987 Acked Write 3 Sent Done Done false
25560 6988 Acked Read 3 Sent Done Done false
25561 6989 Acked Write 7 Sent Done Done false
25562 6990 Acked Write 2 Sent Done Done false
25563 6991 Acked Read 10 Sent Done Done false
25564 6992 Acked Read 3 Sent Done Done false
25565 6993 Acked Read 5 Sent Done Done false
25566 6994 Acked Flush 0 Sent Done Done false
25567 6995 Acked Read 9 Sent Done Done false
25568 6996 Acked Read 6 Sent Done Done false
25569 6997 Acked Write 5 Sent Done Done false
25570 6998 Acked Write 6 Sent Done Done false
25571 6999 Acked Write 10 Sent Done Done false
25572 7000 Acked Write 5 Sent Done Done false
25573 7001 Acked Write 7 Sent Done Done false
25574 7002 Acked Flush 0 Sent Done Done false
25575 7003 Acked Read 8 Sent Done Done false
25576 7004 Acked Read 10 Sent Done Done false
25577 7005 Acked Flush 0 Sent Done Done false
25578 7006 Acked Read 10 Sent Done Done false
25579 7007 Acked Write 9 Sent Done Done false
25580 7008 Acked Read 6 Sent Done Done false
25581 7009 Acked Flush 0 Sent Done Done false
25582 7010 Acked Write 5 Sent Done Done false
25583 7011 Acked Read 6 Sent Done Done false
25584 7012 Acked Write 10 Sent Done Done false
25585 7013 Acked Write 7 Sent Done Done false
25586 7014 Acked Write 2 Sent Done Done false
25587 7015 Acked Read 7 Sent Done Done false
25588 7016 Acked Flush 0 Sent Done Done false
25589 7017 Acked Read 9 Sent Done Done false
25590 7018 Acked Write 9 Sent Done Done false
25591 7019 Acked Read 7 Sent Done Done false
25592 7020 Acked Flush 0 Sent Done Done false
25593 7021 Acked Flush 0 Sent Done Done false
25594 7022 Acked Read 8 Sent Done Done false
25595 7023 Acked Flush 0 Sent Done Done false
25596 7024 Acked Read 9 Sent Done Done false
25597 7025 Acked Read 6 Sent Done Done false
25598 7026 Acked Write 5 Sent Done Done false
25599 7027 Acked Write 5 Sent Done Done false
25600 7028 Acked Read 3 Sent Done Done false
25601 7029 Acked Flush 0 Sent Done Done false
25602 7030 Acked Write 10 Sent Done Done false
25603 7031 Acked Write 6 Sent Done Done false
25604 7032 Acked Flush 0 Sent Done Done false
25605 7033 Acked Write 8 Sent Done Done false
25606 7034 Acked Write 4 Sent Done Done false
25607 7035 Acked Write 8 Sent Done Done false
25608 7036 Acked Read 1 Sent Done Done false
25609 STATES DS:0 DS:1 DS:2 TOTAL
25610 Sent 301 0 0 301
25611 Done 0 301 301 602
25612 Skipped 0 0 0 0
25613 Error 0 0 0 0
25614 Last Flush: Some(JobId(6735)) Some(JobId(7032)) Some(JobId(7032))
25615 Downstairs last five completed: 6735 6734 6733 6732 6731
25616 Upstairs last five completed: 7036 7035 7034 7033 7032
25617 CLIENT: Up:0 ds:301 act:2
256182025-03-17T22:24:36.115ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
256192025-03-17T22:24:37.424ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
256202025-03-17T22:24:38.118ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
256212025-03-17T22:24:38.923ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
256222025-03-17T22:24:38.923ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:57328) = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
256232025-03-17T22:24:38.924ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
256242025-03-17T22:24:38.925ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
256252025-03-17T22:24:38.925ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
256262025-03-17T22:24:38.925ZINFOcrucible: send last flush ID to this DS: Some(JobId(6735)) = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
256272025-03-17T22:24:38.925ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
256282025-03-17T22:24:38.925ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(6735)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
256292025-03-17T22:24:38.925ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(6735)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
25630 ----------------------------------------------------------------
25631 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:4
25632 States: Active Active Active
25633 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25634 7033 Acked Write 8 Done Done Done true
25635 7034 Acked Write 4 Done Done Done true
25636 7035 Acked Write 8 Done Done Done true
25637 7036 Acked Read 1 Done Done Done true
25638 STATES DS:0 DS:1 DS:2 TOTAL
25639 Sent 0 0 0 0
25640 Done 4 4 4 12
25641 Skipped 0 0 0 0
25642 Error 0 0 0 0
25643 Last Flush: Some(JobId(7032)) Some(JobId(7032)) Some(JobId(7032))
25644 Downstairs last five completed: 7032 7031 7030 7029 7028
25645 Upstairs last five completed: 7036 7035 7034 7033 7032
25646 CLIENT: Up:0 ds:4 act:3
25647 ----------------------------------------------------------------
25648 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25649 Downstairs last five completed: 7037 7036 7035 7034 7033
25650 Upstairs last five completed: 7037 7036 7035 7034 7033
25651 CLIENT: Up:0 ds:0 act:3
25652 Replay: All jobs finished
25653 Main task has work to do, go find it
25654 got dsc Stop(1)
25655 stop 1
25656 Mar 17 22:24:43.177 INFO request completed, latency_us: 369[8820] Got stop action so:false kr:false
25657 , response_code: 204, uri: /stop/cid/1, method: POST, req_id: 5365a235-2260-4e73-8106-31d5615205fd, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
25658 Mar 17 22:24:43.178 INFO request completed, latency_us: 141, response_code: 200, uri: /state/cid/1, method: GET, req_id: a9368f8a-419e-4183-a732-68811c3aba92, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
256592025-03-17T22:24:43.178ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
256602025-03-17T22:24:43.178ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
256612025-03-17T22:24:43.178ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
256622025-03-17T22:24:43.178ZINFOcrucible: client task is exiting = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
25663 [8820] Exited with: ExitStatus(unix_wait_status(9))
25664 [8820][1] reports Exit
25665 Mar 17 22:24:47.180 INFO request completed, latency_us: 295, response_code: 200, uri: /state/cid/1, method: GET, req_id: 8431a41e-8195-4f03-a314-100046b37649, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
25666 001/300 Flush
25667 002/300 Write block 4327 len 36864 data: 6 6 5 5 5 6 5 5 5
25668 003/300 Read block 3662 len 36864
25669 004/300 Read block 4523 len 8192
25670 005/300 Read block 827 len 4096
25671 006/300 Read block 517 len 12288
25672 007/300 Read block 4778 len 8192
25673 008/300 Flush
25674 009/300 Write block 1905 len 20480 data: 6 7 7 8 7
25675 010/300 Write block 2759 len 12288 data: 2 2 2
25676 011/300 Read block 2039 len 8192
25677 012/300 Read block 2773 len 20480
25678 013/300 Read block 876 len 40960
25679 014/300 Flush
25680 015/300 Read block 991 len 32768
25681 016/300 Write block 3853 len 8192 data: 7 7
25682 017/300 Read block 1207 len 36864
25683 018/300 Flush
25684 019/300 Write block 3487 len 20480 data: 2 2 4 5 5
25685 020/300 Read block 222 len 16384
25686 021/300 Write block 540 len 20480 data: 5 3 3 2 2
25687 022/300 Read block 2411 len 40960
25688 023/300 Flush
25689 024/300 Flush
25690 025/300 Read block 4898 len 36864
25691 026/300 Write block 2873 len 40960 data: 4 4 5 5 4 5 5 5 6 5
25692 027/300 Read block 2706 len 12288
25693 028/300 Write block 2927 len 36864 data: 5 5 5 4 4 5 3 3 4
25694 029/300 Write block 774 len 32768 data: 5 4 3 3 5 5 6 8
25695 030/300 Flush
25696 031/300 Read block 4579 len 28672
25697 032/300 Read block 3935 len 28672
25698 033/300 Read block 4466 len 40960
25699 034/300 Read block 872 len 24576
25700 035/300 Write block 3751 len 20480 data: 5 7 7 7 7
25701 036/300 Write block 2038 len 32768 data: 6 5 6 6 5 4 4 5
25702 037/300 Read block 1795 len 24576
25703 038/300 Read block 2654 len 20480
25704 039/300 Write block 3603 len 28672 data: 5 5 6 4 4 3 4
25705 040/300 Write block 528 len 16384 data: 4 4 5 3
25706 041/300 Write block 1408 len 28672 data: 2 2 2 2 3 3 4
25707 042/300 Read block 1733 len 40960
25708 043/300 Read block 452 len 16384
25709 044/300 Read block 3263 len 32768
25710 045/300 Write block 3056 len 12288 data: 7 7 5
25711 046/300 Write block 4710 len 12288 data: 7 6 6
25712 047/300 Read block 2059 len 8192
25713 048/300 Write block 4909 len 12288 data: 4 5 6
25714 049/300 Read block 4969 len 12288
25715 050/300 Write block 200 len 16384 data: 4 6 6 5
25716 051/300 Flush
25717 052/300 Write block 2600 len 4096 data: 6
25718 053/300 Read block 1336 len 16384
25719 054/300 Write block 2305 len 24576 data: 3 4 4 4 4 5
25720 055/300 Write block 1111 len 12288 data: 7 6 5
25721 056/300 Write block 2952 len 20480 data: 6 6 5 6 5
25722 057/300 Read block 3913 len 40960
25723 058/300 Read block 748 len 8192
25724 059/300 Write block 1976 len 20480 data: 4 3 4 6 7
25725 060/300 Flush
25726 061/300 Write block 503 len 12288 data: 3 4 5
25727 062/300 Read block 4937 len 32768
25728 063/300 Write block 3792 len 16384 data: 4 3 2 3
25729 064/300 Read block 3612 len 24576
25730 065/300 Write block 3693 len 16384 data: 7 8 8 8
25731 066/300 Read block 4483 len 8192
25732 067/300 Write block 1077 len 28672 data: 8 7 7 6 6 6 6
25733 068/300 Read block 1316 len 40960
25734 069/300 Write block 4966 len 20480 data: 6 5 3 2 2
25735 070/300 Read block 4622 len 36864
25736 071/300 Write block 3539 len 4096 data: 5
25737 072/300 Read block 3829 len 24576
25738 073/300 Write block 1774 len 20480 data: 4 4 5 5 4
25739 074/300 Write block 309 len 12288 data: 4 4 5
25740 075/300 Read block 107 len 40960
25741 076/300 Write block 3088 len 16384 data: 5 5 4 4
25742 077/300 Read block 628 len 20480
25743 078/300 Write block 170 len 4096 data: 4
25744 079/300 Write block 3909 len 20480 data: 2 2 2 2 2
25745 080/300 Read block 4118 len 32768
25746 081/300 Read block 4497 len 20480
25747 082/300 Read block 3253 len 32768
25748 083/300 Flush
25749 084/300 Flush
25750 085/300 Write block 3248 len 40960 data: 9 7 8 7 5 5 5 4 5 4
25751 086/300 Read block 1055 len 24576
25752 087/300 Read block 17 len 28672
25753 088/300 Write block 1622 len 40960 data: 3 2 2 2 4 3 3 3 3 3
25754 089/300 Read block 4192 len 28672
25755 090/300 Write block 4071 len 16384 data: 3 3 4 6
25756 091/300 Read block 954 len 36864
25757 092/300 Flush
25758 093/300 Write block 4974 len 8192 data: 4 4
25759 094/300 Read block 55 len 36864
25760 095/300 Write block 3872 len 28672 data: 3 4 6 7 7 7 6
25761 096/300 Write block 3567 len 8192 data: 3 3
25762 097/300 Flush
25763 098/300 Read block 4699 len 12288
25764 099/300 Read block 3759 len 8192
25765 100/300 Write block 482 len 36864 data: 4 5 5 5 5 5 4 4 4
25766 101/300 Read block 244 len 32768
25767 102/300 Read block 1196 len 12288
25768 103/300 Read block 4839 len 36864
25769 104/300 Write block 2738 len 12288 data: 5 6 5
25770 105/300 Write block 4740 len 4096 data: 7
25771 106/300 Flush
25772 107/300 Read block 391 len 12288
25773 108/300 Read block 4824 len 28672
25774 109/300 Read block 4947 len 20480
25775 110/300 Read block 4221 len 40960
25776 111/300 Read block 1115 len 20480
25777 112/300 Read block 3336 len 36864
25778 113/300 Read block 3721 len 4096
25779 114/300 Read block 4548 len 28672
25780 115/300 Write block 3750 len 8192 data: 5 6
25781 116/300 Write block 4740 len 16384 data: 8 7 5 4
25782 117/300 Write block 2964 len 28672 data: 5 7 6 6 6 8 7
25783 118/300 Read block 835 len 8192
25784 119/300 Write block 784 len 36864 data: 7 7 7 6 6 6 5 5 4
25785 120/300 Write block 4955 len 16384 data: 4 4 4 4
25786 121/300 Read block 2424 len 16384
25787 122/300 Read block 1178 len 16384
25788 123/300 Read block 1294 len 4096
25789 124/300 Write block 2769 len 12288 data: 3 4 4
25790 125/300 Read block 324 len 24576
25791 126/300 Read block 3216 len 16384
25792 127/300 Read block 393 len 16384
25793 128/300 Write block 1702 len 40960 data: 3 3 3 4 5 5 5 7 11 8
25794 129/300 Write block 1232 len 20480 data: 3 6 7 5 5
25795 130/300 Write block 563 len 36864 data: 4 4 3 3 3 3 3 3 3
25796 131/300 Write block 4654 len 28672 data: 4 4 3 4 4 5 5
25797 132/300 Read block 3265 len 12288
25798 133/300 Read block 945 len 4096
25799 134/300 Read block 1712 len 4096
25800 135/300 Flush
25801 136/300 Write block 4086 len 36864 data: 3 3 2 2 3 4 4 5 5
25802 137/300 Read block 4120 len 32768
25803 138/300 Read block 2213 len 20480
25804 139/300 Read block 4454 len 16384
25805 140/300 Read block 839 len 4096
25806 141/300 Write block 3722 len 40960 data: 5 7 5 3 4 5 5 5 8 8
25807 142/300 Write block 3261 len 40960 data: 5 5 4 4 3 2 2 4 3 3
25808 143/300 Write block 2957 len 8192 data: 5 5
25809 144/300 Write block 971 len 4096 data: 4
25810 145/300 Read block 4952 len 32768
25811 146/300 Write block 1968 len 40960 data: 7 6 5 5 3 3 4 4 5 4
25812 147/300 Flush
25813 148/300 Write block 4659 len 28672 data: 6 6 5 6 6 6 6
25814 149/300 Read block 3256 len 32768
25815 150/300 Write block 2587 len 4096 data: 6
25816 151/300 Write block 450 len 20480 data: 9 8 6 6 4
25817 152/300 Write block 2911 len 28672 data: 4 4 4 5 3 3 3
25818 153/300 Write block 4692 len 32768 data: 5 5 4 3 3 5 5 5
25819 154/300 Read block 802 len 28672
25820 155/300 Read block 3555 len 40960
25821 156/300 Read block 1392 len 40960
25822 157/300 Read block 518 len 4096
25823 158/300 Read block 4103 len 24576
25824 159/300 Flush
25825 160/300 Write block 4067 len 12288 data: 2 2 3
25826 161/300 Flush
25827 162/300 Read block 2808 len 12288
25828 163/300 Write block 40 len 24576 data: 6 6 6 6 7 9
25829 164/300 Write block 1895 len 28672 data: 5 5 5 4 4 3 4
25830 165/300 Read block 504 len 16384
25831 166/300 Write block 1765 len 24576 data: 2 3 2 2 2 2
25832 167/300 Read block 1267 len 8192
25833 168/300 Write block 579 len 12288 data: 3 5 4
25834 169/300 Read block 2887 len 36864
25835 170/300 Read block 3484 len 36864
25836 171/300 Read block 1078 len 16384
25837 172/300 Write block 1877 len 20480 data: 7 8 10 8 7
25838 173/300 Write block 1665 len 4096 data: 5
25839 174/300 Read block 4567 len 36864
25840 175/300 Read block 2161 len 8192
25841 176/300 Write block 2906 len 16384 data: 5 5 7 6
25842 177/300 Read block 4624 len 40960
25843 178/300 Read block 3405 len 12288
25844 179/300 Read block 586 len 36864
25845 180/300 Write block 1484 len 12288 data: 4 4 2
25846 181/300 Read block 2402 len 20480
25847 182/300 Write block 2946 len 36864 data: 5 7 7 7 7 7 7 7 6
25848 183/300 Write block 4061 len 32768 data: 3 3 3 3 3 3 3 3
25849 184/300 Read block 4563 len 16384
25850 185/300 Read block 4788 len 8192
25851 186/300 Read block 236 len 28672
25852 187/300 Read block 4793 len 8192
25853 188/300 Write block 4120 len 8192 data: 5 5
25854 189/300 Write block 2154 len 12288 data: 3 3 4
25855 190/300 Flush
25856 191/300 Write block 3183 len 8192 data: 6 6
25857 192/300 Read block 1343 len 16384
25858 193/300 Write block 1240 len 40960 data: 5 6 6 4 3 3 3 3 4 5
25859 194/300 Write block 4307 len 20480 data: 3 3 4 5 6
25860 195/300 Flush
25861 196/300 Read block 2413 len 12288
25862 197/300 Read block 3889 len 28672
25863 198/300 Write block 214 len 28672 data: 5 2 3 4 3 3 5
25864 199/300 Read block 2842 len 40960
25865 200/300 Flush
25866 201/300 Read block 3111 len 16384
25867 202/300 Read block 522 len 20480
25868 203/300 Read block 846 len 16384
25869 204/300 Write block 4862 len 36864 data: 7 6 7 7 8 7 8 8 6
25870 205/300 Flush
25871 206/300 Read block 3437 len 32768
25872 207/300 Read block 1449 len 36864
25873 208/300 Read block 577 len 8192
25874 209/300 Write block 3954 len 12288 data: 4 5 6
25875 210/300 Read block 1750 len 28672
25876 211/300 Write block 3398 len 28672 data: 4 6 7 6 6 7 5
25877 212/300 Write block 4029 len 36864 data: 4 4 5 5 5 4 5 2 2
25878 213/300 Read block 4596 len 16384
25879 214/300 Read block 3009 len 32768
25880 215/300 Write block 1713 len 4096 data: 7
25881 216/300 Read block 4269 len 8192
25882 217/300 Read block 4985 len 16384
25883 218/300 Write block 3918 len 32768 data: 2 3 4 4 4 4 6 7
25884 219/300 Read block 543 len 12288
25885 220/300 Write block 385 len 20480 data: 3 3 3 5 6
25886 221/300 Read block 274 len 16384
25887 222/300 Write block 2077 len 28672 data: 2 2 5 5 4 5 5
25888 223/300 Read block 2993 len 28672
25889 224/300 Read block 3585 len 36864
25890 225/300 Read block 214 len 32768
25891 226/300 Write block 4485 len 40960 data: 5 4 4 4 4 5 4 4 4 6
25892 227/300 Read block 4861 len 12288
25893 228/300 Flush
25894 229/300 Write block 4171 len 28672 data: 5 5 4 3 5 7 7
25895 230/300 Write block 4426 len 16384 data: 4 4 5 5
25896 231/300 Flush
25897 232/300 Write block 501 len 8192 data: 4 3
25898 233/300 Read block 2940 len 24576
25899 234/300 Write block 4978 len 28672 data: 4 5 4 5 5 4 3
25900 235/300 Write block 4840 len 20480 data: 4 4 5 7 6
25901 236/300 Read block 1534 len 36864
25902 237/300 Write block 4974 len 36864 data: 5 5 4 4 5 6 5 6 6
25903 238/300 Flush
25904 239/300 Read block 3453 len 40960
25905 240/300 Write block 3421 len 40960 data: 5 4 4 3 5 7 6 8 8 8
25906 241/300 Write block 3544 len 40960 data: 6 7 8 7 6 7 7 7 5 4
25907 242/300 Read block 793 len 28672
25908 243/300 Read block 3564 len 8192
25909 244/300 Write block 658 len 28672 data: 6 7 8 8 8 6 6
25910 245/300 Read block 4978 len 24576
25911 246/300 Read block 2824 len 12288
25912 247/300 Write block 1745 len 32768 data: 4 4 4 4 4 4 6 6
25913 248/300 Read block 4876 len 24576
25914 249/300 Read block 158 len 12288
25915 250/300 Read block 4561 len 32768
25916 251/300 Write block 581 len 4096 data: 5
25917 252/300 Flush
25918 253/300 Write block 2329 len 16384 data: 3 3 3 2
25919 254/300 Read block 2584 len 4096
25920 255/300 Read block 261 len 40960
25921 256/300 Write block 2558 len 36864 data: 7 7 6 6 5 3 4 4 3
25922 257/300 Read block 1304 len 32768
25923 258/300 Flush
25924 259/300 Write block 3273 len 36864 data: 4 4 4 3 2 4 4 5 4
25925 260/300 Read block 3103 len 32768
25926 261/300 Flush
25927 262/300 Write block 246 len 20480 data: 5 3 2 3 4
25928 263/300 Read block 397 len 8192
25929 264/300 Read block 479 len 16384
25930 265/300 Read block 13 len 36864
25931 266/300 Write block 4658 len 36864 data: 5 7 7 6 7 7 7 7 7
25932 267/300 Write block 1410 len 32768 data: 3 3 4 4 5 4 4 4
25933 268/300 Read block 3041 len 24576
25934 269/300 Read block 3645 len 32768
25935 270/300 Flush
25936 271/300 Read block 3770 len 32768
25937 272/300 Read block 689 len 4096
25938 273/300 Write block 783 len 4096 data: 8
25939 274/300 Read block 1004 len 32768
25940 275/300 Write block 4174 len 8192 data: 4 6
25941 276/300 Read block 2246 len 36864
25942 277/300 Read block 11 len 32768
25943 278/300 Read block 4482 len 24576
25944 279/300 Read block 658 len 12288
25945 280/300 Write block 3352 len 8192 data: 4 4
25946 281/300 Read block 2879 len 24576
25947 282/300 Read block 353 len 24576
25948 283/300 Read block 2553 len 20480
25949 284/300 Write block 1494 len 16384 data: 6 4 3 3
25950 285/300 Write block 4865 len 16384 data: 8 9 8 9
25951 286/300 Write block 2735 len 8192 data: 6 6
25952 287/300 Read block 4069 len 36864
25953 288/300 Write block 253 len 40960 data: 4 3 2 2 3 2 2 3 3 6
25954 289/300 Read block 4203 len 36864
25955 290/300 Read block 4494 len 12288
25956 291/300 Write block 421 len 28672 data: 10 9 8 7 6 4 3
25957 292/300 Write block 96 len 16384 data: 4 5 6 6
25958 293/300 Write block 416 len 20480 data: 7 7 7 8 9
25959 294/300 Read block 4669 len 28672
25960 295/300 Flush
25961 296/300 Write block 3193 len 8192 data: 6 5
25962 297/300 Read block 2047 len 32768
25963 298/300 Read block 2252 len 4096
25964 299/300 Write block 3627 len 20480 data: 2 2 2 3 3
25965 300/300 Write block 4445 len 20480 data: 5 5 6 7 8
25966 301/300 Read block 4840 len 20480
25967 Main task has work to do, go find it
25968 got dsc Start(1)
25969 start 1
25970 Mar 17 22:24:47.429 INFO request completed, latency_us: 208, response_code: 204, uri: /start/cid/1, method: POST, req_id: fc370471-3f75-4f8f-a491-d0e3a7b46347, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
25971 Replay: started 1, returned:Ok(())
25972 ----------------------------------------------------------------
25973 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25974 States: Active Offline Active
25975 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25976 7038 Acked Flush 0 Done Sent Done false
25977 7039 Acked Write 9 Done Sent Done false
25978 7040 Acked Read 9 Done Sent Done false
25979 7041 Acked Read 2 Done Sent Done false
25980 7042 Acked Read 1 Done Sent Done false
25981 7043 Acked Read 3 Done Sent Done false
25982 7044 Acked Read 2 Done Sent Done false
25983 7045 Acked Flush 0 Done Sent Done false
25984 7046 Acked Write 5 Done Sent Done false
25985 7047 Acked Write 3 Done Sent Done false
25986 7048 Acked Read 2 Done Sent Done false
25987 7049 Acked Read 5 Done Sent Done false
25988 7050 Acked Read 10 Done Sent Done false
25989 7051 Acked Flush 0 Done Sent Done false
25990 7052 Acked Read 8 Done Sent Done false
25991 7053 Acked Write 2 Done Sent Done false
25992 7054 Acked Read 9 Done Sent Done false
25993 7055 Acked Flush 0 Done Sent Done false
25994 7056 Acked Write 5 Done Sent Done false
25995 7057 Acked Read 4 Done Sent Done false
25996 7058 Acked Write 5 Done Sent Done false
25997 7059 Acked Read 10 Done Sent Done false
25998 7060 Acked Flush 0 Done Sent Done false
25999 7061 Acked Flush 0 Done Sent Done false
26000 7062 Acked Read 9 Done Sent Done false
26001 7063 Acked Write 10 Done Sent Done false
26002 7064 Acked Read 3 Done Sent Done false
26003 7065 Acked Write 9 Done Sent Done false
26004 7066 Acked Write 8 Done Sent Done false
26005 7067 Acked Flush 0 Done Sent Done false
26006 7068 Acked Read 7 Done Sent Done false
26007 7069 Acked Read 7 Done Sent Done false
26008 7070 Acked Read 10 Done Sent Done false
26009 7071 Acked Read 6 Done Sent Done false
26010 7072 Acked Write 5 Done Sent Done false
26011 7073 Acked Write 8 Done Sent Done false
26012 7074 Acked Read 6 Done Sent Done false
26013 7075 Acked Read 5 Done Sent Done false
26014 7076 Acked Write 7 Done Sent Done false
26015 7077 Acked Write 4 Done Sent Done false
26016 7078 Acked Write 7 Done Sent Done false
26017 7079 Acked Read 10 Done Sent Done false
26018 7080 Acked Read 4 Done Sent Done false
26019 7081 Acked Read 8 Done Sent Done false
26020 7082 Acked Write 3 Done Sent Done false
26021 7083 Acked Write 3 Done Sent Done false
26022 7084 Acked Read 2 Done Sent Done false
26023 7085 Acked Write 3 Done Sent Done false
26024 7086 Acked Read 3 Done Sent Done false
26025 7087 Acked Write 4 Done Sent Done false
26026 7088 Acked Flush 0 Done Sent Done false
26027 7089 Acked Write 1 Done Sent Done false
26028 7090 Acked Read 4 Done Sent Done false
26029 7091 Acked Write 6 Done Sent Done false
26030 7092 Acked Write 3 Done Sent Done false
26031 7093 Acked Write 5 Done Sent Done false
26032 7094 Acked Read 10 Done Sent Done false
26033 7095 Acked Read 2 Done Sent Done false
26034 7096 Acked Write 5 Done Sent Done false
26035 7097 Acked Flush 0 Done Sent Done false
26036 7098 Acked Write 3 Done Sent Done false
26037 7099 Acked Read 8 Done Sent Done false
26038 7100 Acked Write 4 Done Sent Done false
26039 7101 Acked Read 6 Done Sent Done false
26040 7102 Acked Write 4 Done Sent Done false
26041 7103 Acked Read 2 Done Sent Done false
26042 7104 Acked Write 7 Done Sent Done false
26043 7105 Acked Read 10 Done Sent Done false
26044 7106 Acked Write 5 Done Sent Done false
26045 7107 Acked Read 9 Done Sent Done false
26046 7108 Acked Write 1 Done Sent Done false
26047 7109 Acked Read 6 Done Sent Done false
26048 7110 Acked Write 5 Done Sent Done false
26049 7111 Acked Write 3 Done Sent Done false
26050 7112 Acked Read 10 Done Sent Done false
26051 7113 Acked Write 4 Done Sent Done false
26052 7114 Acked Read 5 Done Sent Done false
26053 7115 Acked Write 1 Done Sent Done false
26054 7116 Acked Write 5 Done Sent Done false
26055 7117 Acked Read 8 Done Sent Done false
26056 7118 Acked Read 5 Done Sent Done false
26057 7119 Acked Read 8 Done Sent Done false
26058 7120 Acked Flush 0 Done Sent Done false
26059 7121 Acked Flush 0 Done Sent Done false
26060 7122 Acked Write 10 Done Sent Done false
26061 7123 Acked Read 6 Done Sent Done false
26062 7124 Acked Read 7 Done Sent Done false
26063 7125 Acked Write 10 Done Sent Done false
26064 7126 Acked Read 7 Done Sent Done false
26065 7127 Acked Write 4 Done Sent Done false
26066 7128 Acked Read 9 Done Sent Done false
26067 7129 Acked Flush 0 Done Sent Done false
26068 7130 Acked Write 2 Done Sent Done false
26069 7131 Acked Read 9 Done Sent Done false
26070 7132 Acked Write 7 Done Sent Done false
26071 7133 Acked Write 2 Done Sent Done false
26072 7134 Acked Flush 0 Done Sent Done false
26073 7135 Acked Read 3 Done Sent Done false
26074 7136 Acked Read 2 Done Sent Done false
26075 7137 Acked Write 9 Done Sent Done false
26076 7138 Acked Read 8 Done Sent Done false
26077 7139 Acked Read 3 Done Sent Done false
26078 7140 Acked Read 9 Done Sent Done false
26079 7141 Acked Write 3 Done Sent Done false
26080 7142 Acked Write 1 Done Sent Done false
26081 7143 Acked Flush 0 Done Sent Done false
26082 7144 Acked Read 3 Done Sent Done false
26083 7145 Acked Read 7 Done Sent Done false
26084 7146 Acked Read 5 Done Sent Done false
26085 7147 Acked Read 10 Done Sent Done false
26086 7148 Acked Read 5 Done Sent Done false
26087 7149 Acked Read 9 Done Sent Done false
26088 7150 Acked Read 1 Done Sent Done false
26089 7151 Acked Read 7 Done Sent Done false
26090 7152 Acked Write 2 Done Sent Done false
26091 7153 Acked Write 4 Done Sent Done false
26092 7154 Acked Write 7 Done Sent Done false
26093 7155 Acked Read 2 Done Sent Done false
26094 7156 Acked Write 9 Done Sent Done false
26095 7157 Acked Write 4 Done Sent Done false
26096 7158 Acked Read 4 Done Sent Done false
26097 7159 Acked Read 4 Done Sent Done false
26098 7160 Acked Read 1 Done Sent Done false
26099 7161 Acked Write 3 Done Sent Done false
26100 7162 Acked Read 6 Done Sent Done false
26101 7163 Acked Read 4 Done Sent Done false
26102 7164 Acked Read 4 Done Sent Done false
26103 7165 Acked Write 10 Done Sent Done false
26104 7166 Acked Write 5 Done Sent Done false
26105 7167 Acked Write 9 Done Sent Done false
26106 7168 Acked Write 7 Done Sent Done false
26107 7169 Acked Read 3 Done Sent Done false
26108 7170 Acked Read 1 Done Sent Done false
26109 7171 Acked Read 1 Done Sent Done false
26110 7172 Acked Flush 0 Done Sent Done false
26111 7173 Acked Write 9 Done Sent Done false
26112 7174 Acked Read 8 Done Sent Done false
26113 7175 Acked Read 5 Done Sent Done false
26114 7176 Acked Read 4 Done Sent Done false
26115 7177 Acked Read 1 Done Sent Done false
26116 7178 Acked Write 10 Done Sent Done false
26117 7179 Acked Write 10 Done Sent Done false
26118 7180 Acked Write 2 Done Sent Done false
26119 7181 Acked Write 1 Done Sent Done false
26120 7182 Acked Read 8 Done Sent Done false
26121 7183 Acked Write 10 Done Sent Done false
26122 7184 Acked Flush 0 Done Sent Done false
26123 7185 Acked Write 7 Done Sent Done false
26124 7186 Acked Read 8 Done Sent Done false
26125 7187 Acked Write 1 Done Sent Done false
26126 7188 Acked Write 5 Done Sent Done false
26127 7189 Acked Write 7 Done Sent Done false
26128 7190 Acked Write 8 Done Sent Done false
26129 7191 Acked Read 7 Done Sent Done false
26130 7192 Acked Read 10 Done Sent Done false
26131 7193 Acked Read 10 Done Sent Done false
26132 7194 Acked Read 1 Done Sent Done false
26133 7195 Acked Read 6 Done Sent Done false
26134 7196 Acked Flush 0 Done Sent Done false
26135 7197 Acked Write 3 Done Sent Done false
26136 7198 Acked Flush 0 Done Sent Done false
26137 7199 Acked Read 3 Done Sent Done false
26138 7200 Acked Write 6 Done Sent Done false
26139 7201 Acked Write 7 Done Sent Done false
26140 7202 Acked Read 4 Done Sent Done false
26141 7203 Acked Write 6 Done Sent Done false
26142 7204 Acked Read 2 Done Sent Done false
26143 7205 Acked Write 3 Done Sent Done false
26144 7206 Acked Read 9 Done Sent Done false
26145 7207 Acked Read 9 Done Sent Done false
26146 7208 Acked Read 4 Done Sent Done false
26147 7209 Acked Write 5 Done Sent Done false
26148 7210 Acked Write 1 Done Sent Done false
26149 7211 Acked Read 9 Done Sent Done false
26150 7212 Acked Read 2 Done Sent Done false
26151 7213 Acked Write 4 Done Sent Done false
26152 7214 Acked Read 10 Done Sent Done false
26153 7215 Acked Read 3 Done Sent Done false
26154 7216 Acked Read 9 Done Sent Done false
26155 7217 Acked Write 3 Done Sent Done false
26156 7218 Acked Read 5 Done Sent Done false
26157 7219 Acked Write 9 Done Sent Done false
26158 7220 Acked Write 8 Done Sent Done false
26159 7221 Acked Read 4 Done Sent Done false
26160 7222 Acked Read 2 Done Sent Done false
26161 7223 Acked Read 7 Done Sent Done false
26162 7224 Acked Read 2 Done Sent Done false
26163 7225 Acked Write 2 Done Sent Done false
26164 7226 Acked Write 3 Done Sent Done false
26165 7227 Acked Flush 0 Done Sent Done false
26166 7228 Acked Write 2 Done Sent Done false
26167 7229 Acked Read 4 Done Sent Done false
26168 7230 Acked Write 10 Done Sent Done false
26169 7231 Acked Write 5 Done Sent Done false
26170 7232 Acked Flush 0 Done Sent Done false
26171 7233 Acked Read 3 Done Sent Done false
26172 7234 Acked Read 7 Done Sent Done false
26173 7235 Acked Write 7 Done Sent Done false
26174 7236 Acked Read 10 Done Sent Done false
26175 7237 Acked Flush 0 Done Sent Done false
26176 7238 Acked Read 4 Done Sent Done false
26177 7239 Acked Read 5 Done Sent Done false
26178 7240 Acked Read 4 Done Sent Done false
26179 7241 Acked Write 9 Done Sent Done false
26180 7242 Acked Flush 0 Done Sent Done false
26181 7243 Acked Read 8 Done Sent Done false
26182 7244 Acked Read 9 Done Sent Done false
26183 7245 Acked Read 2 Done Sent Done false
26184 7246 Acked Write 3 Done Sent Done false
26185 7247 Acked Read 7 Done Sent Done false
26186 7248 Acked Write 7 Done Sent Done false
26187 7249 Acked Write 9 Done Sent Done false
26188 7250 Acked Read 4 Done Sent Done false
26189 7251 Acked Read 8 Done Sent Done false
26190 7252 Acked Write 1 Done Sent Done false
26191 7253 Acked Read 2 Done Sent Done false
26192 7254 Acked Read 4 Done Sent Done false
26193 7255 Acked Write 8 Done Sent Done false
26194 7256 Acked Read 3 Done Sent Done false
26195 7257 Acked Write 5 Done Sent Done false
26196 7258 Acked Read 4 Done Sent Done false
26197 7259 Acked Write 7 Done Sent Done false
26198 7260 Acked Read 7 Done Sent Done false
26199 7261 Acked Read 9 Done Sent Done false
26200 7262 Acked Read 8 Done Sent Done false
26201 7263 Acked Write 10 Done Sent Done false
26202 7264 Acked Read 3 Done Sent Done false
26203 7265 Acked Flush 0 Done Sent Done false
26204 7266 Acked Write 7 Done Sent Done false
26205 7267 Acked Write 4 Done Sent Done false
26206 7268 Acked Flush 0 Done Sent Done false
26207 7269 Acked Write 2 Done Sent Done false
26208 7270 Acked Read 6 Done Sent Done false
26209 7271 Acked Write 7 Done Sent Done false
26210 7272 Acked Write 5 Done Sent Done false
26211 7273 Acked Read 9 Done Sent Done false
26212 7274 Acked Write 9 Done Sent Done false
26213 7275 Acked Flush 0 Done Sent Done false
26214 7276 Acked Read 10 Done Sent Done false
26215 7277 Acked Write 10 Done Sent Done false
26216 7278 Acked Write 10 Done Sent Done false
26217 7279 Acked Read 7 Done Sent Done false
26218 7280 Acked Read 2 Done Sent Done false
26219 7281 Acked Write 7 Done Sent Done false
26220 7282 Acked Read 6 Done Sent Done false
26221 7283 Acked Read 3 Done Sent Done false
26222 7284 Acked Write 8 Done Sent Done false
26223 7285 Acked Read 6 Done Sent Done false
26224 7286 Acked Read 3 Done Sent Done false
26225 7287 Acked Read 8 Done Sent Done false
26226 7288 Acked Write 1 Done Sent Done false
26227 7289 Acked Flush 0 Done Sent Done false
26228 7290 Acked Write 4 Done Sent Done false
26229 7291 Acked Read 1 Done Sent Done false
26230 7292 Acked Read 10 Done Sent Done false
26231 7293 Acked Write 9 Done Sent Done false
26232 7294 Acked Read 8 Done Sent Done false
26233 7295 Acked Flush 0 Done Sent Done false
26234 7296 Acked Write 9 Done Sent Done false
26235 7297 Acked Read 8 Done Sent Done false
26236 7298 Acked Flush 0 Done Sent Done false
26237 7299 Acked Write 5 Done Sent Done false
26238 7300 Acked Read 2 Done Sent Done false
26239 7301 Acked Read 4 Done Sent Done false
26240 7302 Acked Read 9 Done Sent Done false
26241 7303 Acked Write 9 Done Sent Done false
26242 7304 Acked Write 8 Done Sent Done false
26243 7305 Acked Read 6 Done Sent Done false
26244 7306 Acked Read 8 Done Sent Done false
26245 7307 Acked Flush 0 Done Sent Done false
26246 7308 Acked Read 8 Done Sent Done false
26247 7309 Acked Read 1 Done Sent Done false
26248 7310 Acked Write 1 Done Sent Done false
26249 7311 Acked Read 8 Done Sent Done false
26250 7312 Acked Write 2 Done Sent Done false
26251 7313 Acked Read 9 Done Sent Done false
26252 7314 Acked Read 8 Done Sent Done false
26253 7315 Acked Read 6 Done Sent Done false
26254 7316 Acked Read 3 Done Sent Done false
26255 7317 Acked Write 2 Done Sent Done false
26256 7318 Acked Read 6 Done Sent Done false
26257 7319 Acked Read 6 Done Sent Done false
26258 7320 Acked Read 5 Done Sent Done false
26259 7321 Acked Write 4 Done Sent Done false
26260 7322 Acked Write 4 Done Sent Done false
26261 7323 Acked Write 2 Done Sent Done false
26262 7324 Acked Read 9 Done Sent Done false
26263 7325 Acked Write 10 Done Sent Done false
26264 7326 Acked Read 9 Done Sent Done false
26265 7327 Acked Read 3 Done Sent Done false
26266 7328 Acked Write 7 Done Sent Done false
26267 7329 Acked Write 4 Done Sent Done false
26268 7330 Acked Write 5 Done Sent Done false
26269 7331 Acked Read 7 Done Sent Done false
26270 7332 Acked Flush 0 Done Sent Done false
26271 7333 Acked Write 2 Done Sent Done false
26272 7334 Acked Read 8 Done Sent Done false
26273 7335 Acked Read 1 Done Sent Done false
26274 7336 Acked Write 5 Done Sent Done false
26275 7337 Acked Write 5 Done Sent Done false
26276 7338 Acked Read 5 Done Sent Done false
26277 STATES DS:0 DS:1 DS:2 TOTAL
26278 Sent 0 301 0 301
26279 Done 301 0 301 602
26280 Skipped 0 0 0 0
26281 Error 0 0 0 0
26282 Last Flush: Some(JobId(7332)) Some(JobId(7037)) Some(JobId(7332))
26283 Downstairs last five completed: 7037 7036 7035 7034 7033
26284 Upstairs last five completed: 7338 7337 7336 7335 7334
26285 CLIENT: Up:0 ds:301 act:2
262862025-03-17T22:24:48.137ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
262872025-03-17T22:24:49.428ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
262882025-03-17T22:24:50.140ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
262892025-03-17T22:24:51.141ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
26290 [8820] got start action
26291 [8820] I am going to restart
26292 Starting downstairs at port 8820
26293 Make output file at "/tmp/dsc/downstairs-8820.txt"
26294 [8820][1] reports Starting
26295 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1232)
26296 [8820][1] reports Running
26297 ----------------------------------------------------------------
26298 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26299 States: Active Offline Active
26300 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26301 7038 Acked Flush 0 Done Sent Done false
26302 7039 Acked Write 9 Done Sent Done false
26303 7040 Acked Read 9 Done Sent Done false
26304 7041 Acked Read 2 Done Sent Done false
26305 7042 Acked Read 1 Done Sent Done false
26306 7043 Acked Read 3 Done Sent Done false
26307 7044 Acked Read 2 Done Sent Done false
26308 7045 Acked Flush 0 Done Sent Done false
26309 7046 Acked Write 5 Done Sent Done false
26310 7047 Acked Write 3 Done Sent Done false
26311 7048 Acked Read 2 Done Sent Done false
26312 7049 Acked Read 5 Done Sent Done false
26313 7050 Acked Read 10 Done Sent Done false
26314 7051 Acked Flush 0 Done Sent Done false
26315 7052 Acked Read 8 Done Sent Done false
26316 7053 Acked Write 2 Done Sent Done false
26317 7054 Acked Read 9 Done Sent Done false
26318 7055 Acked Flush 0 Done Sent Done false
26319 7056 Acked Write 5 Done Sent Done false
26320 7057 Acked Read 4 Done Sent Done false
26321 7058 Acked Write 5 Done Sent Done false
26322 7059 Acked Read 10 Done Sent Done false
26323 7060 Acked Flush 0 Done Sent Done false
26324 7061 Acked Flush 0 Done Sent Done false
26325 7062 Acked Read 9 Done Sent Done false
26326 7063 Acked Write 10 Done Sent Done false
26327 7064 Acked Read 3 Done Sent Done false
26328 7065 Acked Write 9 Done Sent Done false
26329 7066 Acked Write 8 Done Sent Done false
26330 7067 Acked Flush 0 Done Sent Done false
26331 7068 Acked Read 7 Done Sent Done false
26332 7069 Acked Read 7 Done Sent Done false
26333 7070 Acked Read 10 Done Sent Done false
26334 7071 Acked Read 6 Done Sent Done false
26335 7072 Acked Write 5 Done Sent Done false
26336 7073 Acked Write 8 Done Sent Done false
26337 7074 Acked Read 6 Done Sent Done false
26338 7075 Acked Read 5 Done Sent Done false
26339 7076 Acked Write 7 Done Sent Done false
26340 7077 Acked Write 4 Done Sent Done false
26341 7078 Acked Write 7 Done Sent Done false
26342 7079 Acked Read 10 Done Sent Done false
26343 7080 Acked Read 4 Done Sent Done false
26344 7081 Acked Read 8 Done Sent Done false
26345 7082 Acked Write 3 Done Sent Done false
26346 7083 Acked Write 3 Done Sent Done false
26347 7084 Acked Read 2 Done Sent Done false
26348 7085 Acked Write 3 Done Sent Done false
26349 7086 Acked Read 3 Done Sent Done false
26350 7087 Acked Write 4 Done Sent Done false
26351 7088 Acked Flush 0 Done Sent Done false
26352 7089 Acked Write 1 Done Sent Done false
26353 7090 Acked Read 4 Done Sent Done false
26354 7091 Acked Write 6 Done Sent Done false
26355 7092 Acked Write 3 Done Sent Done false
26356 7093 Acked Write 5 Done Sent Done false
26357 7094 Acked Read 10 Done Sent Done false
26358 7095 Acked Read 2 Done Sent Done false
26359 7096 Acked Write 5 Done Sent Done false
26360 7097 Acked Flush 0 Done Sent Done false
26361 7098 Acked Write 3 Done Sent Done false
26362 7099 Acked Read 8 Done Sent Done false
26363 7100 Acked Write 4 Done Sent Done false
26364 7101 Acked Read 6 Done Sent Done false
26365 7102 Acked Write 4 Done Sent Done false
26366 7103 Acked Read 2 Done Sent Done false
26367 7104 Acked Write 7 Done Sent Done false
26368 7105 Acked Read 10 Done Sent Done false
26369 7106 Acked Write 5 Done Sent Done false
26370 7107 Acked Read 9 Done Sent Done false
26371 7108 Acked Write 1 Done Sent Done false
26372 7109 Acked Read 6 Done Sent Done false
26373 7110 Acked Write 5 Done Sent Done false
26374 7111 Acked Write 3 Done Sent Done false
26375 7112 Acked Read 10 Done Sent Done false
26376 7113 Acked Write 4 Done Sent Done false
26377 7114 Acked Read 5 Done Sent Done false
26378 7115 Acked Write 1 Done Sent Done false
26379 7116 Acked Write 5 Done Sent Done false
26380 7117 Acked Read 8 Done Sent Done false
26381 7118 Acked Read 5 Done Sent Done false
26382 7119 Acked Read 8 Done Sent Done false
26383 7120 Acked Flush 0 Done Sent Done false
26384 7121 Acked Flush 0 Done Sent Done false
26385 7122 Acked Write 10 Done Sent Done false
26386 7123 Acked Read 6 Done Sent Done false
26387 7124 Acked Read 7 Done Sent Done false
26388 7125 Acked Write 10 Done Sent Done false
26389 7126 Acked Read 7 Done Sent Done false
26390 7127 Acked Write 4 Done Sent Done false
26391 7128 Acked Read 9 Done Sent Done false
26392 7129 Acked Flush 0 Done Sent Done false
26393 7130 Acked Write 2 Done Sent Done false
26394 7131 Acked Read 9 Done Sent Done false
26395 7132 Acked Write 7 Done Sent Done false
26396 7133 Acked Write 2 Done Sent Done false
26397 7134 Acked Flush 0 Done Sent Done false
26398 7135 Acked Read 3 Done Sent Done false
26399 7136 Acked Read 2 Done Sent Done false
26400 7137 Acked Write 9 Done Sent Done false
26401 7138 Acked Read 8 Done Sent Done false
26402 7139 Acked Read 3 Done Sent Done false
26403 7140 Acked Read 9 Done Sent Done false
26404 7141 Acked Write 3 Done Sent Done false
26405 7142 Acked Write 1 Done Sent Done false
26406 7143 Acked Flush 0 Done Sent Done false
26407 7144 Acked Read 3 Done Sent Done false
26408 7145 Acked Read 7 Done Sent Done false
26409 7146 Acked Read 5 Done Sent Done false
26410 7147 Acked Read 10 Done Sent Done false
26411 7148 Acked Read 5 Done Sent Done false
26412 7149 Acked Read 9 Done Sent Done false
26413 7150 Acked Read 1 Done Sent Done false
26414 7151 Acked Read 7 Done Sent Done false
26415 7152 Acked Write 2 Done Sent Done false
26416 7153 Acked Write 4 Done Sent Done false
26417 7154 Acked Write 7 Done Sent Done false
26418 7155 Acked Read 2 Done Sent Done false
26419 7156 Acked Write 9 Done Sent Done false
26420 7157 Acked Write 4 Done Sent Done false
26421 7158 Acked Read 4 Done Sent Done false
26422 7159 Acked Read 4 Done Sent Done false
26423 7160 Acked Read 1 Done Sent Done false
26424 7161 Acked Write 3 Done Sent Done false
26425 7162 Acked Read 6 Done Sent Done false
26426 7163 Acked Read 4 Done Sent Done false
26427 7164 Acked Read 4 Done Sent Done false
26428 7165 Acked Write 10 Done Sent Done false
26429 7166 Acked Write 5 Done Sent Done false
26430 7167 Acked Write 9 Done Sent Done false
26431 7168 Acked Write 7 Done Sent Done false
26432 7169 Acked Read 3 Done Sent Done false
26433 7170 Acked Read 1 Done Sent Done false
26434 7171 Acked Read 1 Done Sent Done false
26435 7172 Acked Flush 0 Done Sent Done false
26436 7173 Acked Write 9 Done Sent Done false
26437 7174 Acked Read 8 Done Sent Done false
26438 7175 Acked Read 5 Done Sent Done false
26439 7176 Acked Read 4 Done Sent Done false
26440 7177 Acked Read 1 Done Sent Done false
26441 7178 Acked Write 10 Done Sent Done false
26442 7179 Acked Write 10 Done Sent Done false
26443 7180 Acked Write 2 Done Sent Done false
26444 7181 Acked Write 1 Done Sent Done false
26445 7182 Acked Read 8 Done Sent Done false
26446 7183 Acked Write 10 Done Sent Done false
26447 7184 Acked Flush 0 Done Sent Done false
26448 7185 Acked Write 7 Done Sent Done false
26449 7186 Acked Read 8 Done Sent Done false
26450 7187 Acked Write 1 Done Sent Done false
26451 7188 Acked Write 5 Done Sent Done false
26452 7189 Acked Write 7 Done Sent Done false
26453 7190 Acked Write 8 Done Sent Done false
26454 7191 Acked Read 7 Done Sent Done false
26455 7192 Acked Read 10 Done Sent Done false
26456 7193 Acked Read 10 Done Sent Done false
26457 7194 Acked Read 1 Done Sent Done false
26458 7195 Acked Read 6 Done Sent Done false
26459 7196 Acked Flush 0 Done Sent Done false
26460 7197 Acked Write 3 Done Sent Done false
26461 7198 Acked Flush 0 Done Sent Done false
26462 7199 Acked Read 3 Done Sent Done false
26463 7200 Acked Write 6 Done Sent Done false
26464 7201 Acked Write 7 Done Sent Done false
26465 7202 Acked Read 4 Done Sent Done false
26466 7203 Acked Write 6 Done Sent Done false
26467 7204 Acked Read 2 Done Sent Done false
26468 7205 Acked Write 3 Done Sent Done false
26469 7206 Acked Read 9 Done Sent Done false
26470 7207 Acked Read 9 Done Sent Done false
26471 7208 Acked Read 4 Done Sent Done false
26472 7209 Acked Write 5 Done Sent Done false
26473 7210 Acked Write 1 Done Sent Done false
26474 7211 Acked Read 9 Done Sent Done false
26475 7212 Acked Read 2 Done Sent Done false
26476 7213 Acked Write 4 Done Sent Done false
26477 7214 Acked Read 10 Done Sent Done false
26478 7215 Acked Read 3 Done Sent Done false
26479 7216 Acked Read 9 Done Sent Done false
26480 7217 Acked Write 3 Done Sent Done false
26481 7218 Acked Read 5 Done Sent Done false
26482 7219 Acked Write 9 Done Sent Done false
26483 7220 Acked Write 8 Done Sent Done false
26484 7221 Acked Read 4 Done Sent Done false
26485 7222 Acked Read 2 Done Sent Done false
26486 7223 Acked Read 7 Done Sent Done false
26487 7224 Acked Read 2 Done Sent Done false
26488 7225 Acked Write 2 Done Sent Done false
26489 7226 Acked Write 3 Done Sent Done false
26490 7227 Acked Flush 0 Done Sent Done false
26491 7228 Acked Write 2 Done Sent Done false
26492 7229 Acked Read 4 Done Sent Done false
26493 7230 Acked Write 10 Done Sent Done false
26494 7231 Acked Write 5 Done Sent Done false
26495 7232 Acked Flush 0 Done Sent Done false
26496 7233 Acked Read 3 Done Sent Done false
26497 7234 Acked Read 7 Done Sent Done false
26498 7235 Acked Write 7 Done Sent Done false
26499 7236 Acked Read 10 Done Sent Done false
26500 7237 Acked Flush 0 Done Sent Done false
26501 7238 Acked Read 4 Done Sent Done false
26502 7239 Acked Read 5 Done Sent Done false
26503 7240 Acked Read 4 Done Sent Done false
26504 7241 Acked Write 9 Done Sent Done false
26505 7242 Acked Flush 0 Done Sent Done false
26506 7243 Acked Read 8 Done Sent Done false
26507 7244 Acked Read 9 Done Sent Done false
26508 7245 Acked Read 2 Done Sent Done false
26509 7246 Acked Write 3 Done Sent Done false
26510 7247 Acked Read 7 Done Sent Done false
26511 7248 Acked Write 7 Done Sent Done false
26512 7249 Acked Write 9 Done Sent Done false
26513 7250 Acked Read 4 Done Sent Done false
26514 7251 Acked Read 8 Done Sent Done false
26515 7252 Acked Write 1 Done Sent Done false
26516 7253 Acked Read 2 Done Sent Done false
26517 7254 Acked Read 4 Done Sent Done false
26518 7255 Acked Write 8 Done Sent Done false
26519 7256 Acked Read 3 Done Sent Done false
26520 7257 Acked Write 5 Done Sent Done false
26521 7258 Acked Read 4 Done Sent Done false
26522 7259 Acked Write 7 Done Sent Done false
26523 7260 Acked Read 7 Done Sent Done false
26524 7261 Acked Read 9 Done Sent Done false
26525 7262 Acked Read 8 Done Sent Done false
26526 7263 Acked Write 10 Done Sent Done false
26527 7264 Acked Read 3 Done Sent Done false
26528 7265 Acked Flush 0 Done Sent Done false
26529 7266 Acked Write 7 Done Sent Done false
26530 7267 Acked Write 4 Done Sent Done false
26531 7268 Acked Flush 0 Done Sent Done false
26532 7269 Acked Write 2 Done Sent Done false
26533 7270 Acked Read 6 Done Sent Done false
26534 7271 Acked Write 7 Done Sent Done false
26535 7272 Acked Write 5 Done Sent Done false
26536 7273 Acked Read 9 Done Sent Done false
26537 7274 Acked Write 9 Done Sent Done false
26538 7275 Acked Flush 0 Done Sent Done false
26539 7276 Acked Read 10 Done Sent Done false
26540 7277 Acked Write 10 Done Sent Done false
26541 7278 Acked Write 10 Done Sent Done false
26542 7279 Acked Read 7 Done Sent Done false
26543 7280 Acked Read 2 Done Sent Done false
26544 7281 Acked Write 7 Done Sent Done false
26545 7282 Acked Read 6 Done Sent Done false
26546 7283 Acked Read 3 Done Sent Done false
26547 7284 Acked Write 8 Done Sent Done false
26548 7285 Acked Read 6 Done Sent Done false
26549 7286 Acked Read 3 Done Sent Done false
26550 7287 Acked Read 8 Done Sent Done false
26551 7288 Acked Write 1 Done Sent Done false
26552 7289 Acked Flush 0 Done Sent Done false
26553 7290 Acked Write 4 Done Sent Done false
26554 7291 Acked Read 1 Done Sent Done false
26555 7292 Acked Read 10 Done Sent Done false
26556 7293 Acked Write 9 Done Sent Done false
26557 7294 Acked Read 8 Done Sent Done false
26558 7295 Acked Flush 0 Done Sent Done false
26559 7296 Acked Write 9 Done Sent Done false
26560 7297 Acked Read 8 Done Sent Done false
26561 7298 Acked Flush 0 Done Sent Done false
26562 7299 Acked Write 5 Done Sent Done false
26563 7300 Acked Read 2 Done Sent Done false
26564 7301 Acked Read 4 Done Sent Done false
26565 7302 Acked Read 9 Done Sent Done false
26566 7303 Acked Write 9 Done Sent Done false
26567 7304 Acked Write 8 Done Sent Done false
26568 7305 Acked Read 6 Done Sent Done false
26569 7306 Acked Read 8 Done Sent Done false
26570 7307 Acked Flush 0 Done Sent Done false
26571 7308 Acked Read 8 Done Sent Done false
26572 7309 Acked Read 1 Done Sent Done false
26573 7310 Acked Write 1 Done Sent Done false
26574 7311 Acked Read 8 Done Sent Done false
26575 7312 Acked Write 2 Done Sent Done false
26576 7313 Acked Read 9 Done Sent Done false
26577 7314 Acked Read 8 Done Sent Done false
26578 7315 Acked Read 6 Done Sent Done false
26579 7316 Acked Read 3 Done Sent Done false
26580 7317 Acked Write 2 Done Sent Done false
26581 7318 Acked Read 6 Done Sent Done false
26582 7319 Acked Read 6 Done Sent Done false
26583 7320 Acked Read 5 Done Sent Done false
26584 7321 Acked Write 4 Done Sent Done false
26585 7322 Acked Write 4 Done Sent Done false
26586 7323 Acked Write 2 Done Sent Done false
26587 7324 Acked Read 9 Done Sent Done false
26588 7325 Acked Write 10 Done Sent Done false
26589 7326 Acked Read 9 Done Sent Done false
26590 7327 Acked Read 3 Done Sent Done false
26591 7328 Acked Write 7 Done Sent Done false
26592 7329 Acked Write 4 Done Sent Done false
26593 7330 Acked Write 5 Done Sent Done false
26594 7331 Acked Read 7 Done Sent Done false
26595 7332 Acked Flush 0 Done Sent Done false
26596 7333 Acked Write 2 Done Sent Done false
26597 7334 Acked Read 8 Done Sent Done false
26598 7335 Acked Read 1 Done Sent Done false
26599 7336 Acked Write 5 Done Sent Done false
26600 7337 Acked Write 5 Done Sent Done false
26601 7338 Acked Read 5 Done Sent Done false
26602 STATES DS:0 DS:1 DS:2 TOTAL
26603 Sent 0 301 0 301
26604 Done 301 0 301 602
26605 Skipped 0 0 0 0
26606 Error 0 0 0 0
26607 Last Flush: Some(JobId(7332)) Some(JobId(7037)) Some(JobId(7332))
26608 Downstairs last five completed: 7037 7036 7035 7034 7033
26609 Upstairs last five completed: 7338 7337 7336 7335 7334
26610 CLIENT: Up:0 ds:301 act:2
266112025-03-17T22:24:52.174ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
266122025-03-17T22:24:53.146ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
266132025-03-17T22:24:53.179ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
266142025-03-17T22:24:53.179ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:42443) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
266152025-03-17T22:24:53.180ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
266162025-03-17T22:24:53.181ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
266172025-03-17T22:24:53.181ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
266182025-03-17T22:24:53.181ZINFOcrucible: send last flush ID to this DS: Some(JobId(7037)) = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
266192025-03-17T22:24:53.181ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
266202025-03-17T22:24:53.181ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(7037)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
266212025-03-17T22:24:53.181ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(7037)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
26622 ----------------------------------------------------------------
26623 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
26624 Downstairs last five completed: 7339 7338 7337 7336 7335
26625 Upstairs last five completed: 7339 7338 7337 7336 7335
26626 CLIENT: Up:0 ds:0 act:3
26627 Replay: All jobs finished
26628 Main task has work to do, go find it
26629 got dsc Stop(1)
26630 stop 1
26631 Mar 17 22:24:55.435 INFO request completed, latency_us: [8820] Got stop action so:false kr:false
26632 348, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 81d8292b-009e-41f9-b778-737afce81391, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
266332025-03-17T22:24:55.435ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
26634 Mar 17 22:24:55.435 INFO request completed, latency_us: 113, response_code: 200, uri: /state/cid/1{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:24:55.435887721Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
26635 , {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:24:55.435926377Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
26636 method: GET, req_id: ccad80db-c811-4732-8ad8-7a6dc9faf893, remote_addr: 127.0.0.1:58317, local_addr: 127.0.{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:24:55.435954483Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
26637 0.1:9998
26638 [8820] Exited with: ExitStatus(unix_wait_status(9))
26639 [8820][1] reports Exit
26640 Mar 17 22:24:59.437 INFO request completed, latency_us: 210, response_code: 200, uri: /state/cid/1, method: GET, req_id: a14f7a5c-6f79-4040-8ab7-fb68590c8d3d, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
26641 001/300 Read block 4121 len 28672
26642 002/300 Write block 3134 len 28672 data: 4 5 7 7 8 7 7
26643 003/300 Flush
26644 004/300 Read block 2794 len 12288
26645 005/300 Read block 1405 len 8192
26646 006/300 Flush
26647 007/300 Write block 4197 len 32768 data: 8 7 6 4 4 4 3 4
26648 008/300 Read block 3167 len 20480
26649 009/300 Flush
26650 010/300 Write block 1427 len 16384 data: 6 5 4 3
26651 011/300 Read block 2335 len 24576
26652 012/300 Read block 989 len 28672
26653 013/300 Flush
26654 014/300 Read block 4939 len 28672
26655 015/300 Write block 4149 len 20480 data: 5 4 5 5 5
26656 016/300 Write block 1579 len 28672 data: 5 4 4 6 8 8 8
26657 017/300 Write block 4608 len 40960 data: 2 3 3 3 3 3 3 3 2 3
26658 018/300 Read block 2294 len 4096
26659 019/300 Read block 2166 len 4096
26660 020/300 Read block 2375 len 32768
26661 021/300 Read block 1302 len 16384
26662 022/300 Write block 274 len 8192 data: 3 6
26663 023/300 Read block 4305 len 32768
26664 024/300 Read block 4417 len 24576
26665 025/300 Read block 4473 len 24576
26666 026/300 Write block 1844 len 8192 data: 2 2
26667 027/300 Read block 1806 len 4096
26668 028/300 Read block 850 len 24576
26669 029/300 Read block 3591 len 24576
26670 030/300 Write block 2802 len 12288 data: 3 3 2
26671 031/300 Write block 1251 len 12288 data: 8 8 7
26672 032/300 Read block 860 len 4096
26673 033/300 Read block 1363 len 20480
26674 034/300 Flush
26675 035/300 Read block 2844 len 4096
26676 036/300 Write block 1183 len 16384 data: 4 4 5 5
26677 037/300 Read block 1135 len 20480
26678 038/300 Read block 287 len 32768
26679 039/300 Write block 510 len 12288 data: 5 5 5
26680 040/300 Read block 2863 len 24576
26681 041/300 Flush
26682 042/300 Read block 4547 len 12288
26683 043/300 Read block 24 len 4096
26684 044/300 Flush
26685 045/300 Read block 2820 len 8192
26686 046/300 Read block 1769 len 12288
26687 047/300 Flush
26688 048/300 Read block 461 len 28672
26689 049/300 Read block 978 len 12288
26690 050/300 Read block 4699 len 36864
26691 051/300 Write block 1036 len 4096 data: 4
26692 052/300 Read block 2625 len 4096
26693 053/300 Flush
26694 054/300 Read block 3110 len 20480
26695 055/300 Read block 2372 len 4096
26696 056/300 Write block 754 len 24576 data: 5 5 6 6 6 7
26697 057/300 Read block 836 len 20480
26698 058/300 Write block 3631 len 36864 data: 4 4 4 4 5 6 7 7 7
26699 059/300 Write block 2151 len 12288 data: 3 3 3
26700 060/300 Write block 3455 len 20480 data: 3 2 2 2 2
26701 061/300 Write block 2385 len 4096 data: 7
26702 062/300 Write block 4983 len 24576 data: 5 4 4 4 5 5
26703 063/300 Read block 3398 len 4096
26704 064/300 Write block 3611 len 28672 data: 5 5 4 4 3 3 3
26705 065/300 Read block 2609 len 32768
26706 066/300 Write block 2164 len 36864 data: 4 5 5 5 6 6 7 5 6
26707 067/300 Write block 49 len 8192 data: 4 4
26708 068/300 Write block 1452 len 40960 data: 4 3 3 3 3 3 5 4 3 2
26709 069/300 Read block 79 len 20480
26710 070/300 Read block 3793 len 4096
26711 071/300 Read block 2881 len 40960
26712 072/300 Write block 4977 len 20480 data: 5 6 7 6 7
26713 073/300 Write block 3597 len 32768 data: 5 5 5 6 6 7 6 6
26714 074/300 Write block 2493 len 32768 data: 5 5 4 4 4 5 6 5
26715 075/300 Write block 4340 len 4096 data: 4
26716 076/300 Write block 2377 len 32768 data: 4 4 6 6 6 7 8 8
26717 077/300 Read block 270 len 8192
26718 078/300 Write block 2530 len 24576 data: 8 8 7 6 5 5
26719 079/300 Read block 759 len 16384
26720 080/300 Flush
26721 081/300 Read block 3907 len 12288
26722 082/300 Write block 1679 len 28672 data: 3 3 4 4 4 4 4
26723 083/300 Read block 123 len 8192
26724 084/300 Read block 1636 len 28672
26725 085/300 Read block 1251 len 16384
26726 086/300 Read block 3930 len 8192
26727 087/300 Write block 2476 len 28672 data: 5 5 4 3 2 3 5
26728 088/300 Write block 830 len 4096 data: 3
26729 089/300 Write block 2355 len 24576 data: 4 4 4 5 6 4
26730 090/300 Read block 4781 len 32768
26731 091/300 Write block 1115 len 32768 data: 5 5 5 5 5 3 3 3
26732 092/300 Flush
26733 093/300 Read block 1315 len 4096
26734 094/300 Read block 4094 len 36864
26735 095/300 Read block 3304 len 32768
26736 096/300 Read block 2082 len 16384
26737 097/300 Write block 4674 len 20480 data: 4 3 3 5 5
26738 098/300 Flush
26739 099/300 Write block 2492 len 36864 data: 5 6 6 5 5 5 6 7 6
26740 100/300 Write block 977 len 40960 data: 6 5 4 4 5 5 4 4 4 4
26741 101/300 Read block 735 len 40960
26742 102/300 Write block 4340 len 12288 data: 5 5 4
26743 103/300 Flush
26744 104/300 Read block 571 len 20480
26745 105/300 Write block 3570 len 36864 data: 3 3 3 3 5 7 7 6 8
26746 106/300 Read block 2346 len 8192
26747 107/300 Write block 2741 len 24576 data: 4 8 6 6 5 7
26748 108/300 Read block 3009 len 16384
26749 109/300 Read block 85 len 28672
26750 110/300 Write block 4030 len 28672 data: 5 6 6 6 5 6 3
26751 111/300 Write block 2441 len 4096 data: 5
26752 112/300 Write block 612 len 28672 data: 6 6 6 5 5 4 4
26753 113/300 Write block 1533 len 8192 data: 4 5
26754 114/300 Read block 3387 len 36864
26755 115/300 Write block 2619 len 20480 data: 4 5 5 7 7
26756 116/300 Read block 226 len 24576
26757 117/300 Read block 1191 len 4096
26758 118/300 Write block 4472 len 16384 data: 4 5 4 3
26759 119/300 Read block 4340 len 28672
26760 120/300 Write block 400 len 12288 data: 4 3 3
26761 121/300 Write block 4982 len 4096 data: 7
26762 122/300 Read block 2411 len 4096
26763 123/300 Read block 2642 len 28672
26764 124/300 Read block 2103 len 36864
26765 125/300 Read block 1148 len 36864
26766 126/300 Write block 3651 len 20480 data: 4 5 7 7 6
26767 127/300 Write block 39 len 24576 data: 4 7 7 7 7 8
26768 128/300 Read block 3870 len 40960
26769 129/300 Flush
26770 130/300 Flush
26771 131/300 Read block 138 len 16384
26772 132/300 Read block 4390 len 20480
26773 133/300 Write block 344 len 28672 data: 5 4 7 7 6 5 4
26774 134/300 Read block 651 len 20480
26775 135/300 Read block 145 len 40960
26776 136/300 Read block 774 len 16384
26777 137/300 Read block 89 len 40960
26778 138/300 Write block 1194 len 24576 data: 3 4 4 4 3 3
26779 139/300 Flush
26780 140/300 Write block 2361 len 28672 data: 4 2 2 3 3 4 4
26781 141/300 Read block 4445 len 28672
26782 142/300 Read block 1627 len 16384
26783 143/300 Write block 3589 len 12288 data: 7 6 6
26784 144/300 Read block 1608 len 16384
26785 145/300 Write block 44 len 40960 data: 9 10 6 6 5 5 5 4 4 3
26786 146/300 Write block 2509 len 8192 data: 4 4
26787 147/300 Read block 2280 len 8192
26788 148/300 Write block 3564 len 16384 data: 4 4 5 4
26789 149/300 Read block 4224 len 4096
26790 150/300 Read block 4361 len 28672
26791 151/300 Read block 467 len 32768
26792 152/300 Flush
26793 153/300 Read block 4801 len 40960
26794 154/300 Write block 3267 len 16384 data: 3 5 4 4
26795 155/300 Read block 4686 len 28672
26796 156/300 Write block 951 len 40960 data: 5 4 4 4 3 5 6 6 6 7
26797 157/300 Write block 3688 len 40960 data: 5 5 5 5 6 8 9 9 9 6
26798 158/300 Read block 4679 len 28672
26799 159/300 Write block 1889 len 24576 data: 3 2 3 4 3 4
26800 160/300 Write block 1726 len 4096 data: 8
26801 161/300 Write block 3043 len 12288 data: 8 8 8
26802 162/300 Read block 2948 len 36864
26803 163/300 Read block 2995 len 32768
26804 164/300 Read block 854 len 16384
26805 165/300 Read block 4145 len 16384
26806 166/300 Read block 962 len 24576
26807 167/300 Write block 2039 len 4096 data: 6
26808 168/300 Write block 2680 len 16384 data: 3 4 6 6
26809 169/300 Read block 304 len 40960
26810 170/300 Flush
26811 171/300 Write block 877 len 12288 data: 4 2 2
26812 172/300 Write block 3316 len 32768 data: 6 4 5 5 5 7 6 6
26813 173/300 Flush
26814 174/300 Read block 1359 len 16384
26815 175/300 Write block 1418 len 12288 data: 3 3 3
26816 176/300 Read block 4309 len 16384
26817 177/300 Write block 2782 len 16384 data: 4 5 5 5
26818 178/300 Write block 409 len 4096 data: 4
26819 179/300 Read block 2805 len 24576
26820 180/300 Write block 727 len 12288 data: 4 5 6
26821 181/300 Read block 3739 len 40960
26822 182/300 Write block 2757 len 36864 data: 2 2 3 3 3 2 3 4 5
26823 183/300 Write block 1179 len 4096 data: 3
26824 184/300 Read block 3774 len 4096
26825 185/300 Write block 2915 len 12288 data: 4 4 4
26826 186/300 Flush
26827 187/300 Write block 3186 len 12288 data: 7 7 6
26828 188/300 Flush
26829 189/300 Read block 3233 len 40960
26830 190/300 Flush
26831 191/300 Flush
26832 192/300 Flush
26833 193/300 Write block 3860 len 16384 data: 3 4 3 3
26834 194/300 Read block 2959 len 8192
26835 195/300 Read block 4693 len 8192
26836 196/300 Write block 522 len 20480 data: 5 4 3 4 4
26837 197/300 Write block 2015 len 4096 data: 6
26838 198/300 Write block 2526 len 32768 data: 5 7 7 7 9 9 8 7
26839 199/300 Write block 2705 len 4096 data: 8
26840 200/300 Write block 4115 len 16384 data: 3 5 6 5
26841 201/300 Write block 1796 len 12288 data: 5 4 4
26842 202/300 Read block 3544 len 16384
26843 203/300 Flush
26844 204/300 Write block 4321 len 36864 data: 5 5 5 5 6 7 7 7 6
26845 205/300 Read block 1789 len 20480
26846 206/300 Write block 1275 len 40960 data: 5 4 2 3 4 5 7 5 7 6
26847 207/300 Write block 3981 len 8192 data: 4 4
26848 208/300 Write block 44 len 20480 data: 10 11 7 7 6
26849 209/300 Write block 4373 len 28672 data: 3 4 5 7 5 6 6
26850 210/300 Write block 4665 len 28672 data: 8 8 7 6 6 6 6
26851 211/300 Write block 2631 len 36864 data: 3 2 2 2 3 4 6 5 6
26852 212/300 Write block 1549 len 36864 data: 8 8 8 8 9 9 8 5 4
26853 213/300 Read block 2845 len 12288
26854 214/300 Write block 411 len 28672 data: 4 5 7 7 7 8 8
26855 215/300 Read block 4992 len 8192
26856 216/300 Write block 4451 len 36864 data: 7 6 5 7 6 4 4 3 3
26857 217/300 Read block 4846 len 20480
26858 218/300 Read block 2207 len 36864
26859 219/300 Write block 4686 len 4096 data: 2
26860 220/300 Read block 298 len 16384
26861 221/300 Read block 1645 len 36864
26862 222/300 Read block 1993 len 24576
26863 223/300 Read block 1876 len 20480
26864 224/300 Write block 3501 len 12288 data: 5 6 7
26865 225/300 Write block 2362 len 4096 data: 3
26866 226/300 Write block 4563 len 12288 data: 7 6 5
26867 227/300 Read block 1425 len 40960
26868 228/300 Read block 3117 len 12288
26869 229/300 Flush
26870 230/300 Read block 4861 len 36864
26871 231/300 Read block 1982 len 4096
26872 232/300 Write block 1835 len 12288 data: 3 4 7
26873 233/300 Read block 4726 len 40960
26874 234/300 Read block 3135 len 12288
26875 235/300 Flush
26876 236/300 Read block 3226 len 20480
26877 237/300 Read block 2853 len 20480
26878 238/300 Write block 4789 len 20480 data: 3 2 2 2 3
26879 239/300 Read block 3772 len 28672
26880 240/300 Write block 3105 len 28672 data: 4 4 5 4 5 6 7
26881 241/300 Read block 4267 len 32768
26882 242/300 Write block 4795 len 36864 data: 3 3 4 4 4 3 5 6 7
26883 243/300 Read block 613 len 24576
26884 244/300 Write block 873 len 8192 data: 4 5
26885 245/300 Read block 4514 len 16384
26886 246/300 Write block 3915 len 20480 data: 4 3 3 3 4
26887 247/300 Write block 852 len 36864 data: 5 4 4 4 5 6 5 5 5
26888 248/300 Flush
26889 249/300 Read block 704 len 24576
26890 250/300 Read block 2108 len 8192
26891 251/300 Write block 1963 len 36864 data: 7 6 7 8 7 8 7 6 6
26892 252/300 Read block 1751 len 24576
26893 253/300 Read block 1523 len 32768
26894 254/300 Read block 2108 len 20480
26895 255/300 Read block 4842 len 28672
26896 256/300 Write block 617 len 28672 data: 5 5 3 4 3 2 2
26897 257/300 Read block 2991 len 12288
26898 258/300 Read block 4955 len 32768
26899 259/300 Read block 519 len 32768
26900 260/300 Read block 1490 len 20480
26901 261/300 Read block 2955 len 8192
26902 262/300 Read block 1328 len 24576
26903 263/300 Read block 1660 len 16384
26904 264/300 Write block 3652 len 32768 data: 6 8 8 7 5 5 3 4
26905 265/300 Write block 2939 len 4096 data: 6
26906 266/300 Write block 974 len 4096 data: 5
26907 267/300 Write block 1575 len 8192 data: 3 2
26908 268/300 Read block 350 len 4096
26909 269/300 Read block 3567 len 36864
26910 270/300 Flush
26911 271/300 Read block 3754 len 32768
26912 272/300 Read block 2562 len 32768
26913 273/300 Read block 719 len 36864
26914 274/300 Write block 707 len 40960 data: 4 5 5 5 5 5 4 3 3 3
26915 275/300 Read block 4909 len 24576
26916 276/300 Read block 2283 len 20480
26917 277/300 Read block 2672 len 12288
26918 278/300 Write block 2399 len 24576 data: 7 7 6 5 4 5
26919 279/300 Write block 3791 len 20480 data: 6 5 4 3 4
26920 280/300 Write block 732 len 24576 data: 4 4 4 3 3 4
26921 281/300 Read block 2590 len 40960
26922 282/300 Read block 2974 len 16384
26923 283/300 Read block 1033 len 8192
26924 284/300 Read block 4986 len 8192
26925 285/300 Write block 2824 len 8192 data: 7 7
26926 286/300 Write block 4731 len 16384 data: 4 4 5 5
26927 287/300 Write block 1310 len 8192 data: 4 4
26928 288/300 Flush
26929 289/300 Flush
26930 290/300 Write block 3284 len 36864 data: 6 6 5 3 2 2 2 2 2
26931 291/300 Write block 4178 len 40960 data: 8 8 7 8 8 7 6 4 3 4
26932 292/300 Read block 825 len 20480
26933 293/300 Read block 1959 len 16384
26934 294/300 Flush
26935 295/300 Read block 938 len 8192
26936 296/300 Flush
26937 297/300 Write block 4716 len 36864 data: 7 7 5 5 4 4 4 3 2
26938 298/300 Write block 2645 len 28672 data: 3 3 2 2 2 2 2
26939 299/300 Write block 3062 len 40960 data: 4 3 3 2 3 3 3 3 4 4
26940 300/300 Read block 4875 len 36864
26941 301/300 Write block 1291 len 32768 data: 4 4 4 4 4 3 3 4
26942 Main task has work to do, go find it
26943 got dsc Start(1)
26944 start 1
26945 Mar 17 22:24:59.673 INFO request completed, latency_us: 214, response_code: 204, uri: /start/cid/1, method: POST, req_id: af208f32-161c-43ff-98fa-a444b27745a3, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
26946 Replay: started 1, returned:Ok(())
26947 ----------------------------------------------------------------
26948 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26949 States: Active Offline Active
26950 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26951 7340 Acked Read 7 Done Sent Done false
26952 7341 Acked Write 7 Done Sent Done false
26953 7342 Acked Flush 0 Done Sent Done false
26954 7343 Acked Read 3 Done Sent Done false
26955 7344 Acked Read 2 Done Sent Done false
26956 7345 Acked Flush 0 Done Sent Done false
26957 7346 Acked Write 8 Done Sent Done false
26958 7347 Acked Read 5 Done Sent Done false
26959 7348 Acked Flush 0 Done Sent Done false
26960 7349 Acked Write 4 Done Sent Done false
26961 7350 Acked Read 6 Done Sent Done false
26962 7351 Acked Read 7 Done Sent Done false
26963 7352 Acked Flush 0 Done Sent Done false
26964 7353 Acked Read 7 Done Sent Done false
26965 7354 Acked Write 5 Done Sent Done false
26966 7355 Acked Write 7 Done Sent Done false
26967 7356 Acked Write 10 Done Sent Done false
26968 7357 Acked Read 1 Done Sent Done false
26969 7358 Acked Read 1 Done Sent Done false
26970 7359 Acked Read 8 Done Sent Done false
26971 7360 Acked Read 4 Done Sent Done false
26972 7361 Acked Write 2 Done Sent Done false
26973 7362 Acked Read 8 Done Sent Done false
26974 7363 Acked Read 6 Done Sent Done false
26975 7364 Acked Read 6 Done Sent Done false
26976 7365 Acked Write 2 Done Sent Done false
26977 7366 Acked Read 1 Done Sent Done false
26978 7367 Acked Read 6 Done Sent Done false
26979 7368 Acked Read 6 Done Sent Done false
26980 7369 Acked Write 3 Done Sent Done false
26981 7370 Acked Write 3 Done Sent Done false
26982 7371 Acked Read 1 Done Sent Done false
26983 7372 Acked Read 5 Done Sent Done false
26984 7373 Acked Flush 0 Done Sent Done false
26985 7374 Acked Read 1 Done Sent Done false
26986 7375 Acked Write 4 Done Sent Done false
26987 7376 Acked Read 5 Done Sent Done false
26988 7377 Acked Read 8 Done Sent Done false
26989 7378 Acked Write 3 Done Sent Done false
26990 7379 Acked Read 6 Done Sent Done false
26991 7380 Acked Flush 0 Done Sent Done false
26992 7381 Acked Read 3 Done Sent Done false
26993 7382 Acked Read 1 Done Sent Done false
26994 7383 Acked Flush 0 Done Sent Done false
26995 7384 Acked Read 2 Done Sent Done false
26996 7385 Acked Read 3 Done Sent Done false
26997 7386 Acked Flush 0 Done Sent Done false
26998 7387 Acked Read 7 Done Sent Done false
26999 7388 Acked Read 3 Done Sent Done false
27000 7389 Acked Read 9 Done Sent Done false
27001 7390 Acked Write 1 Done Sent Done false
27002 7391 Acked Read 1 Done Sent Done false
27003 7392 Acked Flush 0 Done Sent Done false
27004 7393 Acked Read 5 Done Sent Done false
27005 7394 Acked Read 1 Done Sent Done false
27006 7395 Acked Write 6 Done Sent Done false
27007 7396 Acked Read 5 Done Sent Done false
27008 7397 Acked Write 9 Done Sent Done false
27009 7398 Acked Write 3 Done Sent Done false
27010 7399 Acked Write 5 Done Sent Done false
27011 7400 Acked Write 1 Done Sent Done false
27012 7401 Acked Write 6 Done Sent Done false
27013 7402 Acked Read 1 Done Sent Done false
27014 7403 Acked Write 7 Done Sent Done false
27015 7404 Acked Read 8 Done Sent Done false
27016 7405 Acked Write 9 Done Sent Done false
27017 7406 Acked Write 2 Done Sent Done false
27018 7407 Acked Write 10 Done Sent Done false
27019 7408 Acked Read 5 Done Sent Done false
27020 7409 Acked Read 1 Done Sent Done false
27021 7410 Acked Read 10 Done Sent Done false
27022 7411 Acked Write 5 Done Sent Done false
27023 7412 Acked Write 8 Done Sent Done false
27024 7413 Acked Write 8 Done Sent Done false
27025 7414 Acked Write 1 Done Sent Done false
27026 7415 Acked Write 8 Done Sent Done false
27027 7416 Acked Read 2 Done Sent Done false
27028 7417 Acked Write 6 Done Sent Done false
27029 7418 Acked Read 4 Done Sent Done false
27030 7419 Acked Flush 0 Done Sent Done false
27031 7420 Acked Read 3 Done Sent Done false
27032 7421 Acked Write 7 Done Sent Done false
27033 7422 Acked Read 2 Done Sent Done false
27034 7423 Acked Read 7 Done Sent Done false
27035 7424 Acked Read 4 Done Sent Done false
27036 7425 Acked Read 2 Done Sent Done false
27037 7426 Acked Write 7 Done Sent Done false
27038 7427 Acked Write 1 Done Sent Done false
27039 7428 Acked Write 6 Done Sent Done false
27040 7429 Acked Read 8 Done Sent Done false
27041 7430 Acked Write 8 Done Sent Done false
27042 7431 Acked Flush 0 Done Sent Done false
27043 7432 Acked Read 1 Done Sent Done false
27044 7433 Acked Read 9 Done Sent Done false
27045 7434 Acked Read 8 Done Sent Done false
27046 7435 Acked Read 4 Done Sent Done false
27047 7436 Acked Write 5 Done Sent Done false
27048 7437 Acked Flush 0 Done Sent Done false
27049 7438 Acked Write 9 Done Sent Done false
27050 7439 Acked Write 10 Done Sent Done false
27051 7440 Acked Read 10 Done Sent Done false
27052 7441 Acked Write 3 Done Sent Done false
27053 7442 Acked Flush 0 Done Sent Done false
27054 7443 Acked Read 5 Done Sent Done false
27055 7444 Acked Write 9 Done Sent Done false
27056 7445 Acked Read 2 Done Sent Done false
27057 7446 Acked Write 6 Done Sent Done false
27058 7447 Acked Read 4 Done Sent Done false
27059 7448 Acked Read 7 Done Sent Done false
27060 7449 Acked Write 7 Done Sent Done false
27061 7450 Acked Write 1 Done Sent Done false
27062 7451 Acked Write 7 Done Sent Done false
27063 7452 Acked Write 2 Done Sent Done false
27064 7453 Acked Read 9 Done Sent Done false
27065 7454 Acked Write 5 Done Sent Done false
27066 7455 Acked Read 6 Done Sent Done false
27067 7456 Acked Read 1 Done Sent Done false
27068 7457 Acked Write 4 Done Sent Done false
27069 7458 Acked Read 7 Done Sent Done false
27070 7459 Acked Write 3 Done Sent Done false
27071 7460 Acked Write 1 Done Sent Done false
27072 7461 Acked Read 1 Done Sent Done false
27073 7462 Acked Read 7 Done Sent Done false
27074 7463 Acked Read 9 Done Sent Done false
27075 7464 Acked Read 9 Done Sent Done false
27076 7465 Acked Write 5 Done Sent Done false
27077 7466 Acked Write 6 Done Sent Done false
27078 7467 Acked Read 10 Done Sent Done false
27079 7468 Acked Flush 0 Done Sent Done false
27080 7469 Acked Flush 0 Done Sent Done false
27081 7470 Acked Read 4 Done Sent Done false
27082 7471 Acked Read 5 Done Sent Done false
27083 7472 Acked Write 7 Done Sent Done false
27084 7473 Acked Read 5 Done Sent Done false
27085 7474 Acked Read 10 Done Sent Done false
27086 7475 Acked Read 4 Done Sent Done false
27087 7476 Acked Read 10 Done Sent Done false
27088 7477 Acked Write 6 Done Sent Done false
27089 7478 Acked Flush 0 Done Sent Done false
27090 7479 Acked Write 7 Done Sent Done false
27091 7480 Acked Read 7 Done Sent Done false
27092 7481 Acked Read 4 Done Sent Done false
27093 7482 Acked Write 3 Done Sent Done false
27094 7483 Acked Read 4 Done Sent Done false
27095 7484 Acked Write 10 Done Sent Done false
27096 7485 Acked Write 2 Done Sent Done false
27097 7486 Acked Read 2 Done Sent Done false
27098 7487 Acked Write 4 Done Sent Done false
27099 7488 Acked Read 1 Done Sent Done false
27100 7489 Acked Read 7 Done Sent Done false
27101 7490 Acked Read 8 Done Sent Done false
27102 7491 Acked Flush 0 Done Sent Done false
27103 7492 Acked Read 10 Done Sent Done false
27104 7493 Acked Write 4 Done Sent Done false
27105 7494 Acked Read 7 Done Sent Done false
27106 7495 Acked Write 10 Done Sent Done false
27107 7496 Acked Write 10 Done Sent Done false
27108 7497 Acked Read 7 Done Sent Done false
27109 7498 Acked Write 6 Done Sent Done false
27110 7499 Acked Write 1 Done Sent Done false
27111 7500 Acked Write 3 Done Sent Done false
27112 7501 Acked Read 9 Done Sent Done false
27113 7502 Acked Read 8 Done Sent Done false
27114 7503 Acked Read 4 Done Sent Done false
27115 7504 Acked Read 4 Done Sent Done false
27116 7505 Acked Read 6 Done Sent Done false
27117 7506 Acked Write 1 Done Sent Done false
27118 7507 Acked Write 4 Done Sent Done false
27119 7508 Acked Read 10 Done Sent Done false
27120 7509 Acked Flush 0 Done Sent Done false
27121 7510 Acked Write 3 Done Sent Done false
27122 7511 Acked Write 8 Done Sent Done false
27123 7512 Acked Flush 0 Done Sent Done false
27124 7513 Acked Read 4 Done Sent Done false
27125 7514 Acked Write 3 Done Sent Done false
27126 7515 Acked Read 4 Done Sent Done false
27127 7516 Acked Write 4 Done Sent Done false
27128 7517 Acked Write 1 Done Sent Done false
27129 7518 Acked Read 6 Done Sent Done false
27130 7519 Acked Write 3 Done Sent Done false
27131 7520 Acked Read 10 Done Sent Done false
27132 7521 Acked Write 9 Done Sent Done false
27133 7522 Acked Write 1 Done Sent Done false
27134 7523 Acked Read 1 Done Sent Done false
27135 7524 Acked Write 3 Done Sent Done false
27136 7525 Acked Flush 0 Done Sent Done false
27137 7526 Acked Write 3 Done Sent Done false
27138 7527 Acked Flush 0 Done Sent Done false
27139 7528 Acked Read 10 Done Sent Done false
27140 7529 Acked Flush 0 Done Sent Done false
27141 7530 Acked Flush 0 Done Sent Done false
27142 7531 Acked Flush 0 Done Sent Done false
27143 7532 Acked Write 4 Done Sent Done false
27144 7533 Acked Read 2 Done Sent Done false
27145 7534 Acked Read 2 Done Sent Done false
27146 7535 Acked Write 5 Done Sent Done false
27147 7536 Acked Write 1 Done Sent Done false
27148 7537 Acked Write 8 Done Sent Done false
27149 7538 Acked Write 1 Done Sent Done false
27150 7539 Acked Write 4 Done Sent Done false
27151 7540 Acked Write 3 Done Sent Done false
27152 7541 Acked Read 4 Done Sent Done false
27153 7542 Acked Flush 0 Done Sent Done false
27154 7543 Acked Write 9 Done Sent Done false
27155 7544 Acked Read 5 Done Sent Done false
27156 7545 Acked Write 10 Done Sent Done false
27157 7546 Acked Write 2 Done Sent Done false
27158 7547 Acked Write 5 Done Sent Done false
27159 7548 Acked Write 7 Done Sent Done false
27160 7549 Acked Write 7 Done Sent Done false
27161 7550 Acked Write 9 Done Sent Done false
27162 7551 Acked Write 9 Done Sent Done false
27163 7552 Acked Read 3 Done Sent Done false
27164 7553 Acked Write 7 Done Sent Done false
27165 7554 Acked Read 2 Done Sent Done false
27166 7555 Acked Write 9 Done Sent Done false
27167 7556 Acked Read 5 Done Sent Done false
27168 7557 Acked Read 9 Done Sent Done false
27169 7558 Acked Write 1 Done Sent Done false
27170 7559 Acked Read 4 Done Sent Done false
27171 7560 Acked Read 9 Done Sent Done false
27172 7561 Acked Read 6 Done Sent Done false
27173 7562 Acked Read 5 Done Sent Done false
27174 7563 Acked Write 3 Done Sent Done false
27175 7564 Acked Write 1 Done Sent Done false
27176 7565 Acked Write 3 Done Sent Done false
27177 7566 Acked Read 10 Done Sent Done false
27178 7567 Acked Read 3 Done Sent Done false
27179 7568 Acked Flush 0 Done Sent Done false
27180 7569 Acked Read 9 Done Sent Done false
27181 7570 Acked Read 1 Done Sent Done false
27182 7571 Acked Write 3 Done Sent Done false
27183 7572 Acked Read 10 Done Sent Done false
27184 7573 Acked Read 3 Done Sent Done false
27185 7574 Acked Flush 0 Done Sent Done false
27186 7575 Acked Read 5 Done Sent Done false
27187 7576 Acked Read 5 Done Sent Done false
27188 7577 Acked Write 5 Done Sent Done false
27189 7578 Acked Read 7 Done Sent Done false
27190 7579 Acked Write 7 Done Sent Done false
27191 7580 Acked Read 8 Done Sent Done false
27192 7581 Acked Write 9 Done Sent Done false
27193 7582 Acked Read 6 Done Sent Done false
27194 7583 Acked Write 2 Done Sent Done false
27195 7584 Acked Read 4 Done Sent Done false
27196 7585 Acked Write 5 Done Sent Done false
27197 7586 Acked Write 9 Done Sent Done false
27198 7587 Acked Flush 0 Done Sent Done false
27199 7588 Acked Read 6 Done Sent Done false
27200 7589 Acked Read 2 Done Sent Done false
27201 7590 Acked Write 9 Done Sent Done false
27202 7591 Acked Read 6 Done Sent Done false
27203 7592 Acked Read 8 Done Sent Done false
27204 7593 Acked Read 5 Done Sent Done false
27205 7594 Acked Read 7 Done Sent Done false
27206 7595 Acked Write 7 Done Sent Done false
27207 7596 Acked Read 3 Done Sent Done false
27208 7597 Acked Read 8 Done Sent Done false
27209 7598 Acked Read 8 Done Sent Done false
27210 7599 Acked Read 5 Done Sent Done false
27211 7600 Acked Read 2 Done Sent Done false
27212 7601 Acked Read 6 Done Sent Done false
27213 7602 Acked Read 4 Done Sent Done false
27214 7603 Acked Write 8 Done Sent Done false
27215 7604 Acked Write 1 Done Sent Done false
27216 7605 Acked Write 1 Done Sent Done false
27217 7606 Acked Write 2 Done Sent Done false
27218 7607 Acked Read 1 Done Sent Done false
27219 7608 Acked Read 9 Done Sent Done false
27220 7609 Acked Flush 0 Done Sent Done false
27221 7610 Acked Read 8 Done Sent Done false
27222 7611 Acked Read 8 Done Sent Done false
27223 7612 Acked Read 9 Done Sent Done false
27224 7613 Acked Write 10 Done Sent Done false
27225 7614 Acked Read 6 Done Sent Done false
27226 7615 Acked Read 5 Done Sent Done false
27227 7616 Acked Read 3 Done Sent Done false
27228 7617 Acked Write 6 Done Sent Done false
27229 7618 Acked Write 5 Done Sent Done false
27230 7619 Acked Write 6 Done Sent Done false
27231 7620 Acked Read 10 Done Sent Done false
27232 7621 Acked Read 4 Done Sent Done false
27233 7622 Acked Read 2 Done Sent Done false
27234 7623 Acked Read 2 Done Sent Done false
27235 7624 Acked Write 2 Done Sent Done false
27236 7625 Acked Write 4 Done Sent Done false
27237 7626 Acked Write 2 Done Sent Done false
27238 7627 Acked Flush 0 Done Sent Done false
27239 7628 Acked Flush 0 Done Sent Done false
27240 7629 Acked Write 9 Done Sent Done false
27241 7630 Acked Write 10 Done Sent Done false
27242 7631 Acked Read 5 Done Sent Done false
27243 7632 Acked Read 4 Done Sent Done false
27244 7633 Acked Flush 0 Done Sent Done false
27245 7634 Acked Read 2 Done Sent Done false
27246 7635 Acked Flush 0 Done Sent Done false
27247 7636 Acked Write 9 Done Sent Done false
27248 7637 Acked Write 7 Done Sent Done false
27249 7638 Acked Write 10 Done Sent Done false
27250 7639 Acked Read 9 Done Sent Done false
27251 7640 Acked Write 8 Sent Sent Done false
27252 STATES DS:0 DS:1 DS:2 TOTAL
27253 Sent 1 301 0 302
27254 Done 300 0 301 601
27255 Skipped 0 0 0 0
27256 Error 0 0 0 0
27257 Last Flush: Some(JobId(7635)) Some(JobId(7339)) Some(JobId(7635))
27258 Downstairs last five completed: 7339 7338 7337 7336 7335
27259 Upstairs last five completed: 7640 7639 7638 7637 7636
27260 CLIENT: Up:0 ds:301 act:2
272612025-03-17T22:25:00.175ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
272622025-03-17T22:25:01.157ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
272632025-03-17T22:25:02.176ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
272642025-03-17T22:25:03.161ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
27265 ----------------------------------------------------------------
27266 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27267 States: Active Offline Active
27268 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27269 7340 Acked Read 7 Done Sent Done false
27270 7341 Acked Write 7 Done Sent Done false
27271 7342 Acked Flush 0 Done Sent Done false
27272 7343 Acked Read 3 Done Sent Done false
27273 7344 Acked Read 2 Done Sent Done false
27274 7345 Acked Flush 0 Done Sent Done false
27275 7346 Acked Write 8 Done Sent Done false
27276 7347 Acked Read 5 Done Sent Done false
27277 7348 Acked Flush 0 Done Sent Done false
27278 7349 Acked Write 4 Done Sent Done false
27279 7350 Acked Read 6 Done Sent Done false
27280 7351 Acked Read 7 Done Sent Done false
27281 7352 Acked Flush 0 Done Sent Done false
27282 7353 Acked Read 7 Done Sent Done false
27283 7354 Acked Write 5 Done Sent Done false
27284 7355 Acked Write 7 Done Sent Done false
27285 7356 Acked Write 10 Done Sent Done false
27286 7357 Acked Read 1 Done Sent Done false
27287 7358 Acked Read 1 Done Sent Done false
27288 7359 Acked Read 8 Done Sent Done false
27289 7360 Acked Read 4 Done Sent Done false
27290 7361 Acked Write 2 Done Sent Done false
27291 7362 Acked Read 8 Done Sent Done false
27292 7363 Acked Read 6 Done Sent Done false
27293 7364 Acked Read 6 Done Sent Done false
27294 7365 Acked Write 2 Done Sent Done false
27295 7366 Acked Read 1 Done Sent Done false
27296 7367 Acked Read 6 Done Sent Done false
27297 7368 Acked Read 6 Done Sent Done false
27298 7369 Acked Write 3 Done Sent Done false
27299 7370 Acked Write 3 Done Sent Done false
27300 7371 Acked Read 1 Done Sent Done false
27301 7372 Acked Read 5 Done Sent Done false
27302 7373 Acked Flush 0 Done Sent Done false
27303 7374 Acked Read 1 Done Sent Done false
27304 7375 Acked Write 4 Done Sent Done false
27305 7376 Acked Read 5 Done Sent Done false
27306 7377 Acked Read 8 Done Sent Done false
27307 7378 Acked Write 3 Done Sent Done false
27308 7379 Acked Read 6 Done Sent Done false
27309 7380 Acked Flush 0 Done Sent Done false
27310 7381 Acked Read 3 Done Sent Done false
27311 7382 Acked Read 1 Done Sent Done false
27312 7383 Acked Flush 0 Done Sent Done false
27313 7384 Acked Read 2 Done Sent Done false
27314 7385 Acked Read 3 Done Sent Done false
27315 7386 Acked Flush 0 Done Sent Done false
27316 7387 Acked Read 7 Done Sent Done false
27317 7388 Acked Read 3 Done Sent Done false
27318 7389 Acked Read 9 Done Sent Done false
27319 7390 Acked Write 1 Done Sent Done false
27320 7391 Acked Read 1 Done Sent Done false
27321 7392 Acked Flush 0 Done Sent Done false
27322 7393 Acked Read 5 Done Sent Done false
27323 7394 Acked Read 1 Done Sent Done false
27324 7395 Acked Write 6 Done Sent Done false
27325 7396 Acked Read 5 Done Sent Done false
27326 7397 Acked Write 9 Done Sent Done false
27327 7398 Acked Write 3 Done Sent Done false
27328 7399 Acked Write 5 Done Sent Done false
27329 7400 Acked Write 1 Done Sent Done false
27330 7401 Acked Write 6 Done Sent Done false
27331 7402 Acked Read 1 Done Sent Done false
27332 7403 Acked Write 7 Done Sent Done false
27333 7404 Acked Read 8 Done Sent Done false
27334 7405 Acked Write 9 Done Sent Done false
27335 7406 Acked Write 2 Done Sent Done false
27336 7407 Acked Write 10 Done Sent Done false
27337 7408 Acked Read 5 Done Sent Done false
27338 7409 Acked Read 1 Done Sent Done false
27339 7410 Acked Read 10 Done Sent Done false
27340 7411 Acked Write 5 Done Sent Done false
27341 7412 Acked Write 8 Done Sent Done false
27342 7413 Acked Write 8 Done Sent Done false
27343 7414 Acked Write 1 Done Sent Done false
27344 7415 Acked Write 8 Done Sent Done false
27345 7416 Acked Read 2 Done Sent Done false
27346 7417 Acked Write 6 Done Sent Done false
27347 7418 Acked Read 4 Done Sent Done false
27348 7419 Acked Flush 0 Done Sent Done false
27349 7420 Acked Read 3 Done Sent Done false
27350 7421 Acked Write 7 Done Sent Done false
27351 7422 Acked Read 2 Done Sent Done false
27352 7423 Acked Read 7 Done Sent Done false
27353 7424 Acked Read 4 Done Sent Done false
27354 7425 Acked Read 2 Done Sent Done false
27355 7426 Acked Write 7 Done Sent Done false
27356 7427 Acked Write 1 Done Sent Done false
27357 7428 Acked Write 6 Done Sent Done false
27358 7429 Acked Read 8 Done Sent Done false
27359 7430 Acked Write 8 Done Sent Done false
27360 7431 Acked Flush 0 Done Sent Done false
27361 7432 Acked Read 1 Done Sent Done false
27362 7433 Acked Read 9 Done Sent Done false
27363 7434 Acked Read 8 Done Sent Done false
27364 7435 Acked Read 4 Done Sent Done false
27365 7436 Acked Write 5 Done Sent Done false
27366 7437 Acked Flush 0 Done Sent Done false
27367 7438 Acked Write 9 Done Sent Done false
27368 7439 Acked Write 10 Done Sent Done false
27369 7440 Acked Read 10 Done Sent Done false
27370 7441 Acked Write 3 Done Sent Done false
27371 7442 Acked Flush 0 Done Sent Done false
27372 7443 Acked Read 5 Done Sent Done false
27373 7444 Acked Write 9 Done Sent Done false
27374 7445 Acked Read 2 Done Sent Done false
27375 7446 Acked Write 6 Done Sent Done false
27376 7447 Acked Read 4 Done Sent Done false
27377 7448 Acked Read 7 Done Sent Done false
27378 7449 Acked Write 7 Done Sent Done false
27379 7450 Acked Write 1 Done Sent Done false
27380 7451 Acked Write 7 Done Sent Done false
27381 7452 Acked Write 2 Done Sent Done false
27382 7453 Acked Read 9 Done Sent Done false
27383 7454 Acked Write 5 Done Sent Done false
27384 7455 Acked Read 6 Done Sent Done false
27385 7456 Acked Read 1 Done Sent Done false
27386 7457 Acked Write 4 Done Sent Done false
27387 7458 Acked Read 7 Done Sent Done false
27388 7459 Acked Write 3 Done Sent Done false
27389 7460 Acked Write 1 Done Sent Done false
27390 7461 Acked Read 1 Done Sent Done false
27391 7462 Acked Read 7 Done Sent Done false
27392 7463 Acked Read 9 Done Sent Done false
27393 7464 Acked Read 9 Done Sent Done false
27394 7465 Acked Write 5 Done Sent Done false
27395 7466 Acked Write 6 Done Sent Done false
27396 7467 Acked Read 10 Done Sent Done false
27397 7468 Acked Flush 0 Done Sent Done false
27398 7469 Acked Flush 0 Done Sent Done false
27399 7470 Acked Read 4 Done Sent Done false
27400 7471 Acked Read 5 Done Sent Done false
27401 7472 Acked Write 7 Done Sent Done false
27402 7473 Acked Read 5 Done Sent Done false
27403 7474 Acked Read 10 Done Sent Done false
27404 7475 Acked Read 4 Done Sent Done false
27405 7476 Acked Read 10 Done Sent Done false
27406 7477 Acked Write 6 Done Sent Done false
27407 7478 Acked Flush 0 Done Sent Done false
27408 7479 Acked Write 7 Done Sent Done false
27409 7480 Acked Read 7 Done Sent Done false
27410 7481 Acked Read 4 Done Sent Done false
27411 7482 Acked Write 3 Done Sent Done false
27412 7483 Acked Read 4 Done Sent Done false
27413 7484 Acked Write 10 Done Sent Done false
27414 7485 Acked Write 2 Done Sent Done false
27415 7486 Acked Read 2 Done Sent Done false
27416 7487 Acked Write 4 Done Sent Done false
27417 7488 Acked Read 1 Done Sent Done false
27418 7489 Acked Read 7 Done Sent Done false
27419 7490 Acked Read 8 Done Sent Done false
27420 7491 Acked Flush 0 Done Sent Done false
27421 7492 Acked Read 10 Done Sent Done false
27422 7493 Acked Write 4 Done Sent Done false
27423 7494 Acked Read 7 Done Sent Done false
27424 7495 Acked Write 10 Done Sent Done false
27425 7496 Acked Write 10 Done Sent Done false
27426 7497 Acked Read 7 Done Sent Done false
27427 7498 Acked Write 6 Done Sent Done false
27428 7499 Acked Write 1 Done Sent Done false
27429 7500 Acked Write 3 Done Sent Done false
27430 7501 Acked Read 9 Done Sent Done false
27431 7502 Acked Read 8 Done Sent Done false
27432 7503 Acked Read 4 Done Sent Done false
27433 7504 Acked Read 4 Done Sent Done false
27434 7505 Acked Read 6 Done Sent Done false
27435 7506 Acked Write 1 Done Sent Done false
27436 7507 Acked Write 4 Done Sent Done false
27437 7508 Acked Read 10 Done Sent Done false
27438 7509 Acked Flush 0 Done Sent Done false
27439 7510 Acked Write 3 Done Sent Done false
27440 7511 Acked Write 8 Done Sent Done false
27441 7512 Acked Flush 0 Done Sent Done false
27442 7513 Acked Read 4 Done Sent Done false
27443 7514 Acked Write 3 Done Sent Done false
27444 7515 Acked Read 4 Done Sent Done false
27445 7516 Acked Write 4 Done Sent Done false
27446 7517 Acked Write 1 Done Sent Done false
27447 7518 Acked Read 6 Done Sent Done false
27448 7519 Acked Write 3 Done Sent Done false
27449 7520 Acked Read 10 Done Sent Done false
27450 7521 Acked Write 9 Done Sent Done false
27451 7522 Acked Write 1 Done Sent Done false
27452 7523 Acked Read 1 Done Sent Done false
27453 7524 Acked Write 3 Done Sent Done false
27454 7525 Acked Flush 0 Done Sent Done false
27455 7526 Acked Write 3 Done Sent Done false
27456 7527 Acked Flush 0 Done Sent Done false
27457 7528 Acked Read 10 Done Sent Done false
27458 7529 Acked Flush 0 Done Sent Done false
27459 7530 Acked Flush 0 Done Sent Done false
27460 7531 Acked Flush 0 Done Sent Done false
27461 7532 Acked Write 4 Done Sent Done false
27462 7533 Acked Read 2 Done Sent Done false
27463 7534 Acked Read 2 Done Sent Done false
27464 7535 Acked Write 5 Done Sent Done false
27465 7536 Acked Write 1 Done Sent Done false
27466 7537 Acked Write 8 Done Sent Done false
27467 7538 Acked Write 1 Done Sent Done false
27468 7539 Acked Write 4 Done Sent Done false
27469 7540 Acked Write 3 Done Sent Done false
27470 7541 Acked Read 4 Done Sent Done false
27471 7542 Acked Flush 0 Done Sent Done false
27472 7543 Acked Write 9 Done Sent Done false
27473 7544 Acked Read 5 Done Sent Done false
27474 7545 Acked Write 10 Done Sent Done false
27475 7546 Acked Write 2 Done Sent Done false
27476 7547 Acked Write 5 Done Sent Done false
27477 7548 Acked Write 7 Done Sent Done false
27478 7549 Acked Write 7 Done Sent Done false
27479 7550 Acked Write 9 Done Sent Done false
27480 7551 Acked Write 9 Done Sent Done false
27481 7552 Acked Read 3 Done Sent Done false
27482 7553 Acked Write 7 Done Sent Done false
27483 7554 Acked Read 2 Done Sent Done false
27484 7555 Acked Write 9 Done Sent Done false
27485 7556 Acked Read 5 Done Sent Done false
27486 7557 Acked Read 9 Done Sent Done false
27487 7558 Acked Write 1 Done Sent Done false
27488 7559 Acked Read 4 Done Sent Done false
27489 7560 Acked Read 9 Done Sent Done false
27490 7561 Acked Read 6 Done Sent Done false
27491 7562 Acked Read 5 Done Sent Done false
27492 7563 Acked Write 3 Done Sent Done false
27493 7564 Acked Write 1 Done Sent Done false
27494 7565 Acked Write 3 Done Sent Done false
27495 7566 Acked Read 10 Done Sent Done false
27496 7567 Acked Read 3 Done Sent Done false
27497 7568 Acked Flush 0 Done Sent Done false
27498 7569 Acked Read 9 Done Sent Done false
27499 7570 Acked Read 1 Done Sent Done false
27500 7571 Acked Write 3 Done Sent Done false
27501 7572 Acked Read 10 Done Sent Done false
27502 7573 Acked Read 3 Done Sent Done false
27503 7574 Acked Flush 0 Done Sent Done false
27504 7575 Acked Read 5 Done Sent Done false
27505 7576 Acked Read 5 Done Sent Done false
27506 7577 Acked Write 5 Done Sent Done false
27507 7578 Acked Read 7 Done Sent Done false
27508 7579 Acked Write 7 Done Sent Done false
27509 7580 Acked Read 8 Done Sent Done false
27510 7581 Acked Write 9 Done Sent Done false
27511 7582 Acked Read 6 Done Sent Done false
27512 7583 Acked Write 2 Done Sent Done false
27513 7584 Acked Read 4 Done Sent Done false
27514 7585 Acked Write 5 Done Sent Done false
27515 7586 Acked Write 9 Done Sent Done false
27516 7587 Acked Flush 0 Done Sent Done false
27517 7588 Acked Read 6 Done Sent Done false
27518 7589 Acked Read 2 Done Sent Done false
27519 7590 Acked Write 9 Done Sent Done false
27520 7591 Acked Read 6 Done Sent Done false
27521 7592 Acked Read 8 Done Sent Done false
27522 7593 Acked Read 5 Done Sent Done false
27523 7594 Acked Read 7 Done Sent Done false
27524 7595 Acked Write 7 Done Sent Done false
27525 7596 Acked Read 3 Done Sent Done false
27526 7597 Acked Read 8 Done Sent Done false
27527 7598 Acked Read 8 Done Sent Done false
27528 7599 Acked Read 5 Done Sent Done false
27529 7600 Acked Read 2 Done Sent Done false
27530 7601 Acked Read 6 Done Sent Done false
27531 7602 Acked Read 4 Done Sent Done false
27532 7603 Acked Write 8 Done Sent Done false
27533 7604 Acked Write 1 Done Sent Done false
27534 7605 Acked Write 1 Done Sent Done false
27535 7606 Acked Write 2 Done Sent Done false
27536 7607 Acked Read 1 Done Sent Done false
27537 7608 Acked Read 9 Done Sent Done false
27538 7609 Acked Flush 0 Done Sent Done false
27539 7610 Acked Read 8 Done Sent Done false
27540 7611 Acked Read 8 Done Sent Done false
27541 7612 Acked Read 9 Done Sent Done false
27542 7613 Acked Write 10 Done Sent Done false
27543 7614 Acked Read 6 Done Sent Done false
27544 7615 Acked Read 5 Done Sent Done false
27545 7616 Acked Read 3 Done Sent Done false
27546 7617 Acked Write 6 Done Sent Done false
27547 7618 Acked Write 5 Done Sent Done false
27548 7619 Acked Write 6 Done Sent Done false
27549 7620 Acked Read 10 Done Sent Done false
27550 7621 Acked Read 4 Done Sent Done false
27551 7622 Acked Read 2 Done Sent Done false
27552 7623 Acked Read 2 Done Sent Done false
27553 7624 Acked Write 2 Done Sent Done false
27554 7625 Acked Write 4 Done Sent Done false
27555 7626 Acked Write 2 Done Sent Done false
27556 7627 Acked Flush 0 Done Sent Done false
27557 7628 Acked Flush 0 Done Sent Done false
27558 7629 Acked Write 9 Done Sent Done false
27559 7630 Acked Write 10 Done Sent Done false
27560 7631 Acked Read 5 Done Sent Done false
27561 7632 Acked Read 4 Done Sent Done false
27562 7633 Acked Flush 0 Done Sent Done false
27563 7634 Acked Read 2 Done Sent Done false
27564 7635 Acked Flush 0 Done Sent Done false
27565 7636 Acked Write 9 Done Sent Done false
27566 7637 Acked Write 7 Done Sent Done false
27567 7638 Acked Write 10 Done Sent Done false
27568 7639 Acked Read 9 Done Sent Done false
27569 7640 Acked Write 8 Done Sent Done false
27570 STATES DS:0 DS:1 DS:2 TOTAL
27571 Sent 0 301 0 301
27572 Done 301 0 301 602
27573 Skipped 0 0 0 0
27574 Error 0 0 0 0
27575 Last Flush: Some(JobId(7635)) Some(JobId(7339)) Some(JobId(7635))
27576 Downstairs last five completed: 7339 7338 7337 7336 7335
27577 Upstairs last five completed: 7640 7639 7638 7637 7636
27578 CLIENT: Up:0 ds:301 act:2
275792025-03-17T22:25:04.432ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
275802025-03-17T22:25:05.164ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
275812025-03-17T22:25:05.436ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
275822025-03-17T22:25:05.436ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
275832025-03-17T22:25:05.436ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
275842025-03-17T22:25:05.436ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
275852025-03-17T22:25:05.436ZINFOcrucible: client task is exiting = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
27586 [8820] got start action
27587 [8820] I am going to restart
27588 Starting downstairs at port 8820
27589 Make output file at "/tmp/dsc/downstairs-8820.txt"
27590 [8820][1] reports Starting
27591 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1233)
27592 [8820][1] reports Running
275932025-03-17T22:25:06.164ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
275942025-03-17T22:25:07.177ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
27595 ----------------------------------------------------------------
27596 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27597 States: Active Offline Active
27598 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27599 7340 Acked Read 7 Done Sent Done false
27600 7341 Acked Write 7 Done Sent Done false
27601 7342 Acked Flush 0 Done Sent Done false
27602 7343 Acked Read 3 Done Sent Done false
27603 7344 Acked Read 2 Done Sent Done false
27604 7345 Acked Flush 0 Done Sent Done false
27605 7346 Acked Write 8 Done Sent Done false
27606 7347 Acked Read 5 Done Sent Done false
27607 7348 Acked Flush 0 Done Sent Done false
27608 7349 Acked Write 4 Done Sent Done false
27609 7350 Acked Read 6 Done Sent Done false
27610 7351 Acked Read 7 Done Sent Done false
27611 7352 Acked Flush 0 Done Sent Done false
27612 7353 Acked Read 7 Done Sent Done false
27613 7354 Acked Write 5 Done Sent Done false
27614 7355 Acked Write 7 Done Sent Done false
27615 7356 Acked Write 10 Done Sent Done false
27616 7357 Acked Read 1 Done Sent Done false
27617 7358 Acked Read 1 Done Sent Done false
27618 7359 Acked Read 8 Done Sent Done false
27619 7360 Acked Read 4 Done Sent Done false
27620 7361 Acked Write 2 Done Sent Done false
27621 7362 Acked Read 8 Done Sent Done false
27622 7363 Acked Read 6 Done Sent Done false
27623 7364 Acked Read 6 Done Sent Done false
27624 7365 Acked Write 2 Done Sent Done false
27625 7366 Acked Read 1 Done Sent Done false
27626 7367 Acked Read 6 Done Sent Done false
27627 7368 Acked Read 6 Done Sent Done false
27628 7369 Acked Write 3 Done Sent Done false
27629 7370 Acked Write 3 Done Sent Done false
27630 7371 Acked Read 1 Done Sent Done false
27631 7372 Acked Read 5 Done Sent Done false
27632 7373 Acked Flush 0 Done Sent Done false
27633 7374 Acked Read 1 Done Sent Done false
27634 7375 Acked Write 4 Done Sent Done false
27635 7376 Acked Read 5 Done Sent Done false
27636 7377 Acked Read 8 Done Sent Done false
27637 7378 Acked Write 3 Done Sent Done false
27638 7379 Acked Read 6 Done Sent Done false
27639 7380 Acked Flush 0 Done Sent Done false
27640 7381 Acked Read 3 Done Sent Done false
27641 7382 Acked Read 1 Done Sent Done false
27642 7383 Acked Flush 0 Done Sent Done false
27643 7384 Acked Read 2 Done Sent Done false
27644 7385 Acked Read 3 Done Sent Done false
27645 7386 Acked Flush 0 Done Sent Done false
27646 7387 Acked Read 7 Done Sent Done false
27647 7388 Acked Read 3 Done Sent Done false
27648 7389 Acked Read 9 Done Sent Done false
27649 7390 Acked Write 1 Done Sent Done false
27650 7391 Acked Read 1 Done Sent Done false
27651 7392 Acked Flush 0 Done Sent Done false
27652 7393 Acked Read 5 Done Sent Done false
27653 7394 Acked Read 1 Done Sent Done false
27654 7395 Acked Write 6 Done Sent Done false
27655 7396 Acked Read 5 Done Sent Done false
27656 7397 Acked Write 9 Done Sent Done false
27657 7398 Acked Write 3 Done Sent Done false
27658 7399 Acked Write 5 Done Sent Done false
27659 7400 Acked Write 1 Done Sent Done false
27660 7401 Acked Write 6 Done Sent Done false
27661 7402 Acked Read 1 Done Sent Done false
27662 7403 Acked Write 7 Done Sent Done false
27663 7404 Acked Read 8 Done Sent Done false
27664 7405 Acked Write 9 Done Sent Done false
27665 7406 Acked Write 2 Done Sent Done false
27666 7407 Acked Write 10 Done Sent Done false
27667 7408 Acked Read 5 Done Sent Done false
27668 7409 Acked Read 1 Done Sent Done false
27669 7410 Acked Read 10 Done Sent Done false
27670 7411 Acked Write 5 Done Sent Done false
27671 7412 Acked Write 8 Done Sent Done false
27672 7413 Acked Write 8 Done Sent Done false
27673 7414 Acked Write 1 Done Sent Done false
27674 7415 Acked Write 8 Done Sent Done false
27675 7416 Acked Read 2 Done Sent Done false
27676 7417 Acked Write 6 Done Sent Done false
27677 7418 Acked Read 4 Done Sent Done false
27678 7419 Acked Flush 0 Done Sent Done false
27679 7420 Acked Read 3 Done Sent Done false
27680 7421 Acked Write 7 Done Sent Done false
27681 7422 Acked Read 2 Done Sent Done false
27682 7423 Acked Read 7 Done Sent Done false
27683 7424 Acked Read 4 Done Sent Done false
27684 7425 Acked Read 2 Done Sent Done false
27685 7426 Acked Write 7 Done Sent Done false
27686 7427 Acked Write 1 Done Sent Done false
27687 7428 Acked Write 6 Done Sent Done false
27688 7429 Acked Read 8 Done Sent Done false
27689 7430 Acked Write 8 Done Sent Done false
27690 7431 Acked Flush 0 Done Sent Done false
27691 7432 Acked Read 1 Done Sent Done false
27692 7433 Acked Read 9 Done Sent Done false
27693 7434 Acked Read 8 Done Sent Done false
27694 7435 Acked Read 4 Done Sent Done false
27695 7436 Acked Write 5 Done Sent Done false
27696 7437 Acked Flush 0 Done Sent Done false
27697 7438 Acked Write 9 Done Sent Done false
27698 7439 Acked Write 10 Done Sent Done false
27699 7440 Acked Read 10 Done Sent Done false
27700 7441 Acked Write 3 Done Sent Done false
27701 7442 Acked Flush 0 Done Sent Done false
27702 7443 Acked Read 5 Done Sent Done false
27703 7444 Acked Write 9 Done Sent Done false
27704 7445 Acked Read 2 Done Sent Done false
27705 7446 Acked Write 6 Done Sent Done false
27706 7447 Acked Read 4 Done Sent Done false
27707 7448 Acked Read 7 Done Sent Done false
27708 7449 Acked Write 7 Done Sent Done false
27709 7450 Acked Write 1 Done Sent Done false
27710 7451 Acked Write 7 Done Sent Done false
27711 7452 Acked Write 2 Done Sent Done false
27712 7453 Acked Read 9 Done Sent Done false
27713 7454 Acked Write 5 Done Sent Done false
27714 7455 Acked Read 6 Done Sent Done false
27715 7456 Acked Read 1 Done Sent Done false
27716 7457 Acked Write 4 Done Sent Done false
27717 7458 Acked Read 7 Done Sent Done false
27718 7459 Acked Write 3 Done Sent Done false
27719 7460 Acked Write 1 Done Sent Done false
27720 7461 Acked Read 1 Done Sent Done false
27721 7462 Acked Read 7 Done Sent Done false
27722 7463 Acked Read 9 Done Sent Done false
27723 7464 Acked Read 9 Done Sent Done false
27724 7465 Acked Write 5 Done Sent Done false
27725 7466 Acked Write 6 Done Sent Done false
27726 7467 Acked Read 10 Done Sent Done false
27727 7468 Acked Flush 0 Done Sent Done false
27728 7469 Acked Flush 0 Done Sent Done false
27729 7470 Acked Read 4 Done Sent Done false
27730 7471 Acked Read 5 Done Sent Done false
27731 7472 Acked Write 7 Done Sent Done false
27732 7473 Acked Read 5 Done Sent Done false
27733 7474 Acked Read 10 Done Sent Done false
27734 7475 Acked Read 4 Done Sent Done false
27735 7476 Acked Read 10 Done Sent Done false
27736 7477 Acked Write 6 Done Sent Done false
27737 7478 Acked Flush 0 Done Sent Done false
27738 7479 Acked Write 7 Done Sent Done false
27739 7480 Acked Read 7 Done Sent Done false
27740 7481 Acked Read 4 Done Sent Done false
27741 7482 Acked Write 3 Done Sent Done false
27742 7483 Acked Read 4 Done Sent Done false
27743 7484 Acked Write 10 Done Sent Done false
27744 7485 Acked Write 2 Done Sent Done false
27745 7486 Acked Read 2 Done Sent Done false
27746 7487 Acked Write 4 Done Sent Done false
27747 7488 Acked Read 1 Done Sent Done false
27748 7489 Acked Read 7 Done Sent Done false
27749 7490 Acked Read 8 Done Sent Done false
27750 7491 Acked Flush 0 Done Sent Done false
27751 7492 Acked Read 10 Done Sent Done false
27752 7493 Acked Write 4 Done Sent Done false
27753 7494 Acked Read 7 Done Sent Done false
27754 7495 Acked Write 10 Done Sent Done false
27755 7496 Acked Write 10 Done Sent Done false
27756 7497 Acked Read 7 Done Sent Done false
27757 7498 Acked Write 6 Done Sent Done false
27758 7499 Acked Write 1 Done Sent Done false
27759 7500 Acked Write 3 Done Sent Done false
27760 7501 Acked Read 9 Done Sent Done false
27761 7502 Acked Read 8 Done Sent Done false
27762 7503 Acked Read 4 Done Sent Done false
27763 7504 Acked Read 4 Done Sent Done false
27764 7505 Acked Read 6 Done Sent Done false
27765 7506 Acked Write 1 Done Sent Done false
27766 7507 Acked Write 4 Done Sent Done false
27767 7508 Acked Read 10 Done Sent Done false
27768 7509 Acked Flush 0 Done Sent Done false
27769 7510 Acked Write 3 Done Sent Done false
27770 7511 Acked Write 8 Done Sent Done false
27771 7512 Acked Flush 0 Done Sent Done false
27772 7513 Acked Read 4 Done Sent Done false
27773 7514 Acked Write 3 Done Sent Done false
27774 7515 Acked Read 4 Done Sent Done false
27775 7516 Acked Write 4 Done Sent Done false
27776 7517 Acked Write 1 Done Sent Done false
27777 7518 Acked Read 6 Done Sent Done false
27778 7519 Acked Write 3 Done Sent Done false
27779 7520 Acked Read 10 Done Sent Done false
27780 7521 Acked Write 9 Done Sent Done false
27781 7522 Acked Write 1 Done Sent Done false
27782 7523 Acked Read 1 Done Sent Done false
27783 7524 Acked Write 3 Done Sent Done false
27784 7525 Acked Flush 0 Done Sent Done false
27785 7526 Acked Write 3 Done Sent Done false
27786 7527 Acked Flush 0 Done Sent Done false
27787 7528 Acked Read 10 Done Sent Done false
27788 7529 Acked Flush 0 Done Sent Done false
27789 7530 Acked Flush 0 Done Sent Done false
27790 7531 Acked Flush 0 Done Sent Done false
27791 7532 Acked Write 4 Done Sent Done false
27792 7533 Acked Read 2 Done Sent Done false
27793 7534 Acked Read 2 Done Sent Done false
27794 7535 Acked Write 5 Done Sent Done false
27795 7536 Acked Write 1 Done Sent Done false
27796 7537 Acked Write 8 Done Sent Done false
27797 7538 Acked Write 1 Done Sent Done false
27798 7539 Acked Write 4 Done Sent Done false
27799 7540 Acked Write 3 Done Sent Done false
27800 7541 Acked Read 4 Done Sent Done false
27801 7542 Acked Flush 0 Done Sent Done false
27802 7543 Acked Write 9 Done Sent Done false
27803 7544 Acked Read 5 Done Sent Done false
27804 7545 Acked Write 10 Done Sent Done false
27805 7546 Acked Write 2 Done Sent Done false
27806 7547 Acked Write 5 Done Sent Done false
27807 7548 Acked Write 7 Done Sent Done false
27808 7549 Acked Write 7 Done Sent Done false
27809 7550 Acked Write 9 Done Sent Done false
27810 7551 Acked Write 9 Done Sent Done false
27811 7552 Acked Read 3 Done Sent Done false
27812 7553 Acked Write 7 Done Sent Done false
27813 7554 Acked Read 2 Done Sent Done false
27814 7555 Acked Write 9 Done Sent Done false
27815 7556 Acked Read 5 Done Sent Done false
27816 7557 Acked Read 9 Done Sent Done false
27817 7558 Acked Write 1 Done Sent Done false
27818 7559 Acked Read 4 Done Sent Done false
27819 7560 Acked Read 9 Done Sent Done false
27820 7561 Acked Read 6 Done Sent Done false
27821 7562 Acked Read 5 Done Sent Done false
27822 7563 Acked Write 3 Done Sent Done false
27823 7564 Acked Write 1 Done Sent Done false
27824 7565 Acked Write 3 Done Sent Done false
27825 7566 Acked Read 10 Done Sent Done false
27826 7567 Acked Read 3 Done Sent Done false
27827 7568 Acked Flush 0 Done Sent Done false
27828 7569 Acked Read 9 Done Sent Done false
27829 7570 Acked Read 1 Done Sent Done false
27830 7571 Acked Write 3 Done Sent Done false
27831 7572 Acked Read 10 Done Sent Done false
27832 7573 Acked Read 3 Done Sent Done false
27833 7574 Acked Flush 0 Done Sent Done false
27834 7575 Acked Read 5 Done Sent Done false
27835 7576 Acked Read 5 Done Sent Done false
27836 7577 Acked Write 5 Done Sent Done false
27837 7578 Acked Read 7 Done Sent Done false
27838 7579 Acked Write 7 Done Sent Done false
27839 7580 Acked Read 8 Done Sent Done false
27840 7581 Acked Write 9 Done Sent Done false
27841 7582 Acked Read 6 Done Sent Done false
27842 7583 Acked Write 2 Done Sent Done false
27843 7584 Acked Read 4 Done Sent Done false
27844 7585 Acked Write 5 Done Sent Done false
27845 7586 Acked Write 9 Done Sent Done false
27846 7587 Acked Flush 0 Done Sent Done false
27847 7588 Acked Read 6 Done Sent Done false
27848 7589 Acked Read 2 Done Sent Done false
27849 7590 Acked Write 9 Done Sent Done false
27850 7591 Acked Read 6 Done Sent Done false
27851 7592 Acked Read 8 Done Sent Done false
27852 7593 Acked Read 5 Done Sent Done false
27853 7594 Acked Read 7 Done Sent Done false
27854 7595 Acked Write 7 Done Sent Done false
27855 7596 Acked Read 3 Done Sent Done false
27856 7597 Acked Read 8 Done Sent Done false
27857 7598 Acked Read 8 Done Sent Done false
27858 7599 Acked Read 5 Done Sent Done false
27859 7600 Acked Read 2 Done Sent Done false
27860 7601 Acked Read 6 Done Sent Done false
27861 7602 Acked Read 4 Done Sent Done false
27862 7603 Acked Write 8 Done Sent Done false
27863 7604 Acked Write 1 Done Sent Done false
27864 7605 Acked Write 1 Done Sent Done false
27865 7606 Acked Write 2 Done Sent Done false
27866 7607 Acked Read 1 Done Sent Done false
27867 7608 Acked Read 9 Done Sent Done false
27868 7609 Acked Flush 0 Done Sent Done false
27869 7610 Acked Read 8 Done Sent Done false
27870 7611 Acked Read 8 Done Sent Done false
27871 7612 Acked Read 9 Done Sent Done false
27872 7613 Acked Write 10 Done Sent Done false
27873 7614 Acked Read 6 Done Sent Done false
27874 7615 Acked Read 5 Done Sent Done false
27875 7616 Acked Read 3 Done Sent Done false
27876 7617 Acked Write 6 Done Sent Done false
27877 7618 Acked Write 5 Done Sent Done false
27878 7619 Acked Write 6 Done Sent Done false
27879 7620 Acked Read 10 Done Sent Done false
27880 7621 Acked Read 4 Done Sent Done false
27881 7622 Acked Read 2 Done Sent Done false
27882 7623 Acked Read 2 Done Sent Done false
27883 7624 Acked Write 2 Done Sent Done false
27884 7625 Acked Write 4 Done Sent Done false
27885 7626 Acked Write 2 Done Sent Done false
27886 7627 Acked Flush 0 Done Sent Done false
27887 7628 Acked Flush 0 Done Sent Done false
27888 7629 Acked Write 9 Done Sent Done false
27889 7630 Acked Write 10 Done Sent Done false
27890 7631 Acked Read 5 Done Sent Done false
27891 7632 Acked Read 4 Done Sent Done false
27892 7633 Acked Flush 0 Done Sent Done false
27893 7634 Acked Read 2 Done Sent Done false
27894 7635 Acked Flush 0 Done Sent Done false
27895 7636 Acked Write 9 Done Sent Done false
27896 7637 Acked Write 7 Done Sent Done false
27897 7638 Acked Write 10 Done Sent Done false
27898 7639 Acked Read 9 Done Sent Done false
27899 7640 Acked Write 8 Done Sent Done false
27900 STATES DS:0 DS:1 DS:2 TOTAL
27901 Sent 0 301 0 301
27902 Done 301 0 301 602
27903 Skipped 0 0 0 0
27904 Error 0 0 0 0
27905 Last Flush: Some(JobId(7635)) Some(JobId(7339)) Some(JobId(7635))
27906 Downstairs last five completed: 7339 7338 7337 7336 7335
27907 Upstairs last five completed: 7640 7639 7638 7637 7636
27908 CLIENT: Up:0 ds:301 act:2
279092025-03-17T22:25:08.179ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
279102025-03-17T22:25:09.432ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
279112025-03-17T22:25:10.169ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
279122025-03-17T22:25:11.172ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
27913 ----------------------------------------------------------------
27914 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27915 States: Active Offline Active
27916 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27917 7340 Acked Read 7 Done Sent Done false
27918 7341 Acked Write 7 Done Sent Done false
27919 7342 Acked Flush 0 Done Sent Done false
27920 7343 Acked Read 3 Done Sent Done false
27921 7344 Acked Read 2 Done Sent Done false
27922 7345 Acked Flush 0 Done Sent Done false
27923 7346 Acked Write 8 Done Sent Done false
27924 7347 Acked Read 5 Done Sent Done false
27925 7348 Acked Flush 0 Done Sent Done false
27926 7349 Acked Write 4 Done Sent Done false
27927 7350 Acked Read 6 Done Sent Done false
27928 7351 Acked Read 7 Done Sent Done false
27929 7352 Acked Flush 0 Done Sent Done false
27930 7353 Acked Read 7 Done Sent Done false
27931 7354 Acked Write 5 Done Sent Done false
27932 7355 Acked Write 7 Done Sent Done false
27933 7356 Acked Write 10 Done Sent Done false
27934 7357 Acked Read 1 Done Sent Done false
27935 7358 Acked Read 1 Done Sent Done false
27936 7359 Acked Read 8 Done Sent Done false
27937 7360 Acked Read 4 Done Sent Done false
27938 7361 Acked Write 2 Done Sent Done false
27939 7362 Acked Read 8 Done Sent Done false
27940 7363 Acked Read 6 Done Sent Done false
27941 7364 Acked Read 6 Done Sent Done false
27942 7365 Acked Write 2 Done Sent Done false
27943 7366 Acked Read 1 Done Sent Done false
27944 7367 Acked Read 6 Done Sent Done false
27945 7368 Acked Read 6 Done Sent Done false
27946 7369 Acked Write 3 Done Sent Done false
27947 7370 Acked Write 3 Done Sent Done false
27948 7371 Acked Read 1 Done Sent Done false
27949 7372 Acked Read 5 Done Sent Done false
27950 7373 Acked Flush 0 Done Sent Done false
27951 7374 Acked Read 1 Done Sent Done false
27952 7375 Acked Write 4 Done Sent Done false
27953 7376 Acked Read 5 Done Sent Done false
27954 7377 Acked Read 8 Done Sent Done false
27955 7378 Acked Write 3 Done Sent Done false
27956 7379 Acked Read 6 Done Sent Done false
27957 7380 Acked Flush 0 Done Sent Done false
27958 7381 Acked Read 3 Done Sent Done false
27959 7382 Acked Read 1 Done Sent Done false
27960 7383 Acked Flush 0 Done Sent Done false
27961 7384 Acked Read 2 Done Sent Done false
27962 7385 Acked Read 3 Done Sent Done false
27963 7386 Acked Flush 0 Done Sent Done false
27964 7387 Acked Read 7 Done Sent Done false
27965 7388 Acked Read 3 Done Sent Done false
27966 7389 Acked Read 9 Done Sent Done false
27967 7390 Acked Write 1 Done Sent Done false
27968 7391 Acked Read 1 Done Sent Done false
27969 7392 Acked Flush 0 Done Sent Done false
27970 7393 Acked Read 5 Done Sent Done false
27971 7394 Acked Read 1 Done Sent Done false
27972 7395 Acked Write 6 Done Sent Done false
27973 7396 Acked Read 5 Done Sent Done false
27974 7397 Acked Write 9 Done Sent Done false
27975 7398 Acked Write 3 Done Sent Done false
27976 7399 Acked Write 5 Done Sent Done false
27977 7400 Acked Write 1 Done Sent Done false
27978 7401 Acked Write 6 Done Sent Done false
27979 7402 Acked Read 1 Done Sent Done false
27980 7403 Acked Write 7 Done Sent Done false
27981 7404 Acked Read 8 Done Sent Done false
27982 7405 Acked Write 9 Done Sent Done false
27983 7406 Acked Write 2 Done Sent Done false
27984 7407 Acked Write 10 Done Sent Done false
27985 7408 Acked Read 5 Done Sent Done false
27986 7409 Acked Read 1 Done Sent Done false
27987 7410 Acked Read 10 Done Sent Done false
27988 7411 Acked Write 5 Done Sent Done false
27989 7412 Acked Write 8 Done Sent Done false
27990 7413 Acked Write 8 Done Sent Done false
27991 7414 Acked Write 1 Done Sent Done false
27992 7415 Acked Write 8 Done Sent Done false
27993 7416 Acked Read 2 Done Sent Done false
27994 7417 Acked Write 6 Done Sent Done false
27995 7418 Acked Read 4 Done Sent Done false
27996 7419 Acked Flush 0 Done Sent Done false
27997 7420 Acked Read 3 Done Sent Done false
27998 7421 Acked Write 7 Done Sent Done false
27999 7422 Acked Read 2 Done Sent Done false
28000 7423 Acked Read 7 Done Sent Done false
28001 7424 Acked Read 4 Done Sent Done false
28002 7425 Acked Read 2 Done Sent Done false
28003 7426 Acked Write 7 Done Sent Done false
28004 7427 Acked Write 1 Done Sent Done false
28005 7428 Acked Write 6 Done Sent Done false
28006 7429 Acked Read 8 Done Sent Done false
28007 7430 Acked Write 8 Done Sent Done false
28008 7431 Acked Flush 0 Done Sent Done false
28009 7432 Acked Read 1 Done Sent Done false
28010 7433 Acked Read 9 Done Sent Done false
28011 7434 Acked Read 8 Done Sent Done false
28012 7435 Acked Read 4 Done Sent Done false
28013 7436 Acked Write 5 Done Sent Done false
28014 7437 Acked Flush 0 Done Sent Done false
28015 7438 Acked Write 9 Done Sent Done false
28016 7439 Acked Write 10 Done Sent Done false
28017 7440 Acked Read 10 Done Sent Done false
28018 7441 Acked Write 3 Done Sent Done false
28019 7442 Acked Flush 0 Done Sent Done false
28020 7443 Acked Read 5 Done Sent Done false
28021 7444 Acked Write 9 Done Sent Done false
28022 7445 Acked Read 2 Done Sent Done false
28023 7446 Acked Write 6 Done Sent Done false
28024 7447 Acked Read 4 Done Sent Done false
28025 7448 Acked Read 7 Done Sent Done false
28026 7449 Acked Write 7 Done Sent Done false
28027 7450 Acked Write 1 Done Sent Done false
28028 7451 Acked Write 7 Done Sent Done false
28029 7452 Acked Write 2 Done Sent Done false
28030 7453 Acked Read 9 Done Sent Done false
28031 7454 Acked Write 5 Done Sent Done false
28032 7455 Acked Read 6 Done Sent Done false
28033 7456 Acked Read 1 Done Sent Done false
28034 7457 Acked Write 4 Done Sent Done false
28035 7458 Acked Read 7 Done Sent Done false
28036 7459 Acked Write 3 Done Sent Done false
28037 7460 Acked Write 1 Done Sent Done false
28038 7461 Acked Read 1 Done Sent Done false
28039 7462 Acked Read 7 Done Sent Done false
28040 7463 Acked Read 9 Done Sent Done false
28041 7464 Acked Read 9 Done Sent Done false
28042 7465 Acked Write 5 Done Sent Done false
28043 7466 Acked Write 6 Done Sent Done false
28044 7467 Acked Read 10 Done Sent Done false
28045 7468 Acked Flush 0 Done Sent Done false
28046 7469 Acked Flush 0 Done Sent Done false
28047 7470 Acked Read 4 Done Sent Done false
28048 7471 Acked Read 5 Done Sent Done false
28049 7472 Acked Write 7 Done Sent Done false
28050 7473 Acked Read 5 Done Sent Done false
28051 7474 Acked Read 10 Done Sent Done false
28052 7475 Acked Read 4 Done Sent Done false
28053 7476 Acked Read 10 Done Sent Done false
28054 7477 Acked Write 6 Done Sent Done false
28055 7478 Acked Flush 0 Done Sent Done false
28056 7479 Acked Write 7 Done Sent Done false
28057 7480 Acked Read 7 Done Sent Done false
28058 7481 Acked Read 4 Done Sent Done false
28059 7482 Acked Write 3 Done Sent Done false
28060 7483 Acked Read 4 Done Sent Done false
28061 7484 Acked Write 10 Done Sent Done false
28062 7485 Acked Write 2 Done Sent Done false
28063 7486 Acked Read 2 Done Sent Done false
28064 7487 Acked Write 4 Done Sent Done false
28065 7488 Acked Read 1 Done Sent Done false
28066 7489 Acked Read 7 Done Sent Done false
28067 7490 Acked Read 8 Done Sent Done false
28068 7491 Acked Flush 0 Done Sent Done false
28069 7492 Acked Read 10 Done Sent Done false
28070 7493 Acked Write 4 Done Sent Done false
28071 7494 Acked Read 7 Done Sent Done false
28072 7495 Acked Write 10 Done Sent Done false
28073 7496 Acked Write 10 Done Sent Done false
28074 7497 Acked Read 7 Done Sent Done false
28075 7498 Acked Write 6 Done Sent Done false
28076 7499 Acked Write 1 Done Sent Done false
28077 7500 Acked Write 3 Done Sent Done false
28078 7501 Acked Read 9 Done Sent Done false
28079 7502 Acked Read 8 Done Sent Done false
28080 7503 Acked Read 4 Done Sent Done false
28081 7504 Acked Read 4 Done Sent Done false
28082 7505 Acked Read 6 Done Sent Done false
28083 7506 Acked Write 1 Done Sent Done false
28084 7507 Acked Write 4 Done Sent Done false
28085 7508 Acked Read 10 Done Sent Done false
28086 7509 Acked Flush 0 Done Sent Done false
28087 7510 Acked Write 3 Done Sent Done false
28088 7511 Acked Write 8 Done Sent Done false
28089 7512 Acked Flush 0 Done Sent Done false
28090 7513 Acked Read 4 Done Sent Done false
28091 7514 Acked Write 3 Done Sent Done false
28092 7515 Acked Read 4 Done Sent Done false
28093 7516 Acked Write 4 Done Sent Done false
28094 7517 Acked Write 1 Done Sent Done false
28095 7518 Acked Read 6 Done Sent Done false
28096 7519 Acked Write 3 Done Sent Done false
28097 7520 Acked Read 10 Done Sent Done false
28098 7521 Acked Write 9 Done Sent Done false
28099 7522 Acked Write 1 Done Sent Done false
28100 7523 Acked Read 1 Done Sent Done false
28101 7524 Acked Write 3 Done Sent Done false
28102 7525 Acked Flush 0 Done Sent Done false
28103 7526 Acked Write 3 Done Sent Done false
28104 7527 Acked Flush 0 Done Sent Done false
28105 7528 Acked Read 10 Done Sent Done false
28106 7529 Acked Flush 0 Done Sent Done false
28107 7530 Acked Flush 0 Done Sent Done false
28108 7531 Acked Flush 0 Done Sent Done false
28109 7532 Acked Write 4 Done Sent Done false
28110 7533 Acked Read 2 Done Sent Done false
28111 7534 Acked Read 2 Done Sent Done false
28112 7535 Acked Write 5 Done Sent Done false
28113 7536 Acked Write 1 Done Sent Done false
28114 7537 Acked Write 8 Done Sent Done false
28115 7538 Acked Write 1 Done Sent Done false
28116 7539 Acked Write 4 Done Sent Done false
28117 7540 Acked Write 3 Done Sent Done false
28118 7541 Acked Read 4 Done Sent Done false
28119 7542 Acked Flush 0 Done Sent Done false
28120 7543 Acked Write 9 Done Sent Done false
28121 7544 Acked Read 5 Done Sent Done false
28122 7545 Acked Write 10 Done Sent Done false
28123 7546 Acked Write 2 Done Sent Done false
28124 7547 Acked Write 5 Done Sent Done false
28125 7548 Acked Write 7 Done Sent Done false
28126 7549 Acked Write 7 Done Sent Done false
28127 7550 Acked Write 9 Done Sent Done false
28128 7551 Acked Write 9 Done Sent Done false
28129 7552 Acked Read 3 Done Sent Done false
28130 7553 Acked Write 7 Done Sent Done false
28131 7554 Acked Read 2 Done Sent Done false
28132 7555 Acked Write 9 Done Sent Done false
28133 7556 Acked Read 5 Done Sent Done false
28134 7557 Acked Read 9 Done Sent Done false
28135 7558 Acked Write 1 Done Sent Done false
28136 7559 Acked Read 4 Done Sent Done false
28137 7560 Acked Read 9 Done Sent Done false
28138 7561 Acked Read 6 Done Sent Done false
28139 7562 Acked Read 5 Done Sent Done false
28140 7563 Acked Write 3 Done Sent Done false
28141 7564 Acked Write 1 Done Sent Done false
28142 7565 Acked Write 3 Done Sent Done false
28143 7566 Acked Read 10 Done Sent Done false
28144 7567 Acked Read 3 Done Sent Done false
28145 7568 Acked Flush 0 Done Sent Done false
28146 7569 Acked Read 9 Done Sent Done false
28147 7570 Acked Read 1 Done Sent Done false
28148 7571 Acked Write 3 Done Sent Done false
28149 7572 Acked Read 10 Done Sent Done false
28150 7573 Acked Read 3 Done Sent Done false
28151 7574 Acked Flush 0 Done Sent Done false
28152 7575 Acked Read 5 Done Sent Done false
28153 7576 Acked Read 5 Done Sent Done false
28154 7577 Acked Write 5 Done Sent Done false
28155 7578 Acked Read 7 Done Sent Done false
28156 7579 Acked Write 7 Done Sent Done false
28157 7580 Acked Read 8 Done Sent Done false
28158 7581 Acked Write 9 Done Sent Done false
28159 7582 Acked Read 6 Done Sent Done false
28160 7583 Acked Write 2 Done Sent Done false
28161 7584 Acked Read 4 Done Sent Done false
28162 7585 Acked Write 5 Done Sent Done false
28163 7586 Acked Write 9 Done Sent Done false
28164 7587 Acked Flush 0 Done Sent Done false
28165 7588 Acked Read 6 Done Sent Done false
28166 7589 Acked Read 2 Done Sent Done false
28167 7590 Acked Write 9 Done Sent Done false
28168 7591 Acked Read 6 Done Sent Done false
28169 7592 Acked Read 8 Done Sent Done false
28170 7593 Acked Read 5 Done Sent Done false
28171 7594 Acked Read 7 Done Sent Done false
28172 7595 Acked Write 7 Done Sent Done false
28173 7596 Acked Read 3 Done Sent Done false
28174 7597 Acked Read 8 Done Sent Done false
28175 7598 Acked Read 8 Done Sent Done false
28176 7599 Acked Read 5 Done Sent Done false
28177 7600 Acked Read 2 Done Sent Done false
28178 7601 Acked Read 6 Done Sent Done false
28179 7602 Acked Read 4 Done Sent Done false
28180 7603 Acked Write 8 Done Sent Done false
28181 7604 Acked Write 1 Done Sent Done false
28182 7605 Acked Write 1 Done Sent Done false
28183 7606 Acked Write 2 Done Sent Done false
28184 7607 Acked Read 1 Done Sent Done false
28185 7608 Acked Read 9 Done Sent Done false
28186 7609 Acked Flush 0 Done Sent Done false
28187 7610 Acked Read 8 Done Sent Done false
28188 7611 Acked Read 8 Done Sent Done false
28189 7612 Acked Read 9 Done Sent Done false
28190 7613 Acked Write 10 Done Sent Done false
28191 7614 Acked Read 6 Done Sent Done false
28192 7615 Acked Read 5 Done Sent Done false
28193 7616 Acked Read 3 Done Sent Done false
28194 7617 Acked Write 6 Done Sent Done false
28195 7618 Acked Write 5 Done Sent Done false
28196 7619 Acked Write 6 Done Sent Done false
28197 7620 Acked Read 10 Done Sent Done false
28198 7621 Acked Read 4 Done Sent Done false
28199 7622 Acked Read 2 Done Sent Done false
28200 7623 Acked Read 2 Done Sent Done false
28201 7624 Acked Write 2 Done Sent Done false
28202 7625 Acked Write 4 Done Sent Done false
28203 7626 Acked Write 2 Done Sent Done false
28204 7627 Acked Flush 0 Done Sent Done false
28205 7628 Acked Flush 0 Done Sent Done false
28206 7629 Acked Write 9 Done Sent Done false
28207 7630 Acked Write 10 Done Sent Done false
28208 7631 Acked Read 5 Done Sent Done false
28209 7632 Acked Read 4 Done Sent Done false
28210 7633 Acked Flush 0 Done Sent Done false
28211 7634 Acked Read 2 Done Sent Done false
28212 7635 Acked Flush 0 Done Sent Done false
28213 7636 Acked Write 9 Done Sent Done false
28214 7637 Acked Write 7 Done Sent Done false
28215 7638 Acked Write 10 Done Sent Done false
28216 7639 Acked Read 9 Done Sent Done false
28217 7640 Acked Write 8 Done Sent Done false
28218 STATES DS:0 DS:1 DS:2 TOTAL
28219 Sent 0 301 0 301
28220 Done 301 0 301 602
28221 Skipped 0 0 0 0
28222 Error 0 0 0 0
28223 Last Flush: Some(JobId(7635)) Some(JobId(7339)) Some(JobId(7635))
28224 Downstairs last five completed: 7339 7338 7337 7336 7335
28225 Upstairs last five completed: 7640 7639 7638 7637 7636
28226 CLIENT: Up:0 ds:301 act:2
282272025-03-17T22:25:12.182ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
282282025-03-17T22:25:13.175ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
282292025-03-17T22:25:14.435ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
282302025-03-17T22:25:15.178ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
282312025-03-17T22:25:15.437ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
282322025-03-17T22:25:15.437ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:34615) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
282332025-03-17T22:25:15.439ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
282342025-03-17T22:25:15.439ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
282352025-03-17T22:25:15.439ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
282362025-03-17T22:25:15.439ZINFOcrucible: send last flush ID to this DS: Some(JobId(7339)) = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
282372025-03-17T22:25:15.439ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
282382025-03-17T22:25:15.439ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(7339)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
282392025-03-17T22:25:15.439ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(7339)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
28240 ----------------------------------------------------------------
28241 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:5
28242 States: Active Active Active
28243 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28244 7636 Acked Write 9 Done Done Done true
28245 7637 Acked Write 7 Done Done Done true
28246 7638 Acked Write 10 Done Done Done true
28247 7639 Acked Read 9 Done Done Done true
28248 7640 Acked Write 8 Done Done Done true
28249 STATES DS:0 DS:1 DS:2 TOTAL
28250 Sent 0 0 0 0
28251 Done 5 5 5 15
28252 Skipped 0 0 0 0
28253 Error 0 0 0 0
28254 Last Flush: Some(JobId(7635)) Some(JobId(7635)) Some(JobId(7635))
28255 Downstairs last five completed: 7635 7634 7633 7632 7631
28256 Upstairs last five completed: 7640 7639 7638 7637 7636
28257 CLIENT: Up:0 ds:5 act:3
28258 ----------------------------------------------------------------
28259 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
28260 Downstairs last five completed: 7641 7640 7639 7638 7637
28261 Upstairs last five completed: 7641 7640 7639 7638 7637
28262 CLIENT: Up:0 ds:0 act:3
28263 Replay: All jobs finished
28264 Main task has work to do, go find it
28265 got dsc Stop(1)
28266 stop 1
28267 Mar 17 22:25:19.684 INFO request completed[8820] Got stop action so:false kr:false
28268 , latency_us: 355, response_code: 204, uri: /stop/cid/1, method: POST, req_id: ed40593d-461f-4c8d-8e14-222e84dceb0d, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
282692025-03-17T22:25:19.685ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
282702025-03-17T22:25:19.685ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
282712025-03-17T22:25:19.685ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
282722025-03-17T22:25:19.685ZINFOcrucible: client task is exiting = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
28273 Mar 17 22:25:19.685 INFO request completed, latency_us: 157, response_code: 200, uri: /state/cid/1, method: GET, req_id: 105be472-3f90-4076-b144-840198c4878a, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
28274 [8820] Exited with: ExitStatus(unix_wait_status(9))
28275 [8820][1] reports Exit
28276 Mar 17 22:25:23.688 INFO request completed, latency_us: 225, response_code: 200, uri: /state/cid/1, method: GET, req_id: c0929523-802a-4913-bb7e-5669cf526aae, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
28277 001/300 Read block 1845 len 20480
28278 002/300 Read block 766 len 12288
28279 003/300 Write block 587 len 4096 data: 5
28280 004/300 Write block 4203 len 8192 data: 4 5
28281 005/300 Read block 2686 len 20480
28282 006/300 Read block 3813 len 4096
28283 007/300 Read block 3621 len 40960
28284 008/300 Read block 3769 len 4096
28285 009/300 Flush
28286 010/300 Read block 1004 len 28672
28287 011/300 Flush
28288 012/300 Write block 1341 len 32768 data: 3 5 4 5 4 4 6 6
28289 013/300 Write block 237 len 16384 data: 3 3 2 3
28290 014/300 Write block 2421 len 12288 data: 8 8 7
28291 015/300 Write block 1120 len 20480 data: 4 4 4 5 5
28292 016/300 Read block 2597 len 36864
28293 017/300 Read block 3619 len 20480
28294 018/300 Read block 4002 len 20480
28295 019/300 Read block 4709 len 24576
28296 020/300 Write block 3137 len 28672 data: 8 9 8 8 4 4 4
28297 021/300 Write block 4339 len 4096 data: 4
28298 022/300 Write block 2405 len 4096 data: 6
28299 023/300 Flush
28300 024/300 Read block 1947 len 20480
28301 025/300 Read block 452 len 12288
28302 026/300 Write block 4054 len 36864 data: 5 4 4 4 5 4 3 4 4
28303 027/300 Flush
28304 028/300 Read block 2813 len 32768
28305 029/300 Read block 4292 len 4096
28306 030/300 Read block 3409 len 20480
28307 031/300 Read block 4460 len 28672
28308 032/300 Read block 2268 len 20480
28309 033/300 Write block 268 len 4096 data: 5
28310 034/300 Write block 3941 len 16384 data: 6 8 7 7
28311 035/300 Read block 136 len 12288
28312 036/300 Read block 407 len 40960
28313 037/300 Write block 1330 len 20480 data: 7 8 7 7 7
28314 038/300 Write block 2025 len 20480 data: 5 5 5 5 3
28315 039/300 Read block 1594 len 16384
28316 040/300 Read block 1702 len 16384
28317 041/300 Read block 3957 len 28672
28318 042/300 Write block 2345 len 28672 data: 6 5 2 2 2 2 2
28319 043/300 Write block 3759 len 8192 data: 7 4
28320 044/300 Write block 1565 len 16384 data: 3 3 4 4
28321 045/300 Read block 4692 len 12288
28322 046/300 Read block 2394 len 12288
28323 047/300 Read block 4992 len 32768
28324 048/300 Read block 3567 len 36864
28325 049/300 Read block 1756 len 40960
28326 050/300 Write block 2190 len 20480 data: 4 4 6 6 6
28327 051/300 Read block 4593 len 8192
28328 052/300 Write block 2581 len 28672 data: 5 6 7 9 9 9 7
28329 053/300 Write block 1761 len 36864 data: 3 3 3 2 3 4 3 3 3
28330 054/300 Write block 1249 len 12288 data: 6 6 9
28331 055/300 Read block 366 len 36864
28332 056/300 Read block 4929 len 36864
28333 057/300 Write block 1800 len 4096 data: 5
28334 058/300 Read block 583 len 8192
28335 059/300 Read block 789 len 32768
28336 060/300 Write block 4170 len 28672 data: 5 6 6 5 5 7 8
28337 061/300 Read block 2284 len 20480
28338 062/300 Write block 2296 len 40960 data: 4 5 6 5 5 3 3 3 2 4
28339 063/300 Read block 2914 len 4096
28340 064/300 Write block 2781 len 32768 data: 4 5 6 6 6 5 4 5
28341 065/300 Read block 2808 len 32768
28342 066/300 Read block 1222 len 16384
28343 067/300 Write block 671 len 16384 data: 3 3 3 4
28344 068/300 Read block 3954 len 24576
28345 069/300 Read block 3416 len 4096
28346 070/300 Write block 4688 len 40960 data: 3 3 3 4 6 6 5 4 4 6
28347 071/300 Read block 2196 len 40960
28348 072/300 Write block 601 len 20480 data: 4 3 4 4 4
28349 073/300 Read block 4242 len 4096
28350 074/300 Flush
28351 075/300 Write block 1623 len 36864 data: 3 3 3 5 4 4 4 4 4
28352 076/300 Write block 4901 len 36864 data: 5 6 9 9 7 7 6 4 5
28353 077/300 Flush
28354 078/300 Write block 1631 len 36864 data: 5 5 6 6 8 7 6 6 7
28355 079/300 Write block 2096 len 24576 data: 3 3 4 5 5 5
28356 080/300 Write block 4712 len 4096 data: 7
28357 081/300 Write block 4012 len 28672 data: 3 4 5 5 5 6 5
28358 082/300 Read block 895 len 32768
28359 083/300 Write block 0 len 16384 data: 2 3 4 4
28360 084/300 Read block 4447 len 16384
28361 085/300 Write block 2807 len 36864 data: 2 2 3 3 2 2 2 2 2
28362 086/300 Read block 2099 len 16384
28363 087/300 Read block 2663 len 36864
28364 088/300 Write block 4323 len 36864 data: 6 6 7 8 8 8 7 6 6
28365 089/300 Read block 392 len 16384
28366 090/300 Write block 4907 len 4096 data: 7
28367 091/300 Write block 967 len 16384 data: 4 5 5 5
28368 092/300 Read block 3699 len 28672
28369 093/300 Write block 496 len 12288 data: 4 4 6
28370 094/300 Read block 747 len 32768
28371 095/300 Write block 3730 len 20480 data: 9 9 9 8 7
28372 096/300 Write block 2788 len 32768 data: 6 5 5 4 4 3 3 4
28373 097/300 Write block 3000 len 20480 data: 3 4 5 5 6
28374 098/300 Write block 3699 len 16384 data: 6 5 5 5
28375 099/300 Write block 2613 len 4096 data: 5
28376 100/300 Read block 1857 len 28672
28377 101/300 Write block 918 len 24576 data: 7 6 6 6 6 8
28378 102/300 Read block 50 len 36864
28379 103/300 Write block 930 len 8192 data: 6 6
28380 104/300 Write block 4808 len 28672 data: 4 4 4 5 4 3 6
28381 105/300 Read block 3720 len 8192
28382 106/300 Read block 2145 len 8192
28383 107/300 Flush
28384 108/300 Flush
28385 109/300 Read block 3828 len 16384
28386 110/300 Write block 4668 len 20480 data: 7 7 7 7 3
28387 111/300 Write block 711 len 8192 data: 6 6
28388 112/300 Read block 2664 len 16384
28389 113/300 Write block 1129 len 8192 data: 5 5
28390 114/300 Read block 581 len 16384
28391 115/300 Flush
28392 116/300 Write block 4171 len 24576 data: 7 7 6 6 8 9
28393 117/300 Read block 887 len 24576
28394 118/300 Flush
28395 119/300 Read block 12 len 32768
28396 120/300 Write block 78 len 16384 data: 6 5 4 5
28397 121/300 Write block 1831 len 8192 data: 5 4
28398 122/300 Write block 1001 len 20480 data: 4 4 4 2 2
28399 123/300 Read block 3125 len 4096
28400 124/300 Read block 1139 len 24576
28401 125/300 Read block 4525 len 24576
28402 126/300 Write block 2969 len 40960 data: 9 8 7 5 5 3 3 3 3 4
28403 127/300 Write block 2277 len 36864 data: 6 6 6 6 7 4 4 4 3
28404 128/300 Write block 2595 len 40960 data: 4 4 7 7 6 7 5 8 8 7
28405 129/300 Read block 2652 len 24576
28406 130/300 Write block 723 len 4096 data: 3
28407 131/300 Read block 188 len 8192
28408 132/300 Flush
28409 133/300 Write block 3453 len 12288 data: 5 4 4
28410 134/300 Read block 1565 len 20480
28411 135/300 Read block 4978 len 12288
28412 136/300 Write block 1706 len 32768 data: 6 6 6 8 12 9 7 8
28413 137/300 Write block 3724 len 28672 data: 6 4 5 6 6 6 10
28414 138/300 Read block 3622 len 12288
28415 139/300 Read block 179 len 24576
28416 140/300 Read block 778 len 8192
28417 141/300 Read block 4669 len 40960
28418 142/300 Flush
28419 143/300 Write block 2472 len 28672 data: 5 4 5 5 6 6 5
28420 144/300 Read block 842 len 40960
28421 145/300 Read block 261 len 36864
28422 146/300 Read block 2761 len 4096
28423 147/300 Write block 4923 len 36864 data: 4 4 5 5 5 5 5 4 6
28424 148/300 Read block 832 len 24576
28425 149/300 Read block 2540 len 36864
28426 150/300 Read block 1084 len 4096
28427 151/300 Read block 1220 len 8192
28428 152/300 Read block 4548 len 24576
28429 153/300 Read block 2564 len 36864
28430 154/300 Write block 609 len 36864 data: 7 7 6 7 7 7 6 6 6
28431 155/300 Read block 169 len 36864
28432 156/300 Write block 4587 len 36864 data: 3 3 4 4 6 6 4 4 5
28433 157/300 Write block 882 len 24576 data: 3 4 4 3 4 5
28434 158/300 Read block 2988 len 8192
28435 159/300 Read block 3349 len 28672
28436 160/300 Read block 3901 len 12288
28437 161/300 Read block 2673 len 12288
28438 162/300 Flush
28439 163/300 Read block 3441 len 4096
28440 164/300 Read block 4581 len 8192
28441 165/300 Read block 2529 len 40960
28442 166/300 Read block 2457 len 36864
28443 167/300 Read block 1380 len 36864
28444 168/300 Read block 3073 len 40960
28445 169/300 Read block 2622 len 24576
28446 170/300 Read block 943 len 4096
28447 171/300 Read block 1286 len 16384
28448 172/300 Read block 619 len 8192
28449 173/300 Write block 2685 len 12288 data: 6 6 7
28450 174/300 Read block 3951 len 36864
28451 175/300 Read block 1584 len 40960
28452 176/300 Write block 1250 len 28672 data: 7 10 9 8 7 7 6
28453 177/300 Read block 1504 len 40960
28454 178/300 Read block 4860 len 36864
28455 179/300 Read block 2659 len 32768
28456 180/300 Write block 2546 len 40960 data: 6 4 5 6 6 5 6 6 6 7
28457 181/300 Write block 4070 len 36864 data: 3 4 4 5 7 8 7 7 6
28458 182/300 Read block 2719 len 24576
28459 183/300 Write block 3521 len 12288 data: 5 6 7
28460 184/300 Write block 4937 len 4096 data: 6
28461 185/300 Read block 3910 len 24576
28462 186/300 Read block 1360 len 8192
28463 187/300 Read block 1106 len 28672
28464 188/300 Write block 3431 len 4096 data: 8
28465 189/300 Read block 4848 len 20480
28466 190/300 Read block 4558 len 12288
28467 191/300 Write block 10 len 40960 data: 4 3 2 2 3 3 3 3 3 3
28468 192/300 Write block 3645 len 16384 data: 3 3 3 4
28469 193/300 Read block 2814 len 4096
28470 194/300 Write block 2970 len 8192 data: 9 8
28471 195/300 Write block 4342 len 36864 data: 5 5 4 5 5 5 4 3 4
28472 196/300 Write block 2221 len 28672 data: 4 5 5 6 6 6 7
28473 197/300 Read block 3486 len 36864
28474 198/300 Read block 1177 len 40960
28475 199/300 Write block 1157 len 36864 data: 5 6 6 4 4 4 4 4 4
28476 200/300 Read block 704 len 16384
28477 201/300 Write block 1058 len 4096 data: 5
28478 202/300 Write block 4136 len 40960 data: 4 5 4 5 6 5 6 5 4 6
28479 203/300 Read block 4735 len 16384
28480 204/300 Flush
28481 205/300 Write block 2129 len 40960 data: 2 3 2 3 2 2 4 4 5 5
28482 206/300 Flush
28483 207/300 Read block 3618 len 24576
28484 208/300 Read block 1190 len 28672
28485 209/300 Write block 3138 len 12288 data: 10 9 9
28486 210/300 Write block 2042 len 32768 data: 6 5 5 6 6 4 4 5
28487 211/300 Read block 3633 len 16384
28488 212/300 Read block 1457 len 32768
28489 213/300 Read block 4970 len 24576
28490 214/300 Write block 3017 len 32768 data: 7 6 6 6 7 7 7 7
28491 215/300 Read block 3390 len 32768
28492 216/300 Read block 3055 len 12288
28493 217/300 Write block 4070 len 12288 data: 4 5 5
28494 218/300 Read block 4472 len 12288
28495 219/300 Write block 4342 len 40960 data: 6 6 5 6 6 6 5 4 5 5
28496 220/300 Write block 3770 len 20480 data: 5 4 4 6 7
28497 221/300 Write block 2119 len 32768 data: 5 4 4 4 4 4 4 3
28498 222/300 Read block 3921 len 24576
28499 223/300 Write block 154 len 4096 data: 9
28500 224/300 Read block 2422 len 4096
28501 225/300 Write block 579 len 32768 data: 4 6 6 5 5 4 4 4
28502 226/300 Read block 4979 len 32768
28503 227/300 Write block 2605 len 36864 data: 8 7 8 8 6 7 6 4 6
28504 228/300 Read block 4130 len 36864
28505 229/300 Write block 3929 len 12288 data: 5 6 6
28506 230/300 Write block 1767 len 28672 data: 4 4 4 3 3 3 3
28507 231/300 Flush
28508 232/300 Write block 4509 len 16384 data: 5 5 6 5
28509 233/300 Read block 4985 len 12288
28510 234/300 Write block 310 len 32768 data: 5 6 4 4 5 5 7 7
28511 235/300 Flush
28512 236/300 Write block 3850 len 40960 data: 10 9 10 8 8 6 5 4 4 4
28513 237/300 Read block 4658 len 12288
28514 238/300 Read block 281 len 24576
28515 239/300 Write block 3740 len 36864 data: 4 5 5 5 5 6 6 6 5
28516 240/300 Write block 520 len 40960 data: 5 5 6 5 4 5 5 5 5 5
28517 241/300 Flush
28518 242/300 Read block 202 len 4096
28519 243/300 Read block 2402 len 4096
28520 244/300 Write block 1474 len 32768 data: 7 8 7 6 7 6 7 6
28521 245/300 Read block 266 len 20480
28522 246/300 Read block 1573 len 20480
28523 247/300 Flush
28524 248/300 Read block 778 len 28672
28525 249/300 Read block 2179 len 4096
28526 250/300 Write block 291 len 20480 data: 6 6 6 6 7
28527 251/300 Write block 3015 len 40960 data: 4 5 8 7 7 7 8 8 8 8
28528 252/300 Read block 516 len 4096
28529 253/300 Read block 4820 len 24576
28530 254/300 Flush
28531 255/300 Write block 3272 len 28672 data: 4 5 5 5 4 3 5
28532 256/300 Read block 1699 len 40960
28533 257/300 Flush
28534 258/300 Write block 2842 len 20480 data: 4 5 6 6 5
28535 259/300 Write block 2045 len 16384 data: 7 7 5 5
28536 260/300 Read block 446 len 16384
28537 261/300 Read block 4228 len 32768
28538 262/300 Flush
28539 263/300 Read block 99 len 16384
28540 264/300 Read block 716 len 28672
28541 265/300 Write block 4757 len 8192 data: 7 5
28542 266/300 Read block 984 len 8192
28543 267/300 Read block 74 len 16384
28544 268/300 Write block 3364 len 36864 data: 3 2 2 2 2 3 5 5 6
28545 269/300 Write block 399 len 12288 data: 4 5 4
28546 270/300 Write block 1892 len 24576 data: 5 4 5 6 6 6
28547 271/300 Write block 3718 len 32768 data: 5 4 5 5 6 8 7 5
28548 272/300 Read block 1177 len 32768
28549 273/300 Read block 1226 len 40960
28550 274/300 Read block 2706 len 16384
28551 275/300 Read block 4612 len 8192
28552 276/300 Write block 2802 len 12288 data: 4 4 3
28553 277/300 Read block 1058 len 20480
28554 278/300 Write block 1414 len 32768 data: 6 5 5 5 4 4 4 2
28555 279/300 Read block 1356 len 28672
28556 280/300 Write block 2417 len 32768 data: 8 8 9 11 9 9 8 5
28557 281/300 Write block 4714 len 16384 data: 9 8 8 8
28558 282/300 Flush
28559 283/300 Read block 2725 len 28672
28560 284/300 Read block 1823 len 24576
28561 285/300 Read block 1112 len 20480
28562 286/300 Read block 1542 len 20480
28563 287/300 Flush
28564 288/300 Write block 2247 len 36864 data: 7 8 9 6 5 5 4 3 2
28565 289/300 Read block 1473 len 40960
28566 290/300 Read block 3500 len 40960
28567 291/300 Flush
28568 292/300 Read block 3619 len 20480
28569 293/300 Write block 36 len 8192 data: 5 6
28570 294/300 Read block 3321 len 12288
28571 295/300 Write block 4722 len 20480 data: 5 4 3 2 3
28572 296/300 Read block 1161 len 20480
28573 297/300 Read block 2601 len 4096
28574 298/300 Read block 4744 len 24576
28575 299/300 Flush
28576 300/300 Read block 4163 len 28672
28577 301/300 Read block 1269 len 28672
28578 Main task has work to do, go find it
28579 got dsc Start(1)
28580 start 1
28581 Mar 17 22:25:23.935 INFO request completed, latency_us: 217, response_code: 204, uri: /start/cid/1, method: POST, req_id: 855ca868-4529-4f90-8fc0-3a93d5719e2f, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
28582 Replay: started 1, returned:Ok(())
28583 ----------------------------------------------------------------
28584 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
28585 States: Active Offline Active
28586 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28587 7642 Acked Read 5 Done Sent Done false
28588 7643 Acked Read 3 Done Sent Done false
28589 7644 Acked Write 1 Done Sent Done false
28590 7645 Acked Write 2 Done Sent Done false
28591 7646 Acked Read 5 Done Sent Done false
28592 7647 Acked Read 1 Done Sent Done false
28593 7648 Acked Read 10 Done Sent Done false
28594 7649 Acked Read 1 Done Sent Done false
28595 7650 Acked Flush 0 Done Sent Done false
28596 7651 Acked Read 7 Done Sent Done false
28597 7652 Acked Flush 0 Done Sent Done false
28598 7653 Acked Write 8 Done Sent Done false
28599 7654 Acked Write 4 Done Sent Done false
28600 7655 Acked Write 3 Done Sent Done false
28601 7656 Acked Write 5 Done Sent Done false
28602 7657 Acked Read 9 Done Sent Done false
28603 7658 Acked Read 5 Done Sent Done false
28604 7659 Acked Read 5 Done Sent Done false
28605 7660 Acked Read 6 Done Sent Done false
28606 7661 Acked Write 7 Done Sent Done false
28607 7662 Acked Write 1 Done Sent Done false
28608 7663 Acked Write 1 Done Sent Done false
28609 7664 Acked Flush 0 Done Sent Done false
28610 7665 Acked Read 5 Done Sent Done false
28611 7666 Acked Read 3 Done Sent Done false
28612 7667 Acked Write 9 Done Sent Done false
28613 7668 Acked Flush 0 Done Sent Done false
28614 7669 Acked Read 8 Done Sent Done false
28615 7670 Acked Read 1 Done Sent Done false
28616 7671 Acked Read 5 Done Sent Done false
28617 7672 Acked Read 7 Done Sent Done false
28618 7673 Acked Read 5 Done Sent Done false
28619 7674 Acked Write 1 Done Sent Done false
28620 7675 Acked Write 4 Done Sent Done false
28621 7676 Acked Read 3 Done Sent Done false
28622 7677 Acked Read 10 Done Sent Done false
28623 7678 Acked Write 5 Done Sent Done false
28624 7679 Acked Write 5 Done Sent Done false
28625 7680 Acked Read 4 Done Sent Done false
28626 7681 Acked Read 4 Done Sent Done false
28627 7682 Acked Read 7 Done Sent Done false
28628 7683 Acked Write 7 Done Sent Done false
28629 7684 Acked Write 2 Done Sent Done false
28630 7685 Acked Write 4 Done Sent Done false
28631 7686 Acked Read 3 Done Sent Done false
28632 7687 Acked Read 3 Done Sent Done false
28633 7688 Acked Read 8 Done Sent Done false
28634 7689 Acked Read 9 Done Sent Done false
28635 7690 Acked Read 10 Done Sent Done false
28636 7691 Acked Write 5 Done Sent Done false
28637 7692 Acked Read 2 Done Sent Done false
28638 7693 Acked Write 7 Done Sent Done false
28639 7694 Acked Write 9 Done Sent Done false
28640 7695 Acked Write 3 Done Sent Done false
28641 7696 Acked Read 9 Done Sent Done false
28642 7697 Acked Read 9 Done Sent Done false
28643 7698 Acked Write 1 Done Sent Done false
28644 7699 Acked Read 2 Done Sent Done false
28645 7700 Acked Read 8 Done Sent Done false
28646 7701 Acked Write 7 Done Sent Done false
28647 7702 Acked Read 5 Done Sent Done false
28648 7703 Acked Write 10 Done Sent Done false
28649 7704 Acked Read 1 Done Sent Done false
28650 7705 Acked Write 8 Done Sent Done false
28651 7706 Acked Read 8 Done Sent Done false
28652 7707 Acked Read 4 Done Sent Done false
28653 7708 Acked Write 4 Done Sent Done false
28654 7709 Acked Read 6 Done Sent Done false
28655 7710 Acked Read 1 Done Sent Done false
28656 7711 Acked Write 10 Done Sent Done false
28657 7712 Acked Read 10 Done Sent Done false
28658 7713 Acked Write 5 Done Sent Done false
28659 7714 Acked Read 1 Done Sent Done false
28660 7715 Acked Flush 0 Done Sent Done false
28661 7716 Acked Write 9 Done Sent Done false
28662 7717 Acked Write 9 Done Sent Done false
28663 7718 Acked Flush 0 Done Sent Done false
28664 7719 Acked Write 9 Done Sent Done false
28665 7720 Acked Write 6 Done Sent Done false
28666 7721 Acked Write 1 Done Sent Done false
28667 7722 Acked Write 7 Done Sent Done false
28668 7723 Acked Read 8 Done Sent Done false
28669 7724 Acked Write 4 Done Sent Done false
28670 7725 Acked Read 4 Done Sent Done false
28671 7726 Acked Write 9 Done Sent Done false
28672 7727 Acked Read 4 Done Sent Done false
28673 7728 Acked Read 9 Done Sent Done false
28674 7729 Acked Write 9 Done Sent Done false
28675 7730 Acked Read 4 Done Sent Done false
28676 7731 Acked Write 1 Done Sent Done false
28677 7732 Acked Write 4 Done Sent Done false
28678 7733 Acked Read 7 Done Sent Done false
28679 7734 Acked Write 3 Done Sent Done false
28680 7735 Acked Read 8 Done Sent Done false
28681 7736 Acked Write 5 Done Sent Done false
28682 7737 Acked Write 8 Done Sent Done false
28683 7738 Acked Write 5 Done Sent Done false
28684 7739 Acked Write 4 Done Sent Done false
28685 7740 Acked Write 1 Done Sent Done false
28686 7741 Acked Read 7 Done Sent Done false
28687 7742 Acked Write 6 Done Sent Done false
28688 7743 Acked Read 9 Done Sent Done false
28689 7744 Acked Write 2 Done Sent Done false
28690 7745 Acked Write 7 Done Sent Done false
28691 7746 Acked Read 2 Done Sent Done false
28692 7747 Acked Read 2 Done Sent Done false
28693 7748 Acked Flush 0 Done Sent Done false
28694 7749 Acked Flush 0 Done Sent Done false
28695 7750 Acked Read 4 Done Sent Done false
28696 7751 Acked Write 5 Done Sent Done false
28697 7752 Acked Write 2 Done Sent Done false
28698 7753 Acked Read 4 Done Sent Done false
28699 7754 Acked Write 2 Done Sent Done false
28700 7755 Acked Read 4 Done Sent Done false
28701 7756 Acked Flush 0 Done Sent Done false
28702 7757 Acked Write 6 Done Sent Done false
28703 7758 Acked Read 6 Done Sent Done false
28704 7759 Acked Flush 0 Done Sent Done false
28705 7760 Acked Read 8 Done Sent Done false
28706 7761 Acked Write 4 Done Sent Done false
28707 7762 Acked Write 2 Done Sent Done false
28708 7763 Acked Write 5 Done Sent Done false
28709 7764 Acked Read 1 Done Sent Done false
28710 7765 Acked Read 6 Done Sent Done false
28711 7766 Acked Read 6 Done Sent Done false
28712 7767 Acked Write 10 Done Sent Done false
28713 7768 Acked Write 9 Done Sent Done false
28714 7769 Acked Write 10 Done Sent Done false
28715 7770 Acked Read 6 Done Sent Done false
28716 7771 Acked Write 1 Done Sent Done false
28717 7772 Acked Read 2 Done Sent Done false
28718 7773 Acked Flush 0 Done Sent Done false
28719 7774 Acked Write 3 Done Sent Done false
28720 7775 Acked Read 5 Done Sent Done false
28721 7776 Acked Read 3 Done Sent Done false
28722 7777 Acked Write 8 Done Sent Done false
28723 7778 Acked Write 7 Done Sent Done false
28724 7779 Acked Read 3 Done Sent Done false
28725 7780 Acked Read 6 Done Sent Done false
28726 7781 Acked Read 2 Done Sent Done false
28727 7782 Acked Read 10 Done Sent Done false
28728 7783 Acked Flush 0 Done Sent Done false
28729 7784 Acked Write 7 Done Sent Done false
28730 7785 Acked Read 10 Done Sent Done false
28731 7786 Acked Read 9 Done Sent Done false
28732 7787 Acked Read 1 Done Sent Done false
28733 7788 Acked Write 9 Done Sent Done false
28734 7789 Acked Read 6 Done Sent Done false
28735 7790 Acked Read 9 Done Sent Done false
28736 7791 Acked Read 1 Done Sent Done false
28737 7792 Acked Read 2 Done Sent Done false
28738 7793 Acked Read 6 Done Sent Done false
28739 7794 Acked Read 9 Done Sent Done false
28740 7795 Acked Write 9 Done Sent Done false
28741 7796 Acked Read 9 Done Sent Done false
28742 7797 Acked Write 9 Done Sent Done false
28743 7798 Acked Write 6 Done Sent Done false
28744 7799 Acked Read 2 Done Sent Done false
28745 7800 Acked Read 7 Done Sent Done false
28746 7801 Acked Read 3 Done Sent Done false
28747 7802 Acked Read 3 Done Sent Done false
28748 7803 Acked Flush 0 Done Sent Done false
28749 7804 Acked Read 1 Done Sent Done false
28750 7805 Acked Read 2 Done Sent Done false
28751 7806 Acked Read 10 Done Sent Done false
28752 7807 Acked Read 9 Done Sent Done false
28753 7808 Acked Read 9 Done Sent Done false
28754 7809 Acked Read 10 Done Sent Done false
28755 7810 Acked Read 6 Done Sent Done false
28756 7811 Acked Read 1 Done Sent Done false
28757 7812 Acked Read 4 Done Sent Done false
28758 7813 Acked Read 2 Done Sent Done false
28759 7814 Acked Write 3 Done Sent Done false
28760 7815 Acked Read 9 Done Sent Done false
28761 7816 Acked Read 10 Done Sent Done false
28762 7817 Acked Write 7 Done Sent Done false
28763 7818 Acked Read 10 Done Sent Done false
28764 7819 Acked Read 9 Done Sent Done false
28765 7820 Acked Read 8 Done Sent Done false
28766 7821 Acked Write 10 Done Sent Done false
28767 7822 Acked Write 9 Done Sent Done false
28768 7823 Acked Read 6 Done Sent Done false
28769 7824 Acked Write 3 Done Sent Done false
28770 7825 Acked Write 1 Done Sent Done false
28771 7826 Acked Read 6 Done Sent Done false
28772 7827 Acked Read 2 Done Sent Done false
28773 7828 Acked Read 7 Done Sent Done false
28774 7829 Acked Write 1 Done Sent Done false
28775 7830 Acked Read 5 Done Sent Done false
28776 7831 Acked Read 3 Done Sent Done false
28777 7832 Acked Write 10 Done Sent Done false
28778 7833 Acked Write 4 Done Sent Done false
28779 7834 Acked Read 1 Done Sent Done false
28780 7835 Acked Write 2 Done Sent Done false
28781 7836 Acked Write 9 Done Sent Done false
28782 7837 Acked Write 7 Done Sent Done false
28783 7838 Acked Read 9 Done Sent Done false
28784 7839 Acked Read 10 Done Sent Done false
28785 7840 Acked Write 9 Done Sent Done false
28786 7841 Acked Read 4 Done Sent Done false
28787 7842 Acked Write 1 Done Sent Done false
28788 7843 Acked Write 10 Done Sent Done false
28789 7844 Acked Read 4 Done Sent Done false
28790 7845 Acked Flush 0 Done Sent Done false
28791 7846 Acked Write 10 Done Sent Done false
28792 7847 Acked Flush 0 Done Sent Done false
28793 7848 Acked Read 6 Done Sent Done false
28794 7849 Acked Read 7 Done Sent Done false
28795 7850 Acked Write 3 Done Sent Done false
28796 7851 Acked Write 8 Done Sent Done false
28797 7852 Acked Read 4 Done Sent Done false
28798 7853 Acked Read 8 Done Sent Done false
28799 7854 Acked Read 6 Done Sent Done false
28800 7855 Acked Write 8 Done Sent Done false
28801 7856 Acked Read 8 Done Sent Done false
28802 7857 Acked Read 3 Done Sent Done false
28803 7858 Acked Write 3 Done Sent Done false
28804 7859 Acked Read 3 Done Sent Done false
28805 7860 Acked Write 10 Done Sent Done false
28806 7861 Acked Write 5 Done Sent Done false
28807 7862 Acked Write 8 Done Sent Done false
28808 7863 Acked Read 6 Done Sent Done false
28809 7864 Acked Write 1 Done Sent Done false
28810 7865 Acked Read 1 Done Sent Done false
28811 7866 Acked Write 8 Done Sent Done false
28812 7867 Acked Read 8 Done Sent Done false
28813 7868 Acked Write 9 Done Sent Done false
28814 7869 Acked Read 9 Done Sent Done false
28815 7870 Acked Write 3 Done Sent Done false
28816 7871 Acked Write 7 Done Sent Done false
28817 7872 Acked Flush 0 Done Sent Done false
28818 7873 Acked Write 4 Done Sent Done false
28819 7874 Acked Read 3 Done Sent Done false
28820 7875 Acked Write 8 Done Sent Done false
28821 7876 Acked Flush 0 Done Sent Done false
28822 7877 Acked Write 10 Done Sent Done false
28823 7878 Acked Read 3 Done Sent Done false
28824 7879 Acked Read 6 Done Sent Done false
28825 7880 Acked Write 9 Done Sent Done false
28826 7881 Acked Write 10 Done Sent Done false
28827 7882 Acked Flush 0 Done Sent Done false
28828 7883 Acked Read 1 Done Sent Done false
28829 7884 Acked Read 1 Done Sent Done false
28830 7885 Acked Write 8 Done Sent Done false
28831 7886 Acked Read 5 Done Sent Done false
28832 7887 Acked Read 5 Done Sent Done false
28833 7888 Acked Flush 0 Done Sent Done false
28834 7889 Acked Read 7 Done Sent Done false
28835 7890 Acked Read 1 Done Sent Done false
28836 7891 Acked Write 5 Done Sent Done false
28837 7892 Acked Write 10 Done Sent Done false
28838 7893 Acked Read 1 Done Sent Done false
28839 7894 Acked Read 6 Done Sent Done false
28840 7895 Acked Flush 0 Done Sent Done false
28841 7896 Acked Write 7 Done Sent Done false
28842 7897 Acked Read 10 Done Sent Done false
28843 7898 Acked Flush 0 Done Sent Done false
28844 7899 Acked Write 5 Done Sent Done false
28845 7900 Acked Write 4 Done Sent Done false
28846 7901 Acked Read 4 Done Sent Done false
28847 7902 Acked Read 8 Done Sent Done false
28848 7903 Acked Flush 0 Done Sent Done false
28849 7904 Acked Read 4 Done Sent Done false
28850 7905 Acked Read 7 Done Sent Done false
28851 7906 Acked Write 2 Done Sent Done false
28852 7907 Acked Read 2 Done Sent Done false
28853 7908 Acked Read 4 Done Sent Done false
28854 7909 Acked Write 9 Done Sent Done false
28855 7910 Acked Write 3 Done Sent Done false
28856 7911 Acked Write 6 Done Sent Done false
28857 7912 Acked Write 8 Done Sent Done false
28858 7913 Acked Read 8 Done Sent Done false
28859 7914 Acked Read 10 Done Sent Done false
28860 7915 Acked Read 4 Done Sent Done false
28861 7916 Acked Read 2 Done Sent Done false
28862 7917 Acked Write 3 Done Sent Done false
28863 7918 Acked Read 5 Done Sent Done false
28864 7919 Acked Write 8 Done Sent Done false
28865 7920 Acked Read 7 Done Sent Done false
28866 7921 Acked Write 8 Done Sent Done false
28867 7922 Acked Write 4 Done Sent Done false
28868 7923 Acked Flush 0 Done Sent Done false
28869 7924 Acked Read 7 Done Sent Done false
28870 7925 Acked Read 6 Done Sent Done false
28871 7926 Acked Read 5 Done Sent Done false
28872 7927 Acked Read 5 Done Sent Done false
28873 7928 Acked Flush 0 Done Sent Done false
28874 7929 Acked Write 9 Done Sent Done false
28875 7930 Acked Read 10 Done Sent Done false
28876 7931 Acked Read 10 Done Sent Done false
28877 7932 Acked Flush 0 Done Sent Done false
28878 7933 Acked Read 5 Done Sent Done false
28879 7934 Acked Write 2 Done Sent Done false
28880 7935 Acked Read 3 Done Sent Done false
28881 7936 Acked Write 5 Done Sent Done false
28882 7937 Acked Read 5 Done Sent Done false
28883 7938 Acked Read 1 Done Sent Done false
28884 7939 Acked Read 6 Done Sent Done false
28885 7940 Acked Flush 0 Done Sent Done false
28886 7941 Acked Read 7 Done Sent Done false
28887 7942 Acked Read 7 Done Sent Done false
28888 STATES DS:0 DS:1 DS:2 TOTAL
28889 Sent 0 301 0 301
28890 Done 301 0 301 602
28891 Skipped 0 0 0 0
28892 Error 0 0 0 0
28893 Last Flush: Some(JobId(7940)) Some(JobId(7641)) Some(JobId(7940))
28894 Downstairs last five completed: 7641 7640 7639 7638 7637
28895 Upstairs last five completed: 7942 7941 7940 7939 7938
28896 CLIENT: Up:0 ds:301 act:2
288972025-03-17T22:25:24.436ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
288982025-03-17T22:25:25.194ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
28899 [8820] got start action
28900 [8820] I am going to restart
28901 Starting downstairs at port 8820
28902 Make output file at "/tmp/dsc/downstairs-8820.txt"
28903 [8820][1] reports Starting
28904 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1234)
28905 [8820][1] reports Running
289062025-03-17T22:25:26.197ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
289072025-03-17T22:25:27.199ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
28908 ----------------------------------------------------------------
28909 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
28910 States: Active Offline Active
28911 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28912 7642 Acked Read 5 Done Sent Done false
28913 7643 Acked Read 3 Done Sent Done false
28914 7644 Acked Write 1 Done Sent Done false
28915 7645 Acked Write 2 Done Sent Done false
28916 7646 Acked Read 5 Done Sent Done false
28917 7647 Acked Read 1 Done Sent Done false
28918 7648 Acked Read 10 Done Sent Done false
28919 7649 Acked Read 1 Done Sent Done false
28920 7650 Acked Flush 0 Done Sent Done false
28921 7651 Acked Read 7 Done Sent Done false
28922 7652 Acked Flush 0 Done Sent Done false
28923 7653 Acked Write 8 Done Sent Done false
28924 7654 Acked Write 4 Done Sent Done false
28925 7655 Acked Write 3 Done Sent Done false
28926 7656 Acked Write 5 Done Sent Done false
28927 7657 Acked Read 9 Done Sent Done false
28928 7658 Acked Read 5 Done Sent Done false
28929 7659 Acked Read 5 Done Sent Done false
28930 7660 Acked Read 6 Done Sent Done false
28931 7661 Acked Write 7 Done Sent Done false
28932 7662 Acked Write 1 Done Sent Done false
28933 7663 Acked Write 1 Done Sent Done false
28934 7664 Acked Flush 0 Done Sent Done false
28935 7665 Acked Read 5 Done Sent Done false
28936 7666 Acked Read 3 Done Sent Done false
28937 7667 Acked Write 9 Done Sent Done false
28938 7668 Acked Flush 0 Done Sent Done false
28939 7669 Acked Read 8 Done Sent Done false
28940 7670 Acked Read 1 Done Sent Done false
28941 7671 Acked Read 5 Done Sent Done false
28942 7672 Acked Read 7 Done Sent Done false
28943 7673 Acked Read 5 Done Sent Done false
28944 7674 Acked Write 1 Done Sent Done false
28945 7675 Acked Write 4 Done Sent Done false
28946 7676 Acked Read 3 Done Sent Done false
28947 7677 Acked Read 10 Done Sent Done false
28948 7678 Acked Write 5 Done Sent Done false
28949 7679 Acked Write 5 Done Sent Done false
28950 7680 Acked Read 4 Done Sent Done false
28951 7681 Acked Read 4 Done Sent Done false
28952 7682 Acked Read 7 Done Sent Done false
28953 7683 Acked Write 7 Done Sent Done false
28954 7684 Acked Write 2 Done Sent Done false
28955 7685 Acked Write 4 Done Sent Done false
28956 7686 Acked Read 3 Done Sent Done false
28957 7687 Acked Read 3 Done Sent Done false
28958 7688 Acked Read 8 Done Sent Done false
28959 7689 Acked Read 9 Done Sent Done false
28960 7690 Acked Read 10 Done Sent Done false
28961 7691 Acked Write 5 Done Sent Done false
28962 7692 Acked Read 2 Done Sent Done false
28963 7693 Acked Write 7 Done Sent Done false
28964 7694 Acked Write 9 Done Sent Done false
28965 7695 Acked Write 3 Done Sent Done false
28966 7696 Acked Read 9 Done Sent Done false
28967 7697 Acked Read 9 Done Sent Done false
28968 7698 Acked Write 1 Done Sent Done false
28969 7699 Acked Read 2 Done Sent Done false
28970 7700 Acked Read 8 Done Sent Done false
28971 7701 Acked Write 7 Done Sent Done false
28972 7702 Acked Read 5 Done Sent Done false
28973 7703 Acked Write 10 Done Sent Done false
28974 7704 Acked Read 1 Done Sent Done false
28975 7705 Acked Write 8 Done Sent Done false
28976 7706 Acked Read 8 Done Sent Done false
28977 7707 Acked Read 4 Done Sent Done false
28978 7708 Acked Write 4 Done Sent Done false
28979 7709 Acked Read 6 Done Sent Done false
28980 7710 Acked Read 1 Done Sent Done false
28981 7711 Acked Write 10 Done Sent Done false
28982 7712 Acked Read 10 Done Sent Done false
28983 7713 Acked Write 5 Done Sent Done false
28984 7714 Acked Read 1 Done Sent Done false
28985 7715 Acked Flush 0 Done Sent Done false
28986 7716 Acked Write 9 Done Sent Done false
28987 7717 Acked Write 9 Done Sent Done false
28988 7718 Acked Flush 0 Done Sent Done false
28989 7719 Acked Write 9 Done Sent Done false
28990 7720 Acked Write 6 Done Sent Done false
28991 7721 Acked Write 1 Done Sent Done false
28992 7722 Acked Write 7 Done Sent Done false
28993 7723 Acked Read 8 Done Sent Done false
28994 7724 Acked Write 4 Done Sent Done false
28995 7725 Acked Read 4 Done Sent Done false
28996 7726 Acked Write 9 Done Sent Done false
28997 7727 Acked Read 4 Done Sent Done false
28998 7728 Acked Read 9 Done Sent Done false
28999 7729 Acked Write 9 Done Sent Done false
29000 7730 Acked Read 4 Done Sent Done false
29001 7731 Acked Write 1 Done Sent Done false
29002 7732 Acked Write 4 Done Sent Done false
29003 7733 Acked Read 7 Done Sent Done false
29004 7734 Acked Write 3 Done Sent Done false
29005 7735 Acked Read 8 Done Sent Done false
29006 7736 Acked Write 5 Done Sent Done false
29007 7737 Acked Write 8 Done Sent Done false
29008 7738 Acked Write 5 Done Sent Done false
29009 7739 Acked Write 4 Done Sent Done false
29010 7740 Acked Write 1 Done Sent Done false
29011 7741 Acked Read 7 Done Sent Done false
29012 7742 Acked Write 6 Done Sent Done false
29013 7743 Acked Read 9 Done Sent Done false
29014 7744 Acked Write 2 Done Sent Done false
29015 7745 Acked Write 7 Done Sent Done false
29016 7746 Acked Read 2 Done Sent Done false
29017 7747 Acked Read 2 Done Sent Done false
29018 7748 Acked Flush 0 Done Sent Done false
29019 7749 Acked Flush 0 Done Sent Done false
29020 7750 Acked Read 4 Done Sent Done false
29021 7751 Acked Write 5 Done Sent Done false
29022 7752 Acked Write 2 Done Sent Done false
29023 7753 Acked Read 4 Done Sent Done false
29024 7754 Acked Write 2 Done Sent Done false
29025 7755 Acked Read 4 Done Sent Done false
29026 7756 Acked Flush 0 Done Sent Done false
29027 7757 Acked Write 6 Done Sent Done false
29028 7758 Acked Read 6 Done Sent Done false
29029 7759 Acked Flush 0 Done Sent Done false
29030 7760 Acked Read 8 Done Sent Done false
29031 7761 Acked Write 4 Done Sent Done false
29032 7762 Acked Write 2 Done Sent Done false
29033 7763 Acked Write 5 Done Sent Done false
29034 7764 Acked Read 1 Done Sent Done false
29035 7765 Acked Read 6 Done Sent Done false
29036 7766 Acked Read 6 Done Sent Done false
29037 7767 Acked Write 10 Done Sent Done false
29038 7768 Acked Write 9 Done Sent Done false
29039 7769 Acked Write 10 Done Sent Done false
29040 7770 Acked Read 6 Done Sent Done false
29041 7771 Acked Write 1 Done Sent Done false
29042 7772 Acked Read 2 Done Sent Done false
29043 7773 Acked Flush 0 Done Sent Done false
29044 7774 Acked Write 3 Done Sent Done false
29045 7775 Acked Read 5 Done Sent Done false
29046 7776 Acked Read 3 Done Sent Done false
29047 7777 Acked Write 8 Done Sent Done false
29048 7778 Acked Write 7 Done Sent Done false
29049 7779 Acked Read 3 Done Sent Done false
29050 7780 Acked Read 6 Done Sent Done false
29051 7781 Acked Read 2 Done Sent Done false
29052 7782 Acked Read 10 Done Sent Done false
29053 7783 Acked Flush 0 Done Sent Done false
29054 7784 Acked Write 7 Done Sent Done false
29055 7785 Acked Read 10 Done Sent Done false
29056 7786 Acked Read 9 Done Sent Done false
29057 7787 Acked Read 1 Done Sent Done false
29058 7788 Acked Write 9 Done Sent Done false
29059 7789 Acked Read 6 Done Sent Done false
29060 7790 Acked Read 9 Done Sent Done false
29061 7791 Acked Read 1 Done Sent Done false
29062 7792 Acked Read 2 Done Sent Done false
29063 7793 Acked Read 6 Done Sent Done false
29064 7794 Acked Read 9 Done Sent Done false
29065 7795 Acked Write 9 Done Sent Done false
29066 7796 Acked Read 9 Done Sent Done false
29067 7797 Acked Write 9 Done Sent Done false
29068 7798 Acked Write 6 Done Sent Done false
29069 7799 Acked Read 2 Done Sent Done false
29070 7800 Acked Read 7 Done Sent Done false
29071 7801 Acked Read 3 Done Sent Done false
29072 7802 Acked Read 3 Done Sent Done false
29073 7803 Acked Flush 0 Done Sent Done false
29074 7804 Acked Read 1 Done Sent Done false
29075 7805 Acked Read 2 Done Sent Done false
29076 7806 Acked Read 10 Done Sent Done false
29077 7807 Acked Read 9 Done Sent Done false
29078 7808 Acked Read 9 Done Sent Done false
29079 7809 Acked Read 10 Done Sent Done false
29080 7810 Acked Read 6 Done Sent Done false
29081 7811 Acked Read 1 Done Sent Done false
29082 7812 Acked Read 4 Done Sent Done false
29083 7813 Acked Read 2 Done Sent Done false
29084 7814 Acked Write 3 Done Sent Done false
29085 7815 Acked Read 9 Done Sent Done false
29086 7816 Acked Read 10 Done Sent Done false
29087 7817 Acked Write 7 Done Sent Done false
29088 7818 Acked Read 10 Done Sent Done false
29089 7819 Acked Read 9 Done Sent Done false
29090 7820 Acked Read 8 Done Sent Done false
29091 7821 Acked Write 10 Done Sent Done false
29092 7822 Acked Write 9 Done Sent Done false
29093 7823 Acked Read 6 Done Sent Done false
29094 7824 Acked Write 3 Done Sent Done false
29095 7825 Acked Write 1 Done Sent Done false
29096 7826 Acked Read 6 Done Sent Done false
29097 7827 Acked Read 2 Done Sent Done false
29098 7828 Acked Read 7 Done Sent Done false
29099 7829 Acked Write 1 Done Sent Done false
29100 7830 Acked Read 5 Done Sent Done false
29101 7831 Acked Read 3 Done Sent Done false
29102 7832 Acked Write 10 Done Sent Done false
29103 7833 Acked Write 4 Done Sent Done false
29104 7834 Acked Read 1 Done Sent Done false
29105 7835 Acked Write 2 Done Sent Done false
29106 7836 Acked Write 9 Done Sent Done false
29107 7837 Acked Write 7 Done Sent Done false
29108 7838 Acked Read 9 Done Sent Done false
29109 7839 Acked Read 10 Done Sent Done false
29110 7840 Acked Write 9 Done Sent Done false
29111 7841 Acked Read 4 Done Sent Done false
29112 7842 Acked Write 1 Done Sent Done false
29113 7843 Acked Write 10 Done Sent Done false
29114 7844 Acked Read 4 Done Sent Done false
29115 7845 Acked Flush 0 Done Sent Done false
29116 7846 Acked Write 10 Done Sent Done false
29117 7847 Acked Flush 0 Done Sent Done false
29118 7848 Acked Read 6 Done Sent Done false
29119 7849 Acked Read 7 Done Sent Done false
29120 7850 Acked Write 3 Done Sent Done false
29121 7851 Acked Write 8 Done Sent Done false
29122 7852 Acked Read 4 Done Sent Done false
29123 7853 Acked Read 8 Done Sent Done false
29124 7854 Acked Read 6 Done Sent Done false
29125 7855 Acked Write 8 Done Sent Done false
29126 7856 Acked Read 8 Done Sent Done false
29127 7857 Acked Read 3 Done Sent Done false
29128 7858 Acked Write 3 Done Sent Done false
29129 7859 Acked Read 3 Done Sent Done false
29130 7860 Acked Write 10 Done Sent Done false
29131 7861 Acked Write 5 Done Sent Done false
29132 7862 Acked Write 8 Done Sent Done false
29133 7863 Acked Read 6 Done Sent Done false
29134 7864 Acked Write 1 Done Sent Done false
29135 7865 Acked Read 1 Done Sent Done false
29136 7866 Acked Write 8 Done Sent Done false
29137 7867 Acked Read 8 Done Sent Done false
29138 7868 Acked Write 9 Done Sent Done false
29139 7869 Acked Read 9 Done Sent Done false
29140 7870 Acked Write 3 Done Sent Done false
29141 7871 Acked Write 7 Done Sent Done false
29142 7872 Acked Flush 0 Done Sent Done false
29143 7873 Acked Write 4 Done Sent Done false
29144 7874 Acked Read 3 Done Sent Done false
29145 7875 Acked Write 8 Done Sent Done false
29146 7876 Acked Flush 0 Done Sent Done false
29147 7877 Acked Write 10 Done Sent Done false
29148 7878 Acked Read 3 Done Sent Done false
29149 7879 Acked Read 6 Done Sent Done false
29150 7880 Acked Write 9 Done Sent Done false
29151 7881 Acked Write 10 Done Sent Done false
29152 7882 Acked Flush 0 Done Sent Done false
29153 7883 Acked Read 1 Done Sent Done false
29154 7884 Acked Read 1 Done Sent Done false
29155 7885 Acked Write 8 Done Sent Done false
29156 7886 Acked Read 5 Done Sent Done false
29157 7887 Acked Read 5 Done Sent Done false
29158 7888 Acked Flush 0 Done Sent Done false
29159 7889 Acked Read 7 Done Sent Done false
29160 7890 Acked Read 1 Done Sent Done false
29161 7891 Acked Write 5 Done Sent Done false
29162 7892 Acked Write 10 Done Sent Done false
29163 7893 Acked Read 1 Done Sent Done false
29164 7894 Acked Read 6 Done Sent Done false
29165 7895 Acked Flush 0 Done Sent Done false
29166 7896 Acked Write 7 Done Sent Done false
29167 7897 Acked Read 10 Done Sent Done false
29168 7898 Acked Flush 0 Done Sent Done false
29169 7899 Acked Write 5 Done Sent Done false
29170 7900 Acked Write 4 Done Sent Done false
29171 7901 Acked Read 4 Done Sent Done false
29172 7902 Acked Read 8 Done Sent Done false
29173 7903 Acked Flush 0 Done Sent Done false
29174 7904 Acked Read 4 Done Sent Done false
29175 7905 Acked Read 7 Done Sent Done false
29176 7906 Acked Write 2 Done Sent Done false
29177 7907 Acked Read 2 Done Sent Done false
29178 7908 Acked Read 4 Done Sent Done false
29179 7909 Acked Write 9 Done Sent Done false
29180 7910 Acked Write 3 Done Sent Done false
29181 7911 Acked Write 6 Done Sent Done false
29182 7912 Acked Write 8 Done Sent Done false
29183 7913 Acked Read 8 Done Sent Done false
29184 7914 Acked Read 10 Done Sent Done false
29185 7915 Acked Read 4 Done Sent Done false
29186 7916 Acked Read 2 Done Sent Done false
29187 7917 Acked Write 3 Done Sent Done false
29188 7918 Acked Read 5 Done Sent Done false
29189 7919 Acked Write 8 Done Sent Done false
29190 7920 Acked Read 7 Done Sent Done false
29191 7921 Acked Write 8 Done Sent Done false
29192 7922 Acked Write 4 Done Sent Done false
29193 7923 Acked Flush 0 Done Sent Done false
29194 7924 Acked Read 7 Done Sent Done false
29195 7925 Acked Read 6 Done Sent Done false
29196 7926 Acked Read 5 Done Sent Done false
29197 7927 Acked Read 5 Done Sent Done false
29198 7928 Acked Flush 0 Done Sent Done false
29199 7929 Acked Write 9 Done Sent Done false
29200 7930 Acked Read 10 Done Sent Done false
29201 7931 Acked Read 10 Done Sent Done false
29202 7932 Acked Flush 0 Done Sent Done false
29203 7933 Acked Read 5 Done Sent Done false
29204 7934 Acked Write 2 Done Sent Done false
29205 7935 Acked Read 3 Done Sent Done false
29206 7936 Acked Write 5 Done Sent Done false
29207 7937 Acked Read 5 Done Sent Done false
29208 7938 Acked Read 1 Done Sent Done false
29209 7939 Acked Read 6 Done Sent Done false
29210 7940 Acked Flush 0 Done Sent Done false
29211 7941 Acked Read 7 Done Sent Done false
29212 7942 Acked Read 7 Done Sent Done false
29213 STATES DS:0 DS:1 DS:2 TOTAL
29214 Sent 0 301 0 301
29215 Done 301 0 301 602
29216 Skipped 0 0 0 0
29217 Error 0 0 0 0
29218 Last Flush: Some(JobId(7940)) Some(JobId(7641)) Some(JobId(7940))
29219 Downstairs last five completed: 7641 7640 7639 7638 7637
29220 Upstairs last five completed: 7942 7941 7940 7939 7938
29221 CLIENT: Up:0 ds:301 act:2
292222025-03-17T22:25:28.439ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
292232025-03-17T22:25:29.438ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
292242025-03-17T22:25:29.686ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
292252025-03-17T22:25:29.686ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58597) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
292262025-03-17T22:25:29.687ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
292272025-03-17T22:25:29.688ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
292282025-03-17T22:25:29.688ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
292292025-03-17T22:25:29.688ZINFOcrucible: send last flush ID to this DS: Some(JobId(7641)) = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
292302025-03-17T22:25:29.688ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
292312025-03-17T22:25:29.688ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(7641)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
292322025-03-17T22:25:29.688ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(7641)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
29233 ----------------------------------------------------------------
29234 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
29235 Downstairs last five completed: 7943 7942 7941 7940 7939
29236 Upstairs last five completed: 7943 7942 7941 7940 7939
29237 CLIENT: Up:0 ds:0 act:3
29238 Replay: All jobs finished
29239 Main task has work to do, go find it
29240 got dsc Stop(2)
29241 stop 2
29242 Mar 17 22:25:31.939 INFO request completed, latency_us: 321, response_code: 204, uri: /stop/cid/2, method: POST, req_id: ee7956cb-4668-4846-aa1a-80cbc355bef7, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
29243 [8830] Got stop action so:false kr:false
29244 Mar 17 22:25:31.940 INFO request completed, latency_us: 184, response_code: 200, uri: /state/cid/2, method: GET, req_id: d46c46c9-efdd-4c04-b23e-edf2f00545c0, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
292452025-03-17T22:25:31.940ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
292462025-03-17T22:25:31.940ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
292472025-03-17T22:25:31.940ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
292482025-03-17T22:25:31.940ZINFOcrucible: client task is exiting = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
29249 [8830] Exited with: ExitStatus(unix_wait_status(9))
29250 [8830][2] reports Exit
29251 Mar 17 22:25:35.942 INFO request completed, latency_us: 194, response_code: 200, uri: /state/cid/2, method: GET, req_id: e439b4a6-f076-441f-9be0-29f454d0119f, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
29252 001/300 Write block 3876 len 32768 data: 8 8 7 6 5 5 4 5
29253 002/300 Read block 1259 len 4096
29254 003/300 Read block 3522 len 40960
29255 004/300 Read block 1331 len 36864
29256 005/300 Read block 2711 len 40960
29257 006/300 Write block 1251 len 12288 data: 11 10 9
29258 007/300 Write block 4034 len 8192 data: 6 7
29259 008/300 Write block 3451 len 4096 data: 5
29260 009/300 Read block 335 len 40960
29261 010/300 Write block 1179 len 16384 data: 4 4 4 3
29262 011/300 Write block 1238 len 12288 data: 4 5 6
29263 012/300 Read block 4090 len 12288
29264 013/300 Read block 294 len 40960
29265 014/300 Write block 1136 len 8192 data: 5 4
29266 015/300 Read block 4659 len 20480
29267 016/300 Read block 2473 len 16384
29268 017/300 Read block 3612 len 8192
29269 018/300 Flush
29270 019/300 Flush
29271 020/300 Read block 4522 len 28672
29272 021/300 Write block 480 len 28672 data: 4 4 5 6 6 6 6
29273 022/300 Read block 1770 len 32768
29274 023/300 Write block 818 len 40960 data: 5 6 4 3 3 5 4 3 3 3
29275 024/300 Write block 2087 len 8192 data: 6 4
29276 025/300 Flush
29277 026/300 Write block 1728 len 24576 data: 7 6 6 7 5 5
29278 027/300 Read block 3339 len 40960
29279 028/300 Write block 4962 len 12288 data: 8 8 5
29280 029/300 Write block 2874 len 24576 data: 5 6 6 5 6 6
29281 030/300 Flush
29282 031/300 Read block 4633 len 40960
29283 032/300 Read block 2084 len 32768
29284 033/300 Write block 2711 len 40960 data: 3 5 5 5 4 3 3 2 2 2
29285 034/300 Read block 273 len 4096
29286 035/300 Flush
29287 036/300 Read block 160 len 12288
29288 037/300 Read block 1556 len 32768
29289 038/300 Write block 1963 len 28672 data: 8 7 8 9 8 9 8
29290 039/300 Read block 346 len 16384
29291 040/300 Write block 2281 len 32768 data: 8 5 5 5 4 4 5 6
29292 041/300 Read block 3373 len 28672
29293 042/300 Write block 109 len 8192 data: 5 4
29294 043/300 Write block 2929 len 32768 data: 6 5 5 6 4 4 5 4
29295 044/300 Flush
29296 045/300 Write block 703 len 16384 data: 7 6 6 4
29297 046/300 Read block 3895 len 20480
29298 047/300 Read block 1403 len 4096
29299 048/300 Write block 3381 len 24576 data: 3 4 4 5 4 3
29300 049/300 Read block 4456 len 12288
29301 050/300 Read block 3070 len 32768
29302 051/300 Read block 1657 len 16384
29303 052/300 Flush
29304 053/300 Read block 2779 len 40960
29305 054/300 Read block 2973 len 20480
29306 055/300 Write block 2064 len 12288 data: 5 6 8
29307 056/300 Read block 4686 len 32768
29308 057/300 Write block 1323 len 32768 data: 4 3 4 5 7 5 5 8
29309 058/300 Write block 597 len 24576 data: 7 6 5 5 5 4
29310 059/300 Write block 1675 len 8192 data: 4 4
29311 060/300 Write block 831 len 8192 data: 3 2
29312 061/300 Read block 3041 len 12288
29313 062/300 Write block 3255 len 8192 data: 5 6
29314 063/300 Read block 453 len 20480
29315 064/300 Flush
29316 065/300 Write block 1469 len 28672 data: 3 4 7 8 8 8 9
29317 066/300 Write block 4835 len 20480 data: 5 6 5 5 5
29318 067/300 Read block 4704 len 36864
29319 068/300 Read block 3207 len 40960
29320 069/300 Read block 1728 len 20480
29321 070/300 Read block 1017 len 16384
29322 071/300 Read block 2802 len 16384
29323 072/300 Write block 1423 len 8192 data: 3 4
29324 073/300 Read block 2010 len 20480
29325 074/300 Read block 3541 len 16384
29326 075/300 Read block 1492 len 20480
29327 076/300 Read block 4846 len 24576
29328 077/300 Write block 1746 len 40960 data: 5 5 5 5 5 7 7 5 6 6
29329 078/300 Read block 3850 len 8192
29330 079/300 Read block 4977 len 36864
29331 080/300 Read block 1979 len 20480
29332 081/300 Read block 4396 len 36864
29333 082/300 Write block 4564 len 40960 data: 7 6 4 4 6 7 6 5 5 5
29334 083/300 Flush
29335 084/300 Read block 2361 len 36864
29336 085/300 Write block 4195 len 12288 data: 5 7 9
29337 086/300 Flush
29338 087/300 Read block 1980 len 40960
29339 088/300 Read block 1444 len 32768
29340 089/300 Read block 3073 len 12288
29341 090/300 Write block 220 len 20480 data: 6 6 6 6 6
29342 091/300 Read block 4085 len 36864
29343 092/300 Read block 1822 len 20480
29344 093/300 Write block 362 len 24576 data: 4 4 4 5 7 7
29345 094/300 Write block 3280 len 8192 data: 6 5
29346 095/300 Read block 187 len 16384
29347 096/300 Write block 2669 len 12288 data: 5 3 3
29348 097/300 Write block 2661 len 36864 data: 4 4 4 4 4 4 5 6 6
29349 098/300 Read block 1974 len 40960
29350 099/300 Flush
29351 100/300 Write block 1804 len 36864 data: 6 4 5 5 4 5 5 6 6
29352 101/300 Read block 1291 len 28672
29353 102/300 Write block 2202 len 20480 data: 4 6 4 4 4
29354 103/300 Read block 1785 len 24576
29355 104/300 Flush
29356 105/300 Read block 1911 len 8192
29357 106/300 Read block 4540 len 20480
29358 107/300 Write block 4046 len 20480 data: 2 3 3 4 4
29359 108/300 Read block 3672 len 4096
29360 109/300 Flush
29361 110/300 Read block 303 len 28672
29362 111/300 Write block 4622 len 24576 data: 4 3 4 5 4 4
29363 112/300 Read block 2387 len 20480
29364 113/300 Write block 1124 len 4096 data: 6
29365 114/300 Write block 3256 len 20480 data: 7 5 4 4 4
29366 115/300 Read block 2387 len 28672
29367 116/300 Write block 2307 len 4096 data: 5
29368 117/300 Write block 2345 len 24576 data: 7 6 3 3 3 3
29369 118/300 Read block 4980 len 16384
29370 119/300 Read block 39 len 20480
29371 120/300 Read block 3933 len 8192
29372 121/300 Write block 4402 len 32768 data: 2 2 3 4 4 4 4 5
29373 122/300 Read block 924 len 4096
29374 123/300 Read block 2268 len 16384
29375 124/300 Write block 3850 len 16384 data: 11 10 11 9
29376 125/300 Read block 2735 len 20480
29377 126/300 Write block 3045 len 4096 data: 9
29378 127/300 Read block 4524 len 16384
29379 128/300 Read block 3486 len 4096
29380 129/300 Read block 4981 len 8192
29381 130/300 Write block 3712 len 20480 data: 3 4 4 4 4
29382 131/300 Read block 4547 len 40960
29383 132/300 Write block 4345 len 12288 data: 7 7 7
29384 133/300 Read block 1851 len 28672
29385 134/300 Read block 2145 len 36864
29386 135/300 Read block 3519 len 40960
29387 136/300 Write block 4151 len 32768 data: 6 6 6 4 6 7 8 4
29388 137/300 Read block 1950 len 24576
29389 138/300 Read block 2735 len 4096
29390 139/300 Write block 2587 len 32768 data: 8 6 6 4 4 4 4 4
29391 140/300 Read block 3583 len 36864
29392 141/300 Read block 3799 len 8192
29393 142/300 Read block 3589 len 12288
29394 143/300 Read block 1571 len 32768
29395 144/300 Read block 4267 len 20480
29396 145/300 Write block 4763 len 12288 data: 7 6 6
29397 146/300 Write block 4123 len 20480 data: 6 4 3 3 3
29398 147/300 Flush
29399 148/300 Write block 4822 len 28672 data: 6 5 4 3 2 2 3
29400 149/300 Write block 119 len 4096 data: 4
29401 150/300 Write block 3028 len 16384 data: 3 4 3 3
29402 151/300 Read block 3307 len 8192
29403 152/300 Write block 3916 len 8192 data: 4 4
29404 153/300 Read block 1100 len 12288
29405 154/300 Write block 2894 len 40960 data: 5 5 6 7 8 8 10 10 9 6
29406 155/300 Read block 1617 len 16384
29407 156/300 Write block 1052 len 16384 data: 5 9 9 8
29408 157/300 Write block 3130 len 8192 data: 7 7
29409 158/300 Write block 3939 len 40960 data: 5 5 7 9 8 8 6 6 6 5
29410 159/300 Write block 1349 len 24576 data: 5 4 3 4 4 5
29411 160/300 Read block 822 len 4096
29412 161/300 Read block 1270 len 20480
29413 162/300 Read block 570 len 36864
29414 163/300 Read block 1289 len 28672
29415 164/300 Write block 1553 len 24576 data: 10 10 9 6 5 5
29416 165/300 Flush
29417 166/300 Read block 1218 len 12288
29418 167/300 Write block 3408 len 28672 data: 6 5 5 4 4 4 4
29419 168/300 Write block 1439 len 32768 data: 5 4 4 5 3 2 2 3
29420 169/300 Read block 1763 len 8192
29421 170/300 Flush
29422 171/300 Write block 4617 len 8192 data: 4 3
29423 172/300 Read block 4542 len 36864
29424 173/300 Flush
29425 174/300 Write block 3436 len 40960 data: 5 7 7 6 7 6 4 5 6 4
29426 175/300 Read block 3371 len 32768
29427 176/300 Write block 1133 len 36864 data: 7 4 4 6 5 4 4 4 4
29428 177/300 Read block 3256 len 20480
29429 178/300 Write block 1364 len 12288 data: 3 4 4
29430 179/300 Read block 1766 len 4096
29431 180/300 Read block 181 len 28672
29432 181/300 Read block 2810 len 12288
29433 182/300 Write block 1082 len 40960 data: 7 7 5 5 3 3 3 3 3 3
29434 183/300 Read block 890 len 4096
29435 184/300 Read block 4320 len 4096
29436 185/300 Flush
29437 186/300 Read block 1648 len 16384
29438 187/300 Read block 3748 len 32768
29439 188/300 Write block 3435 len 4096 data: 6
29440 189/300 Write block 3220 len 16384 data: 2 2 2 2
29441 190/300 Write block 613 len 28672 data: 8 8 7 7 7 6 4
29442 191/300 Write block 4583 len 28672 data: 5 5 3 3 4 4 5
29443 192/300 Write block 1680 len 36864 data: 4 5 5 5 5 5 5 5 6
29444 193/300 Write block 3704 len 8192 data: 2 2
29445 194/300 Flush
29446 195/300 Write block 514 len 20480 data: 4 3 2 2 3
29447 196/300 Read block 1462 len 36864
29448 197/300 Write block 2573 len 32768 data: 4 5 5 5 7 7 5 5
29449 198/300 Flush
29450 199/300 Read block 1608 len 20480
29451 200/300 Read block 2077 len 16384
29452 201/300 Write block 3950 len 12288 data: 6 6 6
29453 202/300 Read block 3048 len 32768
29454 203/300 Read block 3746 len 12288
29455 204/300 Write block 3352 len 40960 data: 5 5 4 4 3 3 2 2 4 4
29456 205/300 Read block 3455 len 4096
29457 206/300 Write block 941 len 32768 data: 3 3 4 6 7 6 7 7
29458 207/300 Flush
29459 208/300 Read block 623 len 16384
29460 209/300 Read block 771 len 32768
29461 210/300 Read block 1400 len 40960
29462 211/300 Write block 457 len 8192 data: 4 4
29463 212/300 Write block 3420 len 32768 data: 5 6 5 5 4 6 8 7
29464 213/300 Write block 3126 len 4096 data: 6
29465 214/300 Read block 1717 len 40960
29466 215/300 Read block 4520 len 12288
29467 216/300 Read block 4852 len 24576
29468 217/300 Flush
29469 218/300 Read block 1242 len 8192
29470 219/300 Write block 3559 len 36864 data: 4 4 5 5 4 5 5 6 5
29471 220/300 Read block 589 len 16384
29472 221/300 Write block 184 len 12288 data: 4 5 4
29473 222/300 Write block 2784 len 36864 data: 7 7 6 5 7 6 6 5 5
29474 223/300 Read block 2869 len 40960
29475 224/300 Read block 1288 len 16384
29476 225/300 Write block 3411 len 32768 data: 5 5 5 5 3 3 3 5
29477 226/300 Write block 2535 len 24576 data: 6 5 5 5 5 6
29478 227/300 Write block 2413 len 24576 data: 9 8 6 6 9 9
29479 228/300 Read block 1652 len 24576
29480 229/300 Read block 3261 len 36864
29481 230/300 Read block 2379 len 8192
29482 231/300 Write block 2493 len 24576 data: 7 7 6 6 6 7
29483 232/300 Flush
29484 233/300 Read block 3945 len 32768
29485 234/300 Write block 3932 len 8192 data: 5 6
29486 235/300 Read block 1088 len 8192
29487 236/300 Read block 152 len 24576
29488 237/300 Read block 793 len 28672
29489 238/300 Read block 4567 len 12288
29490 239/300 Read block 249 len 40960
29491 240/300 Flush
29492 241/300 Read block 248 len 36864
29493 242/300 Read block 4660 len 36864
29494 243/300 Read block 1676 len 40960
29495 244/300 Write block 442 len 24576 data: 3 5 5 5 8 9
29496 245/300 Write block 3652 len 32768 data: 7 9 9 8 6 6 4 5
29497 246/300 Flush
29498 247/300 Write block 4405 len 12288 data: 5 5 5
29499 248/300 Write block 4701 len 12288 data: 6 4 3
29500 249/300 Write block 1159 len 24576 data: 7 5 5 5 5 5
29501 250/300 Read block 1750 len 24576
29502 251/300 Write block 362 len 32768 data: 5 5 5 6 8 8 7 8
29503 252/300 Write block 4510 len 36864 data: 6 7 6 6 6 6 6 4 4
29504 253/300 Read block 4799 len 20480
29505 254/300 Write block 4781 len 40960 data: 5 6 6 5 5 4 4 4 4 3
29506 255/300 Write block 604 len 28672 data: 5 5 5 5 5 8 8
29507 256/300 Read block 2621 len 4096
29508 257/300 Read block 2186 len 28672
29509 258/300 Write block 3814 len 12288 data: 4 4 2
29510 259/300 Read block 4638 len 40960
29511 260/300 Write block 3381 len 40960 data: 4 5 5 6 5 4 2 2 2 3
29512 261/300 Read block 1541 len 36864
29513 262/300 Write block 568 len 12288 data: 4 4 4
29514 263/300 Read block 964 len 24576
29515 264/300 Write block 4922 len 32768 data: 4 5 5 6 6 6 6 6
29516 265/300 Read block 4821 len 4096
29517 266/300 Write block 141 len 12288 data: 3 3 4
29518 267/300 Flush
29519 268/300 Read block 4722 len 16384
29520 269/300 Read block 1530 len 32768
29521 270/300 Write block 3899 len 20480 data: 3 2 2 2 2
29522 271/300 Write block 605 len 40960 data: 6 6 6 6 9 9 7 8 9 9
29523 272/300 Read block 1747 len 32768
29524 273/300 Flush
29525 274/300 Read block 2461 len 24576
29526 275/300 Write block 1462 len 28672 data: 3 3 3 2 2 2 2
29527 276/300 Write block 4561 len 4096 data: 8
29528 277/300 Write block 2443 len 36864 data: 4 4 4 8 8 7 7 6 3
29529 278/300 Write block 3429 len 36864 data: 9 9 9 7 7 6 7 6 8
29530 279/300 Write block 820 len 36864 data: 5 4 4 6 5 4 4 4 4
29531 280/300 Read block 4027 len 28672
29532 281/300 Write block 1247 len 4096 data: 4
29533 282/300 Write block 241 len 24576 data: 4 4 4 4 5 6
29534 283/300 Read block 2728 len 40960
29535 284/300 Read block 2007 len 40960
29536 285/300 Write block 4392 len 4096 data: 3
29537 286/300 Read block 3510 len 24576
29538 287/300 Read block 2613 len 36864
29539 288/300 Write block 3623 len 32768 data: 6 5 4 3 3 3 3 4
29540 289/300 Write block 2383 len 20480 data: 9 9 8 7 5
29541 290/300 Write block 1616 len 28672 data: 6 5 4 3 3 3 4
29542 291/300 Write block 1357 len 20480 data: 5 7 6 5 5
29543 292/300 Write block 3789 len 12288 data: 6 6 7
29544 293/300 Write block 659 len 40960 data: 8 9 9 9 7 7 4 3 3 2
29545 294/300 Write block 3809 len 40960 data: 6 6 4 4 4 5 5 3 2 3
29546 295/300 Read block 4780 len 24576
29547 296/300 Read block 3959 len 12288
29548 297/300 Read block 4989 len 36864
29549 298/300 Flush
29550 299/300 Write block 3411 len 20480 data: 6 6 6 6 4
29551 300/300 Read block 1233 len 20480
29552 301/300 Write block 2043 len 36864 data: 6 6 8 8 6 6 6 3 4
29553 Main task has work to do, go find it
29554 got dsc Start(2)
29555 start 2
29556 Mar 17 22:25:36.194 INFO request completed, latency_us: 249, response_code: 204, uri: /start/cid/2, method: POST, req_id: 98f768b1-e85b-4712-a315-3177af5d034d, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
29557 Replay: started 2, returned:Ok(())
29558 ----------------------------------------------------------------
29559 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29560 States: Active Active Offline
29561 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29562 7944 Acked Write 8 Done Done Sent false
29563 7945 Acked Read 1 Done Done Sent false
29564 7946 Acked Read 10 Done Done Sent false
29565 7947 Acked Read 9 Done Done Sent false
29566 7948 Acked Read 10 Done Done Sent false
29567 7949 Acked Write 3 Done Done Sent false
29568 7950 Acked Write 2 Done Done Sent false
29569 7951 Acked Write 1 Done Done Sent false
29570 7952 Acked Read 10 Done Done Sent false
29571 7953 Acked Write 4 Done Done Sent false
29572 7954 Acked Write 3 Done Done Sent false
29573 7955 Acked Read 3 Done Done Sent false
29574 7956 Acked Read 10 Done Done Sent false
29575 7957 Acked Write 2 Done Done Sent false
29576 7958 Acked Read 5 Done Done Sent false
29577 7959 Acked Read 4 Done Done Sent false
29578 7960 Acked Read 2 Done Done Sent false
29579 7961 Acked Flush 0 Done Done Sent false
29580 7962 Acked Flush 0 Done Done Sent false
29581 7963 Acked Read 7 Done Done Sent false
29582 7964 Acked Write 7 Done Done Sent false
29583 7965 Acked Read 8 Done Done Sent false
29584 7966 Acked Write 10 Done Done Sent false
29585 7967 Acked Write 2 Done Done Sent false
29586 7968 Acked Flush 0 Done Done Sent false
29587 7969 Acked Write 6 Done Done Sent false
29588 7970 Acked Read 10 Done Done Sent false
29589 7971 Acked Write 3 Done Done Sent false
29590 7972 Acked Write 6 Done Done Sent false
29591 7973 Acked Flush 0 Done Done Sent false
29592 7974 Acked Read 10 Done Done Sent false
29593 7975 Acked Read 8 Done Done Sent false
29594 7976 Acked Write 10 Done Done Sent false
29595 7977 Acked Read 1 Done Done Sent false
29596 7978 Acked Flush 0 Done Done Sent false
29597 7979 Acked Read 3 Done Done Sent false
29598 7980 Acked Read 8 Done Done Sent false
29599 7981 Acked Write 7 Done Done Sent false
29600 7982 Acked Read 4 Done Done Sent false
29601 7983 Acked Write 8 Done Done Sent false
29602 7984 Acked Read 7 Done Done Sent false
29603 7985 Acked Write 2 Done Done Sent false
29604 7986 Acked Write 8 Done Done Sent false
29605 7987 Acked Flush 0 Done Done Sent false
29606 7988 Acked Write 4 Done Done Sent false
29607 7989 Acked Read 5 Done Done Sent false
29608 7990 Acked Read 1 Done Done Sent false
29609 7991 Acked Write 6 Done Done Sent false
29610 7992 Acked Read 3 Done Done Sent false
29611 7993 Acked Read 8 Done Done Sent false
29612 7994 Acked Read 4 Done Done Sent false
29613 7995 Acked Flush 0 Done Done Sent false
29614 7996 Acked Read 10 Done Done Sent false
29615 7997 Acked Read 5 Done Done Sent false
29616 7998 Acked Write 3 Done Done Sent false
29617 7999 Acked Read 8 Done Done Sent false
29618 8000 Acked Write 8 Done Done Sent false
29619 8001 Acked Write 6 Done Done Sent false
29620 8002 Acked Write 2 Done Done Sent false
29621 8003 Acked Write 2 Done Done Sent false
29622 8004 Acked Read 3 Done Done Sent false
29623 8005 Acked Write 2 Done Done Sent false
29624 8006 Acked Read 5 Done Done Sent false
29625 8007 Acked Flush 0 Done Done Sent false
29626 8008 Acked Write 7 Done Done Sent false
29627 8009 Acked Write 5 Done Done Sent false
29628 8010 Acked Read 9 Done Done Sent false
29629 8011 Acked Read 10 Done Done Sent false
29630 8012 Acked Read 5 Done Done Sent false
29631 8013 Acked Read 4 Done Done Sent false
29632 8014 Acked Read 4 Done Done Sent false
29633 8015 Acked Write 2 Done Done Sent false
29634 8016 Acked Read 5 Done Done Sent false
29635 8017 Acked Read 4 Done Done Sent false
29636 8018 Acked Read 5 Done Done Sent false
29637 8019 Acked Read 6 Done Done Sent false
29638 8020 Acked Write 10 Done Done Sent false
29639 8021 Acked Read 2 Done Done Sent false
29640 8022 Acked Read 9 Done Done Sent false
29641 8023 Acked Read 5 Done Done Sent false
29642 8024 Acked Read 9 Done Done Sent false
29643 8025 Acked Write 10 Done Done Sent false
29644 8026 Acked Flush 0 Done Done Sent false
29645 8027 Acked Read 9 Done Done Sent false
29646 8028 Acked Write 3 Done Done Sent false
29647 8029 Acked Flush 0 Done Done Sent false
29648 8030 Acked Read 10 Done Done Sent false
29649 8031 Acked Read 8 Done Done Sent false
29650 8032 Acked Read 3 Done Done Sent false
29651 8033 Acked Write 5 Done Done Sent false
29652 8034 Acked Read 9 Done Done Sent false
29653 8035 Acked Read 5 Done Done Sent false
29654 8036 Acked Write 6 Done Done Sent false
29655 8037 Acked Write 2 Done Done Sent false
29656 8038 Acked Read 4 Done Done Sent false
29657 8039 Acked Write 3 Done Done Sent false
29658 8040 Acked Write 9 Done Done Sent false
29659 8041 Acked Read 10 Done Done Sent false
29660 8042 Acked Flush 0 Done Done Sent false
29661 8043 Acked Write 9 Done Done Sent false
29662 8044 Acked Read 7 Done Done Sent false
29663 8045 Acked Write 5 Done Done Sent false
29664 8046 Acked Read 6 Done Done Sent false
29665 8047 Acked Flush 0 Done Done Sent false
29666 8048 Acked Read 2 Done Done Sent false
29667 8049 Acked Read 5 Done Done Sent false
29668 8050 Acked Write 5 Done Done Sent false
29669 8051 Acked Read 1 Done Done Sent false
29670 8052 Acked Flush 0 Done Done Sent false
29671 8053 Acked Read 7 Done Done Sent false
29672 8054 Acked Write 6 Done Done Sent false
29673 8055 Acked Read 5 Done Done Sent false
29674 8056 Acked Write 1 Done Done Sent false
29675 8057 Acked Write 5 Done Done Sent false
29676 8058 Acked Read 7 Done Done Sent false
29677 8059 Acked Write 1 Done Done Sent false
29678 8060 Acked Write 6 Done Done Sent false
29679 8061 Acked Read 4 Done Done Sent false
29680 8062 Acked Read 5 Done Done Sent false
29681 8063 Acked Read 2 Done Done Sent false
29682 8064 Acked Write 8 Done Done Sent false
29683 8065 Acked Read 1 Done Done Sent false
29684 8066 Acked Read 4 Done Done Sent false
29685 8067 Acked Write 4 Done Done Sent false
29686 8068 Acked Read 5 Done Done Sent false
29687 8069 Acked Write 1 Done Done Sent false
29688 8070 Acked Read 4 Done Done Sent false
29689 8071 Acked Read 1 Done Done Sent false
29690 8072 Acked Read 2 Done Done Sent false
29691 8073 Acked Write 5 Done Done Sent false
29692 8074 Acked Read 10 Done Done Sent false
29693 8075 Acked Write 3 Done Done Sent false
29694 8076 Acked Read 7 Done Done Sent false
29695 8077 Acked Read 9 Done Done Sent false
29696 8078 Acked Read 10 Done Done Sent false
29697 8079 Acked Write 8 Done Done Sent false
29698 8080 Acked Read 6 Done Done Sent false
29699 8081 Acked Read 1 Done Done Sent false
29700 8082 Acked Write 8 Done Done Sent false
29701 8083 Acked Read 9 Done Done Sent false
29702 8084 Acked Read 2 Done Done Sent false
29703 8085 Acked Read 3 Done Done Sent false
29704 8086 Acked Read 8 Done Done Sent false
29705 8087 Acked Read 5 Done Done Sent false
29706 8088 Acked Write 3 Done Done Sent false
29707 8089 Acked Write 5 Done Done Sent false
29708 8090 Acked Flush 0 Done Done Sent false
29709 8091 Acked Write 7 Done Done Sent false
29710 8092 Acked Write 1 Done Done Sent false
29711 8093 Acked Write 4 Done Done Sent false
29712 8094 Acked Read 2 Done Done Sent false
29713 8095 Acked Write 2 Done Done Sent false
29714 8096 Acked Read 3 Done Done Sent false
29715 8097 Acked Write 10 Done Done Sent false
29716 8098 Acked Read 4 Done Done Sent false
29717 8099 Acked Write 4 Done Done Sent false
29718 8100 Acked Write 2 Done Done Sent false
29719 8101 Acked Write 10 Done Done Sent false
29720 8102 Acked Write 6 Done Done Sent false
29721 8103 Acked Read 1 Done Done Sent false
29722 8104 Acked Read 5 Done Done Sent false
29723 8105 Acked Read 9 Done Done Sent false
29724 8106 Acked Read 7 Done Done Sent false
29725 8107 Acked Write 6 Done Done Sent false
29726 8108 Acked Flush 0 Done Done Sent false
29727 8109 Acked Read 3 Done Done Sent false
29728 8110 Acked Write 7 Done Done Sent false
29729 8111 Acked Write 8 Done Done Sent false
29730 8112 Acked Read 2 Done Done Sent false
29731 8113 Acked Flush 0 Done Done Sent false
29732 8114 Acked Write 2 Done Done Sent false
29733 8115 Acked Read 9 Done Done Sent false
29734 8116 Acked Flush 0 Done Done Sent false
29735 8117 Acked Write 10 Done Done Sent false
29736 8118 Acked Read 8 Done Done Sent false
29737 8119 Acked Write 9 Done Done Sent false
29738 8120 Acked Read 5 Done Done Sent false
29739 8121 Acked Write 3 Done Done Sent false
29740 8122 Acked Read 1 Done Done Sent false
29741 8123 Acked Read 7 Done Done Sent false
29742 8124 Acked Read 3 Done Done Sent false
29743 8125 Acked Write 10 Done Done Sent false
29744 8126 Acked Read 1 Done Done Sent false
29745 8127 Acked Read 1 Done Done Sent false
29746 8128 Acked Flush 0 Done Done Sent false
29747 8129 Acked Read 4 Done Done Sent false
29748 8130 Acked Read 8 Done Done Sent false
29749 8131 Acked Write 1 Done Done Sent false
29750 8132 Acked Write 4 Done Done Sent false
29751 8133 Acked Write 7 Done Done Sent false
29752 8134 Acked Write 7 Done Done Sent false
29753 8135 Acked Write 9 Done Done Sent false
29754 8136 Acked Write 2 Done Done Sent false
29755 8137 Acked Flush 0 Done Done Sent false
29756 8138 Acked Write 5 Done Done Sent false
29757 8139 Acked Read 9 Done Done Sent false
29758 8140 Acked Write 8 Done Done Sent false
29759 8141 Acked Flush 0 Done Done Sent false
29760 8142 Acked Read 5 Done Done Sent false
29761 8143 Acked Read 4 Done Done Sent false
29762 8144 Acked Write 3 Done Done Sent false
29763 8145 Acked Read 8 Done Done Sent false
29764 8146 Acked Read 3 Done Done Sent false
29765 8147 Acked Write 10 Done Done Sent false
29766 8148 Acked Read 1 Done Done Sent false
29767 8149 Acked Write 8 Done Done Sent false
29768 8150 Acked Flush 0 Done Done Sent false
29769 8151 Acked Read 4 Done Done Sent false
29770 8152 Acked Read 8 Done Done Sent false
29771 8153 Acked Read 10 Done Done Sent false
29772 8154 Acked Write 2 Done Done Sent false
29773 8155 Acked Write 8 Done Done Sent false
29774 8156 Acked Write 1 Done Done Sent false
29775 8157 Acked Read 10 Done Done Sent false
29776 8158 Acked Read 3 Done Done Sent false
29777 8159 Acked Read 6 Done Done Sent false
29778 8160 Acked Flush 0 Done Done Sent false
29779 8161 Acked Read 2 Done Done Sent false
29780 8162 Acked Write 9 Done Done Sent false
29781 8163 Acked Read 4 Done Done Sent false
29782 8164 Acked Write 3 Done Done Sent false
29783 8165 Acked Write 9 Done Done Sent false
29784 8166 Acked Read 10 Done Done Sent false
29785 8167 Acked Read 4 Done Done Sent false
29786 8168 Acked Write 8 Done Done Sent false
29787 8169 Acked Write 6 Done Done Sent false
29788 8170 Acked Write 6 Done Done Sent false
29789 8171 Acked Read 6 Done Done Sent false
29790 8172 Acked Read 9 Done Done Sent false
29791 8173 Acked Read 2 Done Done Sent false
29792 8174 Acked Write 6 Done Done Sent false
29793 8175 Acked Flush 0 Done Done Sent false
29794 8176 Acked Read 8 Done Done Sent false
29795 8177 Acked Write 2 Done Done Sent false
29796 8178 Acked Read 2 Done Done Sent false
29797 8179 Acked Read 6 Done Done Sent false
29798 8180 Acked Read 7 Done Done Sent false
29799 8181 Acked Read 3 Done Done Sent false
29800 8182 Acked Read 10 Done Done Sent false
29801 8183 Acked Flush 0 Done Done Sent false
29802 8184 Acked Read 9 Done Done Sent false
29803 8185 Acked Read 9 Done Done Sent false
29804 8186 Acked Read 10 Done Done Sent false
29805 8187 Acked Write 6 Done Done Sent false
29806 8188 Acked Write 8 Done Done Sent false
29807 8189 Acked Flush 0 Done Done Sent false
29808 8190 Acked Write 3 Done Done Sent false
29809 8191 Acked Write 3 Done Done Sent false
29810 8192 Acked Write 6 Done Done Sent false
29811 8193 Acked Read 6 Done Done Sent false
29812 8194 Acked Write 8 Done Done Sent false
29813 8195 Acked Write 9 Done Done Sent false
29814 8196 Acked Read 5 Done Done Sent false
29815 8197 Acked Write 10 Done Done Sent false
29816 8198 Acked Write 7 Done Done Sent false
29817 8199 Acked Read 1 Done Done Sent false
29818 8200 Acked Read 7 Done Done Sent false
29819 8201 Acked Write 3 Done Done Sent false
29820 8202 Acked Read 10 Done Done Sent false
29821 8203 Acked Write 10 Done Done Sent false
29822 8204 Acked Read 9 Done Done Sent false
29823 8205 Acked Write 3 Done Done Sent false
29824 8206 Acked Read 6 Done Done Sent false
29825 8207 Acked Write 8 Done Done Sent false
29826 8208 Acked Read 1 Done Done Sent false
29827 8209 Acked Write 3 Done Done Sent false
29828 8210 Acked Flush 0 Done Done Sent false
29829 8211 Acked Read 4 Done Done Sent false
29830 8212 Acked Read 8 Done Done Sent false
29831 8213 Acked Write 5 Done Done Sent false
29832 8214 Acked Write 10 Done Done Sent false
29833 8215 Acked Read 8 Done Done Sent false
29834 8216 Acked Flush 0 Done Done Sent false
29835 8217 Acked Read 6 Done Done Sent false
29836 8218 Acked Write 7 Done Done Sent false
29837 8219 Acked Write 1 Done Done Sent false
29838 8220 Acked Write 9 Done Done Sent false
29839 8221 Acked Write 9 Done Done Sent false
29840 8222 Acked Write 9 Done Done Sent false
29841 8223 Acked Read 7 Done Done Sent false
29842 8224 Acked Write 1 Done Done Sent false
29843 8225 Acked Write 6 Done Done Sent false
29844 8226 Acked Read 10 Done Done Sent false
29845 8227 Acked Read 10 Done Done Sent false
29846 8228 Acked Write 1 Done Done Sent false
29847 8229 Acked Read 6 Done Done Sent false
29848 8230 Acked Read 9 Done Done Sent false
29849 8231 Acked Write 8 Done Done Sent false
29850 8232 Acked Write 5 Done Done Sent false
29851 8233 Acked Write 7 Done Done Sent false
29852 8234 Acked Write 5 Done Done Sent false
29853 8235 Acked Write 3 Done Done Sent false
29854 8236 Acked Write 10 Done Done Sent false
29855 8237 Acked Write 10 Done Done Sent false
29856 8238 Acked Read 6 Done Done Sent false
29857 8239 Acked Read 3 Done Done Sent false
29858 8240 Acked Read 9 Done Done Sent false
29859 8241 Acked Flush 0 Done Done Sent false
29860 8242 Acked Write 5 Done Done Sent false
29861 8243 Acked Read 5 Done Done Sent false
29862 8244 Acked Write 9 Done Done Sent false
29863 STATES DS:0 DS:1 DS:2 TOTAL
29864 Sent 0 0 301 301
29865 Done 301 301 0 602
29866 Skipped 0 0 0 0
29867 Error 0 0 0 0
29868 Last Flush: Some(JobId(8241)) Some(JobId(8241)) Some(JobId(7943))
29869 Downstairs last five completed: 7943 7942 7941 7940 7939
29870 Upstairs last five completed: 8244 8243 8242 8241 8240
29871 CLIENT: Up:0 ds:301 act:2
298722025-03-17T22:25:36.696ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
298732025-03-17T22:25:37.215ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
298742025-03-17T22:25:37.716ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
298752025-03-17T22:25:38.217ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
298762025-03-17T22:25:39.442ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
29877 [8830] got start action
29878 [8830] I am going to restart
29879 Starting downstairs at port 8830
29880 Make output file at "/tmp/dsc/downstairs-8830.txt"
29881 [8830][2] reports Starting
29882 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1235)
29883 [8830][2] reports Running
29884 ----------------------------------------------------------------
29885 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29886 States: Active Active Offline
29887 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29888 7944 Acked Write 8 Done Done Sent false
29889 7945 Acked Read 1 Done Done Sent false
29890 7946 Acked Read 10 Done Done Sent false
29891 7947 Acked Read 9 Done Done Sent false
29892 7948 Acked Read 10 Done Done Sent false
29893 7949 Acked Write 3 Done Done Sent false
29894 7950 Acked Write 2 Done Done Sent false
29895 7951 Acked Write 1 Done Done Sent false
29896 7952 Acked Read 10 Done Done Sent false
29897 7953 Acked Write 4 Done Done Sent false
29898 7954 Acked Write 3 Done Done Sent false
29899 7955 Acked Read 3 Done Done Sent false
29900 7956 Acked Read 10 Done Done Sent false
29901 7957 Acked Write 2 Done Done Sent false
29902 7958 Acked Read 5 Done Done Sent false
29903 7959 Acked Read 4 Done Done Sent false
29904 7960 Acked Read 2 Done Done Sent false
29905 7961 Acked Flush 0 Done Done Sent false
29906 7962 Acked Flush 0 Done Done Sent false
29907 7963 Acked Read 7 Done Done Sent false
29908 7964 Acked Write 7 Done Done Sent false
29909 7965 Acked Read 8 Done Done Sent false
29910 7966 Acked Write 10 Done Done Sent false
29911 7967 Acked Write 2 Done Done Sent false
29912 7968 Acked Flush 0 Done Done Sent false
29913 7969 Acked Write 6 Done Done Sent false
29914 7970 Acked Read 10 Done Done Sent false
29915 7971 Acked Write 3 Done Done Sent false
29916 7972 Acked Write 6 Done Done Sent false
29917 7973 Acked Flush 0 Done Done Sent false
29918 7974 Acked Read 10 Done Done Sent false
29919 7975 Acked Read 8 Done Done Sent false
29920 7976 Acked Write 10 Done Done Sent false
29921 7977 Acked Read 1 Done Done Sent false
29922 7978 Acked Flush 0 Done Done Sent false
29923 7979 Acked Read 3 Done Done Sent false
29924 7980 Acked Read 8 Done Done Sent false
29925 7981 Acked Write 7 Done Done Sent false
29926 7982 Acked Read 4 Done Done Sent false
29927 7983 Acked Write 8 Done Done Sent false
29928 7984 Acked Read 7 Done Done Sent false
29929 7985 Acked Write 2 Done Done Sent false
29930 7986 Acked Write 8 Done Done Sent false
29931 7987 Acked Flush 0 Done Done Sent false
29932 7988 Acked Write 4 Done Done Sent false
29933 7989 Acked Read 5 Done Done Sent false
29934 7990 Acked Read 1 Done Done Sent false
29935 7991 Acked Write 6 Done Done Sent false
29936 7992 Acked Read 3 Done Done Sent false
29937 7993 Acked Read 8 Done Done Sent false
29938 7994 Acked Read 4 Done Done Sent false
29939 7995 Acked Flush 0 Done Done Sent false
29940 7996 Acked Read 10 Done Done Sent false
29941 7997 Acked Read 5 Done Done Sent false
29942 7998 Acked Write 3 Done Done Sent false
29943 7999 Acked Read 8 Done Done Sent false
29944 8000 Acked Write 8 Done Done Sent false
29945 8001 Acked Write 6 Done Done Sent false
29946 8002 Acked Write 2 Done Done Sent false
29947 8003 Acked Write 2 Done Done Sent false
29948 8004 Acked Read 3 Done Done Sent false
29949 8005 Acked Write 2 Done Done Sent false
29950 8006 Acked Read 5 Done Done Sent false
29951 8007 Acked Flush 0 Done Done Sent false
29952 8008 Acked Write 7 Done Done Sent false
29953 8009 Acked Write 5 Done Done Sent false
29954 8010 Acked Read 9 Done Done Sent false
29955 8011 Acked Read 10 Done Done Sent false
29956 8012 Acked Read 5 Done Done Sent false
29957 8013 Acked Read 4 Done Done Sent false
29958 8014 Acked Read 4 Done Done Sent false
29959 8015 Acked Write 2 Done Done Sent false
29960 8016 Acked Read 5 Done Done Sent false
29961 8017 Acked Read 4 Done Done Sent false
29962 8018 Acked Read 5 Done Done Sent false
29963 8019 Acked Read 6 Done Done Sent false
29964 8020 Acked Write 10 Done Done Sent false
29965 8021 Acked Read 2 Done Done Sent false
29966 8022 Acked Read 9 Done Done Sent false
29967 8023 Acked Read 5 Done Done Sent false
29968 8024 Acked Read 9 Done Done Sent false
29969 8025 Acked Write 10 Done Done Sent false
29970 8026 Acked Flush 0 Done Done Sent false
29971 8027 Acked Read 9 Done Done Sent false
29972 8028 Acked Write 3 Done Done Sent false
29973 8029 Acked Flush 0 Done Done Sent false
29974 8030 Acked Read 10 Done Done Sent false
29975 8031 Acked Read 8 Done Done Sent false
29976 8032 Acked Read 3 Done Done Sent false
29977 8033 Acked Write 5 Done Done Sent false
29978 8034 Acked Read 9 Done Done Sent false
29979 8035 Acked Read 5 Done Done Sent false
29980 8036 Acked Write 6 Done Done Sent false
29981 8037 Acked Write 2 Done Done Sent false
29982 8038 Acked Read 4 Done Done Sent false
29983 8039 Acked Write 3 Done Done Sent false
29984 8040 Acked Write 9 Done Done Sent false
29985 8041 Acked Read 10 Done Done Sent false
29986 8042 Acked Flush 0 Done Done Sent false
29987 8043 Acked Write 9 Done Done Sent false
29988 8044 Acked Read 7 Done Done Sent false
29989 8045 Acked Write 5 Done Done Sent false
29990 8046 Acked Read 6 Done Done Sent false
29991 8047 Acked Flush 0 Done Done Sent false
29992 8048 Acked Read 2 Done Done Sent false
29993 8049 Acked Read 5 Done Done Sent false
29994 8050 Acked Write 5 Done Done Sent false
29995 8051 Acked Read 1 Done Done Sent false
29996 8052 Acked Flush 0 Done Done Sent false
29997 8053 Acked Read 7 Done Done Sent false
29998 8054 Acked Write 6 Done Done Sent false
29999 8055 Acked Read 5 Done Done Sent false
30000 8056 Acked Write 1 Done Done Sent false
30001 8057 Acked Write 5 Done Done Sent false
30002 8058 Acked Read 7 Done Done Sent false
30003 8059 Acked Write 1 Done Done Sent false
30004 8060 Acked Write 6 Done Done Sent false
30005 8061 Acked Read 4 Done Done Sent false
30006 8062 Acked Read 5 Done Done Sent false
30007 8063 Acked Read 2 Done Done Sent false
30008 8064 Acked Write 8 Done Done Sent false
30009 8065 Acked Read 1 Done Done Sent false
30010 8066 Acked Read 4 Done Done Sent false
30011 8067 Acked Write 4 Done Done Sent false
30012 8068 Acked Read 5 Done Done Sent false
30013 8069 Acked Write 1 Done Done Sent false
30014 8070 Acked Read 4 Done Done Sent false
30015 8071 Acked Read 1 Done Done Sent false
30016 8072 Acked Read 2 Done Done Sent false
30017 8073 Acked Write 5 Done Done Sent false
30018 8074 Acked Read 10 Done Done Sent false
30019 8075 Acked Write 3 Done Done Sent false
30020 8076 Acked Read 7 Done Done Sent false
30021 8077 Acked Read 9 Done Done Sent false
30022 8078 Acked Read 10 Done Done Sent false
30023 8079 Acked Write 8 Done Done Sent false
30024 8080 Acked Read 6 Done Done Sent false
30025 8081 Acked Read 1 Done Done Sent false
30026 8082 Acked Write 8 Done Done Sent false
30027 8083 Acked Read 9 Done Done Sent false
30028 8084 Acked Read 2 Done Done Sent false
30029 8085 Acked Read 3 Done Done Sent false
30030 8086 Acked Read 8 Done Done Sent false
30031 8087 Acked Read 5 Done Done Sent false
30032 8088 Acked Write 3 Done Done Sent false
30033 8089 Acked Write 5 Done Done Sent false
30034 8090 Acked Flush 0 Done Done Sent false
30035 8091 Acked Write 7 Done Done Sent false
30036 8092 Acked Write 1 Done Done Sent false
30037 8093 Acked Write 4 Done Done Sent false
30038 8094 Acked Read 2 Done Done Sent false
30039 8095 Acked Write 2 Done Done Sent false
30040 8096 Acked Read 3 Done Done Sent false
30041 8097 Acked Write 10 Done Done Sent false
30042 8098 Acked Read 4 Done Done Sent false
30043 8099 Acked Write 4 Done Done Sent false
30044 8100 Acked Write 2 Done Done Sent false
30045 8101 Acked Write 10 Done Done Sent false
30046 8102 Acked Write 6 Done Done Sent false
30047 8103 Acked Read 1 Done Done Sent false
30048 8104 Acked Read 5 Done Done Sent false
30049 8105 Acked Read 9 Done Done Sent false
30050 8106 Acked Read 7 Done Done Sent false
30051 8107 Acked Write 6 Done Done Sent false
30052 8108 Acked Flush 0 Done Done Sent false
30053 8109 Acked Read 3 Done Done Sent false
30054 8110 Acked Write 7 Done Done Sent false
30055 8111 Acked Write 8 Done Done Sent false
30056 8112 Acked Read 2 Done Done Sent false
30057 8113 Acked Flush 0 Done Done Sent false
30058 8114 Acked Write 2 Done Done Sent false
30059 8115 Acked Read 9 Done Done Sent false
30060 8116 Acked Flush 0 Done Done Sent false
30061 8117 Acked Write 10 Done Done Sent false
30062 8118 Acked Read 8 Done Done Sent false
30063 8119 Acked Write 9 Done Done Sent false
30064 8120 Acked Read 5 Done Done Sent false
30065 8121 Acked Write 3 Done Done Sent false
30066 8122 Acked Read 1 Done Done Sent false
30067 8123 Acked Read 7 Done Done Sent false
30068 8124 Acked Read 3 Done Done Sent false
30069 8125 Acked Write 10 Done Done Sent false
30070 8126 Acked Read 1 Done Done Sent false
30071 8127 Acked Read 1 Done Done Sent false
30072 8128 Acked Flush 0 Done Done Sent false
30073 8129 Acked Read 4 Done Done Sent false
30074 8130 Acked Read 8 Done Done Sent false
30075 8131 Acked Write 1 Done Done Sent false
30076 8132 Acked Write 4 Done Done Sent false
30077 8133 Acked Write 7 Done Done Sent false
30078 8134 Acked Write 7 Done Done Sent false
30079 8135 Acked Write 9 Done Done Sent false
30080 8136 Acked Write 2 Done Done Sent false
30081 8137 Acked Flush 0 Done Done Sent false
30082 8138 Acked Write 5 Done Done Sent false
30083 8139 Acked Read 9 Done Done Sent false
30084 8140 Acked Write 8 Done Done Sent false
30085 8141 Acked Flush 0 Done Done Sent false
30086 8142 Acked Read 5 Done Done Sent false
30087 8143 Acked Read 4 Done Done Sent false
30088 8144 Acked Write 3 Done Done Sent false
30089 8145 Acked Read 8 Done Done Sent false
30090 8146 Acked Read 3 Done Done Sent false
30091 8147 Acked Write 10 Done Done Sent false
30092 8148 Acked Read 1 Done Done Sent false
30093 8149 Acked Write 8 Done Done Sent false
30094 8150 Acked Flush 0 Done Done Sent false
30095 8151 Acked Read 4 Done Done Sent false
30096 8152 Acked Read 8 Done Done Sent false
30097 8153 Acked Read 10 Done Done Sent false
30098 8154 Acked Write 2 Done Done Sent false
30099 8155 Acked Write 8 Done Done Sent false
30100 8156 Acked Write 1 Done Done Sent false
30101 8157 Acked Read 10 Done Done Sent false
30102 8158 Acked Read 3 Done Done Sent false
30103 8159 Acked Read 6 Done Done Sent false
30104 8160 Acked Flush 0 Done Done Sent false
30105 8161 Acked Read 2 Done Done Sent false
30106 8162 Acked Write 9 Done Done Sent false
30107 8163 Acked Read 4 Done Done Sent false
30108 8164 Acked Write 3 Done Done Sent false
30109 8165 Acked Write 9 Done Done Sent false
30110 8166 Acked Read 10 Done Done Sent false
30111 8167 Acked Read 4 Done Done Sent false
30112 8168 Acked Write 8 Done Done Sent false
30113 8169 Acked Write 6 Done Done Sent false
30114 8170 Acked Write 6 Done Done Sent false
30115 8171 Acked Read 6 Done Done Sent false
30116 8172 Acked Read 9 Done Done Sent false
30117 8173 Acked Read 2 Done Done Sent false
30118 8174 Acked Write 6 Done Done Sent false
30119 8175 Acked Flush 0 Done Done Sent false
30120 8176 Acked Read 8 Done Done Sent false
30121 8177 Acked Write 2 Done Done Sent false
30122 8178 Acked Read 2 Done Done Sent false
30123 8179 Acked Read 6 Done Done Sent false
30124 8180 Acked Read 7 Done Done Sent false
30125 8181 Acked Read 3 Done Done Sent false
30126 8182 Acked Read 10 Done Done Sent false
30127 8183 Acked Flush 0 Done Done Sent false
30128 8184 Acked Read 9 Done Done Sent false
30129 8185 Acked Read 9 Done Done Sent false
30130 8186 Acked Read 10 Done Done Sent false
30131 8187 Acked Write 6 Done Done Sent false
30132 8188 Acked Write 8 Done Done Sent false
30133 8189 Acked Flush 0 Done Done Sent false
30134 8190 Acked Write 3 Done Done Sent false
30135 8191 Acked Write 3 Done Done Sent false
30136 8192 Acked Write 6 Done Done Sent false
30137 8193 Acked Read 6 Done Done Sent false
30138 8194 Acked Write 8 Done Done Sent false
30139 8195 Acked Write 9 Done Done Sent false
30140 8196 Acked Read 5 Done Done Sent false
30141 8197 Acked Write 10 Done Done Sent false
30142 8198 Acked Write 7 Done Done Sent false
30143 8199 Acked Read 1 Done Done Sent false
30144 8200 Acked Read 7 Done Done Sent false
30145 8201 Acked Write 3 Done Done Sent false
30146 8202 Acked Read 10 Done Done Sent false
30147 8203 Acked Write 10 Done Done Sent false
30148 8204 Acked Read 9 Done Done Sent false
30149 8205 Acked Write 3 Done Done Sent false
30150 8206 Acked Read 6 Done Done Sent false
30151 8207 Acked Write 8 Done Done Sent false
30152 8208 Acked Read 1 Done Done Sent false
30153 8209 Acked Write 3 Done Done Sent false
30154 8210 Acked Flush 0 Done Done Sent false
30155 8211 Acked Read 4 Done Done Sent false
30156 8212 Acked Read 8 Done Done Sent false
30157 8213 Acked Write 5 Done Done Sent false
30158 8214 Acked Write 10 Done Done Sent false
30159 8215 Acked Read 8 Done Done Sent false
30160 8216 Acked Flush 0 Done Done Sent false
30161 8217 Acked Read 6 Done Done Sent false
30162 8218 Acked Write 7 Done Done Sent false
30163 8219 Acked Write 1 Done Done Sent false
30164 8220 Acked Write 9 Done Done Sent false
30165 8221 Acked Write 9 Done Done Sent false
30166 8222 Acked Write 9 Done Done Sent false
30167 8223 Acked Read 7 Done Done Sent false
30168 8224 Acked Write 1 Done Done Sent false
30169 8225 Acked Write 6 Done Done Sent false
30170 8226 Acked Read 10 Done Done Sent false
30171 8227 Acked Read 10 Done Done Sent false
30172 8228 Acked Write 1 Done Done Sent false
30173 8229 Acked Read 6 Done Done Sent false
30174 8230 Acked Read 9 Done Done Sent false
30175 8231 Acked Write 8 Done Done Sent false
30176 8232 Acked Write 5 Done Done Sent false
30177 8233 Acked Write 7 Done Done Sent false
30178 8234 Acked Write 5 Done Done Sent false
30179 8235 Acked Write 3 Done Done Sent false
30180 8236 Acked Write 10 Done Done Sent false
30181 8237 Acked Write 10 Done Done Sent false
30182 8238 Acked Read 6 Done Done Sent false
30183 8239 Acked Read 3 Done Done Sent false
30184 8240 Acked Read 9 Done Done Sent false
30185 8241 Acked Flush 0 Done Done Sent false
30186 8242 Acked Write 5 Done Done Sent false
30187 8243 Acked Read 5 Done Done Sent false
30188 8244 Acked Write 9 Done Done Sent false
30189 STATES DS:0 DS:1 DS:2 TOTAL
30190 Sent 0 0 301 301
30191 Done 301 301 0 602
30192 Skipped 0 0 0 0
30193 Error 0 0 0 0
30194 Last Flush: Some(JobId(8241)) Some(JobId(8241)) Some(JobId(7943))
30195 Downstairs last five completed: 7943 7942 7941 7940 7939
30196 Upstairs last five completed: 8244 8243 8242 8241 8240
30197 CLIENT: Up:0 ds:301 act:2
301982025-03-17T22:25:40.699ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
301992025-03-17T22:25:41.222ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
302002025-03-17T22:25:41.941ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
302012025-03-17T22:25:41.941ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:52566) = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
302022025-03-17T22:25:41.942ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
302032025-03-17T22:25:41.943ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
302042025-03-17T22:25:41.943ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
302052025-03-17T22:25:41.943ZINFOcrucible: send last flush ID to this DS: Some(JobId(7943)) = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
302062025-03-17T22:25:41.943ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
302072025-03-17T22:25:41.943ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(7943)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
302082025-03-17T22:25:41.943ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(7943)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
30209 ----------------------------------------------------------------
30210 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
30211 Downstairs last five completed: 8245 8244 8243 8242 8241
30212 Upstairs last five completed: 8245 8244 8243 8242 8241
30213 CLIENT: Up:0 ds:0 act:3
30214 Replay: All jobs finished
30215 Main task has work to do, go find it
30216 got dsc Stop(0)
30217 stop 0
30218 Mar 17 22:25:44.200 INFO request completed[8810] Got stop action so:false kr:false
30219 , latency_us: 379, response_code: 204, uri: /stop/cid/0, method: POST, req_id: edc5340d-b7bf-45ca-b743-92c94803e6a1, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
302202025-03-17T22:25:44.201ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
302212025-03-17T22:25:44.201ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
302222025-03-17T22:25:44.201ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
30223 Mar 17 22:25:44.201 INFO request completed, latency_us: 160, response_code: 200, uri: /state/cid/0, method: GET, req_id: ea7e83c4-0bed-4d38-811a-83a1a4a9a8e9, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
302242025-03-17T22:25:44.201ZINFOcrucible: client task is exiting = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
30225 [8810] Exited with: ExitStatus(unix_wait_status(9))
30226 [8810][0] reports Exit
30227 Mar 17 22:25:48.203 INFO request completed, latency_us: 216, response_code: 200, uri: /state/cid/0, method: GET, req_id: 34a0ab63-0649-4328-906b-602caa1d7db4, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
30228 001/300 Write block 3177 len 40960 data: 5 7 6 8 8 7 7 7 7 8
30229 002/300 Write block 3555 len 12288 data: 3 3 3
30230 003/300 Read block 212 len 20480
30231 004/300 Write block 1162 len 32768 data: 6 6 6 5 3 4 4 3
30232 005/300 Read block 2223 len 32768
30233 006/300 Read block 4615 len 32768
30234 007/300 Write block 1769 len 8192 data: 5 4
30235 008/300 Write block 539 len 36864 data: 6 6 4 4 3 3 2 2 2
30236 009/300 Write block 1398 len 32768 data: 4 4 4 5 5 5 4 4
30237 010/300 Read block 1206 len 20480
30238 011/300 Write block 3125 len 20480 data: 5 7 8 7 7
30239 012/300 Write block 337 len 20480 data: 4 4 4 5 6
30240 013/300 Read block 4927 len 12288
30241 014/300 Read block 1148 len 24576
30242 015/300 Read block 2117 len 32768
30243 016/300 Write block 4666 len 20480 data: 9 8 8 8 8
30244 017/300 Read block 4722 len 36864
30245 018/300 Read block 4156 len 8192
30246 019/300 Read block 1223 len 28672
30247 020/300 Write block 4219 len 8192 data: 6 7
30248 021/300 Flush
30249 022/300 Read block 4986 len 32768
30250 023/300 Read block 1281 len 32768
30251 024/300 Read block 370 len 24576
30252 025/300 Write block 1086 len 28672 data: 4 4 4 4 4 4 4
30253 026/300 Write block 1438 len 40960 data: 4 6 5 5 6 4 3 3 4 3
30254 027/300 Read block 1180 len 28672
30255 028/300 Write block 4162 len 24576 data: 6 7 7 7 8 9
30256 029/300 Write block 3472 len 40960 data: 5 6 6 6 6 7 6 7 6 5
30257 030/300 Read block 3891 len 32768
30258 031/300 Write block 4963 len 12288 data: 9 6 6
30259 032/300 Read block 411 len 16384
30260 033/300 Read block 2560 len 28672
30261 034/300 Read block 1231 len 4096
30262 035/300 Read block 1348 len 36864
30263 036/300 Write block 3484 len 16384 data: 4 2 2 3
30264 037/300 Write block 4533 len 36864 data: 6 6 5 4 5 8 8 8 5
30265 038/300 Read block 3568 len 36864
30266 039/300 Read block 2512 len 40960
30267 040/300 Read block 797 len 36864
30268 041/300 Write block 3039 len 24576 data: 7 8 8 9 9 9
30269 042/300 Flush
30270 043/300 Read block 705 len 28672
30271 044/300 Read block 934 len 12288
30272 045/300 Read block 1999 len 36864
30273 046/300 Flush
30274 047/300 Read block 823 len 36864
30275 048/300 Read block 803 len 4096
30276 049/300 Write block 898 len 12288 data: 4 4 4
30277 050/300 Write block 2119 len 4096 data: 6
30278 051/300 Write block 2095 len 4096 data: 2
30279 052/300 Write block 3106 len 4096 data: 5
30280 053/300 Read block 2108 len 24576
30281 054/300 Write block 1084 len 40960 data: 6 6 5 5 5 5 5 5 5 4
30282 055/300 Write block 1670 len 36864 data: 9 8 6 7 5 5 5 4 4
30283 056/300 Read block 3366 len 40960
30284 057/300 Write block 851 len 24576 data: 4 6 5 5 5 6
30285 058/300 Flush
30286 059/300 Read block 4963 len 28672
30287 060/300 Read block 1267 len 4096
30288 061/300 Read block 4126 len 16384
30289 062/300 Write block 771 len 4096 data: 5
30290 063/300 Read block 2811 len 24576
30291 064/300 Flush
30292 065/300 Write block 3717 len 32768 data: 5 6 5 6 6 7 9 8
30293 066/300 Read block 1845 len 24576
30294 067/300 Write block 4009 len 20480 data: 3 3 3 4 5
30295 068/300 Write block 815 len 32768 data: 5 4 5 6 7 6 5 5
30296 069/300 Read block 2348 len 36864
30297 070/300 Write block 2346 len 32768 data: 7 4 4 4 4 3 3 3
30298 071/300 Write block 2798 len 32768 data: 4 3 3 3 5 5 4 2
30299 072/300 Read block 3426 len 8192
30300 073/300 Flush
30301 074/300 Read block 3762 len 32768
30302 075/300 Read block 2306 len 12288
30303 076/300 Read block 3892 len 40960
30304 077/300 Write block 511 len 40960 data: 6 6 4 5 4 3 3 4 4 6
30305 078/300 Read block 1389 len 4096
30306 079/300 Write block 4832 len 16384 data: 3 4 4 6
30307 080/300 Read block 3279 len 16384
30308 081/300 Read block 3301 len 28672
30309 082/300 Read block 4767 len 4096
30310 083/300 Read block 4857 len 28672
30311 084/300 Write block 2173 len 28672 data: 7 6 7 6 6 5 5
30312 085/300 Write block 1193 len 32768 data: 3 4 5 5 5 4 4 4
30313 086/300 Write block 131 len 20480 data: 5 5 4 6 5
30314 087/300 Read block 3564 len 4096
30315 088/300 Flush
30316 089/300 Write block 2036 len 24576 data: 4 5 7 7 7 7
30317 090/300 Read block 2657 len 16384
30318 091/300 Write block 414 len 8192 data: 8 8
30319 092/300 Write block 1557 len 16384 data: 6 6 7 7
30320 093/300 Write block 2681 len 32768 data: 5 7 7 5 7 7 8 8
30321 094/300 Write block 3761 len 24576 data: 4 4 5 4 4 4
30322 095/300 Read block 4199 len 36864
30323 096/300 Read block 1214 len 12288
30324 097/300 Read block 2693 len 28672
30325 098/300 Flush
30326 099/300 Flush
30327 100/300 Read block 2121 len 24576
30328 101/300 Read block 3079 len 32768
30329 102/300 Read block 4292 len 16384
30330 103/300 Read block 327 len 40960
30331 104/300 Write block 2499 len 20480 data: 8 7 5 5 6
30332 105/300 Read block 572 len 12288
30333 106/300 Read block 520 len 32768
30334 107/300 Write block 1546 len 16384 data: 3 3 6 9
30335 108/300 Read block 2299 len 4096
30336 109/300 Read block 773 len 36864
30337 110/300 Write block 2486 len 24576 data: 5 4 4 3 4 5
30338 111/300 Write block 2562 len 24576 data: 6 4 5 5 4 3
30339 112/300 Write block 35 len 20480 data: 4 6 7 6 5
30340 113/300 Read block 848 len 32768
30341 114/300 Flush
30342 115/300 Read block 640 len 24576
30343 116/300 Read block 4248 len 40960
30344 117/300 Write block 2389 len 12288 data: 4 6 5
30345 118/300 Write block 4464 len 32768 data: 2 2 2 2 3 3 3 4
30346 119/300 Read block 4161 len 8192
30347 120/300 Write block 4677 len 4096 data: 6
30348 121/300 Read block 2682 len 32768
30349 122/300 Write block 857 len 40960 data: 7 6 6 6 4 2 2 2 2 2
30350 123/300 Read block 1440 len 16384
30351 124/300 Write block 496 len 20480 data: 5 5 7 4 3
30352 125/300 Write block 3615 len 12288 data: 4 4 4
30353 126/300 Write block 4415 len 16384 data: 5 4 3 3
30354 127/300 Write block 1504 len 40960 data: 4 4 4 4 4 5 4 5 5 6
30355 128/300 Write block 4106 len 24576 data: 4 3 3 3 4 3
30356 129/300 Read block 4121 len 20480
30357 130/300 Write block 4144 len 8192 data: 5 7
30358 131/300 Write block 4299 len 32768 data: 4 5 6 4 4 3 3 2
30359 132/300 Read block 127 len 20480
30360 133/300 Flush
30361 134/300 Read block 261 len 4096
30362 135/300 Flush
30363 136/300 Read block 2760 len 20480
30364 137/300 Read block 2436 len 16384
30365 138/300 Flush
30366 139/300 Write block 816 len 4096 data: 5
30367 140/300 Read block 1536 len 40960
30368 141/300 Write block 4026 len 40960 data: 4 4 5 5 6 7 7 7 7 8
30369 142/300 Read block 97 len 28672
30370 143/300 Read block 4329 len 20480
30371 144/300 Write block 3941 len 24576 data: 8 10 9 9 7 7
30372 145/300 Write block 68 len 32768 data: 6 5 5 6 7 9 8 6
30373 146/300 Write block 4777 len 36864 data: 5 5 5 5 6 7 7 6 6
30374 147/300 Flush
30375 148/300 Write block 4090 len 24576 data: 4 5 5 6 6 5
30376 149/300 Flush
30377 150/300 Write block 497 len 40960 data: 6 8 5 4 5 4 4 5 6 4
30378 151/300 Read block 1595 len 4096
30379 152/300 Write block 2494 len 20480 data: 8 7 7 7 8
30380 153/300 Write block 2516 len 16384 data: 4 4 4 6
30381 154/300 Read block 3019 len 36864
30382 155/300 Read block 3978 len 4096
30383 156/300 Read block 691 len 12288
30384 157/300 Read block 1969 len 4096
30385 158/300 Write block 3661 len 16384 data: 4 5 5 5
30386 159/300 Write block 4733 len 8192 data: 6 6
30387 160/300 Write block 1279 len 4096 data: 5
30388 161/300 Flush
30389 162/300 Read block 3526 len 12288
30390 163/300 Read block 3635 len 16384
30391 164/300 Write block 4271 len 36864 data: 4 5 6 5 5 4 4 3 5
30392 165/300 Write block 1243 len 32768 data: 5 4 4 4 5 5 7 8
30393 166/300 Read block 114 len 40960
30394 167/300 Read block 2956 len 40960
30395 168/300 Write block 1769 len 20480 data: 6 5 4 4 4
30396 169/300 Write block 1905 len 36864 data: 7 8 8 9 8 6 4 3 3
30397 170/300 Write block 797 len 4096 data: 8
30398 171/300 Write block 2636 len 40960 data: 5 7 6 7 6 5 4 4 3 4
30399 172/300 Read block 639 len 24576
30400 173/300 Read block 768 len 40960
30401 174/300 Write block 1548 len 32768 data: 7 10 9 9 9 11 11 10
30402 175/300 Read block 2620 len 40960
30403 176/300 Flush
30404 177/300 Read block 3138 len 24576
30405 178/300 Flush
30406 179/300 Read block 2300 len 20480
30407 180/300 Read block 761 len 32768
30408 181/300 Read block 3349 len 16384
30409 182/300 Read block 3957 len 28672
30410 183/300 Write block 2245 len 28672 data: 7 7 8 9 10 7 6
30411 184/300 Flush
30412 185/300 Read block 4845 len 28672
30413 186/300 Read block 1253 len 32768
30414 187/300 Read block 457 len 12288
30415 188/300 Write block 1601 len 36864 data: 3 4 6 6 6 6 5 7 8
30416 189/300 Write block 3296 len 4096 data: 3
30417 190/300 Write block 3617 len 4096 data: 5
30418 191/300 Write block 997 len 40960 data: 3 4 4 5 5 5 5 3 3 2
30419 192/300 Write block 306 len 24576 data: 3 2 3 5 6 7
30420 193/300 Read block 172 len 8192
30421 194/300 Write block 2217 len 36864 data: 2 2 3 3 5 6 6 7 7
30422 195/300 Read block 1911 len 36864
30423 196/300 Read block 1036 len 8192
30424 197/300 Flush
30425 198/300 Flush
30426 199/300 Read block 3705 len 40960
30427 200/300 Flush
30428 201/300 Write block 1746 len 20480 data: 6 6 6 6 6
30429 202/300 Read block 240 len 16384
30430 203/300 Read block 3612 len 24576
30431 204/300 Write block 2568 len 36864 data: 4 3 3 4 4 5 6 6 6
30432 205/300 Write block 1690 len 24576 data: 4 3 2 3 3 5
30433 206/300 Read block 397 len 16384
30434 207/300 Write block 2951 len 24576 data: 8 8 8 7 7 6
30435 208/300 Write block 3084 len 20480 data: 5 4 4 5 6
30436 209/300 Flush
30437 210/300 Write block 1717 len 40960 data: 3 3 4 5 6 6 6 7 7 9
30438 211/300 Write block 4093 len 32768 data: 7 7 6 5 3 3 3 3
30439 212/300 Read block 329 len 8192
30440 213/300 Read block 3854 len 4096
30441 214/300 Write block 4530 len 4096 data: 3
30442 215/300 Write block 2787 len 28672 data: 6 8 7 7 6 6 4
30443 216/300 Flush
30444 217/300 Read block 99 len 28672
30445 218/300 Write block 2413 len 24576 data: 10 9 7 7 10 10
30446 219/300 Read block 869 len 24576
30447 220/300 Read block 3182 len 4096
30448 221/300 Write block 992 len 40960 data: 3 3 3 2 2 4 5 5 6 6
30449 222/300 Flush
30450 223/300 Read block 4637 len 12288
30451 224/300 Read block 2985 len 16384
30452 225/300 Read block 336 len 20480
30453 226/300 Flush
30454 227/300 Write block 3474 len 36864 data: 7 7 7 8 7 8 7 6 4
30455 228/300 Read block 1861 len 40960
30456 229/300 Read block 2738 len 12288
30457 230/300 Write block 4 len 4096 data: 4
30458 231/300 Write block 751 len 28672 data: 3 4 5 6 6 7 7
30459 232/300 Write block 268 len 8192 data: 6 6
30460 233/300 Read block 2359 len 4096
30461 234/300 Read block 3089 len 4096
30462 235/300 Read block 4997 len 12288
30463 236/300 Read block 4604 len 24576
30464 237/300 Flush
30465 238/300 Flush
30466 239/300 Read block 4550 len 28672
30467 240/300 Write block 717 len 16384 data: 3 3 3 2
30468 241/300 Write block 3143 len 36864 data: 5 3 3 3 5 5 4 4 5
30469 242/300 Write block 2918 len 36864 data: 4 4 5 5 4 4 6 6 5
30470 243/300 Read block 1616 len 4096
30471 244/300 Write block 2290 len 24576 data: 5 6 5 6 4 4
30472 245/300 Read block 36 len 16384
30473 246/300 Flush
30474 247/300 Write block 4471 len 4096 data: 5
30475 248/300 Write block 3130 len 28672 data: 8 8 6 5 5 6 8
30476 249/300 Flush
30477 250/300 Write block 1696 len 36864 data: 4 4 4 4 4 3 4 4 4
30478 251/300 Write block 214 len 16384 data: 6 3 4 5
30479 252/300 Write block 463 len 16384 data: 5 6 6 5
30480 253/300 Write block 2549 len 8192 data: 7 7
30481 254/300 Read block 4846 len 32768
30482 255/300 Read block 3781 len 8192
30483 256/300 Flush
30484 257/300 Write block 2014 len 20480 data: 6 7 6 5 6
30485 258/300 Write block 4551 len 12288 data: 9 7 6
30486 259/300 Write block 4932 len 24576 data: 8 8 5 6 6 7
30487 260/300 Write block 3514 len 28672 data: 3 4 5 5 6 8 8
30488 261/300 Write block 2517 len 20480 data: 5 5 7 5 6
30489 262/300 Flush
30490 263/300 Write block 1414 len 40960 data: 7 6 6 6 5 5 5 3 3 4
30491 264/300 Write block 1551 len 24576 data: 10 10 12 12 11 7
30492 265/300 Read block 1972 len 4096
30493 266/300 Write block 1308 len 28672 data: 4 4 5 5 3 3 3
30494 267/300 Read block 335 len 36864
30495 268/300 Read block 279 len 32768
30496 269/300 Read block 4214 len 16384
30497 270/300 Write block 1562 len 8192 data: 6 7
30498 271/300 Read block 3876 len 40960
30499 272/300 Write block 4975 len 24576 data: 6 5 6 7 8 7
30500 273/300 Read block 783 len 12288
30501 274/300 Read block 4631 len 40960
30502 275/300 Write block 2618 len 32768 data: 3 5 6 6 8 8 6 5
30503 276/300 Write block 4306 len 12288 data: 3 4 4
30504 277/300 Read block 2961 len 20480
30505 278/300 Read block 863 len 32768
30506 279/300 Write block 273 len 8192 data: 4 4
30507 280/300 Read block 251 len 36864
30508 281/300 Flush
30509 282/300 Write block 51 len 40960 data: 5 5 4 4 4 5 5 5 5 5
30510 283/300 Read block 1781 len 20480
30511 284/300 Flush
30512 285/300 Read block 592 len 8192
30513 286/300 Write block 3270 len 32768 data: 5 4 5 6 6 6 5 4
30514 287/300 Write block 490 len 32768 data: 5 4 5 6 7 6 6 7
30515 288/300 Read block 4915 len 32768
30516 289/300 Read block 4092 len 36864
30517 290/300 Write block 925 len 24576 data: 5 4 4 4 5 7
30518 291/300 Read block 4694 len 4096
30519 292/300 Read block 645 len 4096
30520 293/300 Write block 957 len 24576 data: 7 7 7 8 5 5
30521 294/300 Write block 3235 len 16384 data: 4 3 4 4
30522 295/300 Read block 4835 len 20480
30523 296/300 Write block 1855 len 16384 data: 3 4 4 4
30524 297/300 Write block 4382 len 12288 data: 7 7 7
30525 298/300 Read block 557 len 32768
30526 299/300 Write block 4600 len 12288 data: 4 4 3
30527 300/300 Read block 3803 len 20480
30528 301/300 Write block 2645 len 24576 data: 5 4 3 3 3 3
30529 Main task has work to do, go find it
30530 got dsc Start(0)
30531 start 0
30532 Mar 17 22:25:48.442 INFO request completed, latency_us: 262, response_code: 204, uri: /start/cid/0, method: POST, req_id: 97e49b5a-2d67-4698-bbf7-a730f910591f, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
30533 Replay: started 0, returned:Ok(())
30534 ----------------------------------------------------------------
30535 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30536 States: Offline Active Active
30537 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30538 8246 Acked Write 10 Sent Done Done false
30539 8247 Acked Write 3 Sent Done Done false
30540 8248 Acked Read 5 Sent Done Done false
30541 8249 Acked Write 8 Sent Done Done false
30542 8250 Acked Read 8 Sent Done Done false
30543 8251 Acked Read 8 Sent Done Done false
30544 8252 Acked Write 2 Sent Done Done false
30545 8253 Acked Write 9 Sent Done Done false
30546 8254 Acked Write 8 Sent Done Done false
30547 8255 Acked Read 5 Sent Done Done false
30548 8256 Acked Write 5 Sent Done Done false
30549 8257 Acked Write 5 Sent Done Done false
30550 8258 Acked Read 3 Sent Done Done false
30551 8259 Acked Read 6 Sent Done Done false
30552 8260 Acked Read 8 Sent Done Done false
30553 8261 Acked Write 5 Sent Done Done false
30554 8262 Acked Read 9 Sent Done Done false
30555 8263 Acked Read 2 Sent Done Done false
30556 8264 Acked Read 7 Sent Done Done false
30557 8265 Acked Write 2 Sent Done Done false
30558 8266 Acked Flush 0 Sent Done Done false
30559 8267 Acked Read 8 Sent Done Done false
30560 8268 Acked Read 8 Sent Done Done false
30561 8269 Acked Read 6 Sent Done Done false
30562 8270 Acked Write 7 Sent Done Done false
30563 8271 Acked Write 10 Sent Done Done false
30564 8272 Acked Read 7 Sent Done Done false
30565 8273 Acked Write 6 Sent Done Done false
30566 8274 Acked Write 10 Sent Done Done false
30567 8275 Acked Read 8 Sent Done Done false
30568 8276 Acked Write 3 Sent Done Done false
30569 8277 Acked Read 4 Sent Done Done false
30570 8278 Acked Read 7 Sent Done Done false
30571 8279 Acked Read 1 Sent Done Done false
30572 8280 Acked Read 9 Sent Done Done false
30573 8281 Acked Write 4 Sent Done Done false
30574 8282 Acked Write 9 Sent Done Done false
30575 8283 Acked Read 9 Sent Done Done false
30576 8284 Acked Read 10 Sent Done Done false
30577 8285 Acked Read 9 Sent Done Done false
30578 8286 Acked Write 6 Sent Done Done false
30579 8287 Acked Flush 0 Sent Done Done false
30580 8288 Acked Read 7 Sent Done Done false
30581 8289 Acked Read 3 Sent Done Done false
30582 8290 Acked Read 9 Sent Done Done false
30583 8291 Acked Flush 0 Sent Done Done false
30584 8292 Acked Read 9 Sent Done Done false
30585 8293 Acked Read 1 Sent Done Done false
30586 8294 Acked Write 3 Sent Done Done false
30587 8295 Acked Write 1 Sent Done Done false
30588 8296 Acked Write 1 Sent Done Done false
30589 8297 Acked Write 1 Sent Done Done false
30590 8298 Acked Read 6 Sent Done Done false
30591 8299 Acked Write 10 Sent Done Done false
30592 8300 Acked Write 9 Sent Done Done false
30593 8301 Acked Read 10 Sent Done Done false
30594 8302 Acked Write 6 Sent Done Done false
30595 8303 Acked Flush 0 Sent Done Done false
30596 8304 Acked Read 7 Sent Done Done false
30597 8305 Acked Read 1 Sent Done Done false
30598 8306 Acked Read 4 Sent Done Done false
30599 8307 Acked Write 1 Sent Done Done false
30600 8308 Acked Read 6 Sent Done Done false
30601 8309 Acked Flush 0 Sent Done Done false
30602 8310 Acked Write 8 Sent Done Done false
30603 8311 Acked Read 6 Sent Done Done false
30604 8312 Acked Write 5 Sent Done Done false
30605 8313 Acked Write 8 Sent Done Done false
30606 8314 Acked Read 9 Sent Done Done false
30607 8315 Acked Write 8 Sent Done Done false
30608 8316 Acked Write 8 Sent Done Done false
30609 8317 Acked Read 2 Sent Done Done false
30610 8318 Acked Flush 0 Sent Done Done false
30611 8319 Acked Read 8 Sent Done Done false
30612 8320 Acked Read 3 Sent Done Done false
30613 8321 Acked Read 10 Sent Done Done false
30614 8322 Acked Write 10 Sent Done Done false
30615 8323 Acked Read 1 Sent Done Done false
30616 8324 Acked Write 4 Sent Done Done false
30617 8325 Acked Read 4 Sent Done Done false
30618 8326 Acked Read 7 Sent Done Done false
30619 8327 Acked Read 1 Sent Done Done false
30620 8328 Acked Read 7 Sent Done Done false
30621 8329 Acked Write 7 Sent Done Done false
30622 8330 Acked Write 8 Sent Done Done false
30623 8331 Acked Write 5 Sent Done Done false
30624 8332 Acked Read 1 Sent Done Done false
30625 8333 Acked Flush 0 Sent Done Done false
30626 8334 Acked Write 6 Sent Done Done false
30627 8335 Acked Read 4 Sent Done Done false
30628 8336 Acked Write 2 Sent Done Done false
30629 8337 Acked Write 4 Sent Done Done false
30630 8338 Acked Write 8 Sent Done Done false
30631 8339 Acked Write 6 Sent Done Done false
30632 8340 Acked Read 9 Sent Done Done false
30633 8341 Acked Read 3 Sent Done Done false
30634 8342 Acked Read 7 Sent Done Done false
30635 8343 Acked Flush 0 Sent Done Done false
30636 8344 Acked Flush 0 Sent Done Done false
30637 8345 Acked Read 6 Sent Done Done false
30638 8346 Acked Read 8 Sent Done Done false
30639 8347 Acked Read 4 Sent Done Done false
30640 8348 Acked Read 10 Sent Done Done false
30641 8349 Acked Write 5 Sent Done Done false
30642 8350 Acked Read 3 Sent Done Done false
30643 8351 Acked Read 8 Sent Done Done false
30644 8352 Acked Write 4 Sent Done Done false
30645 8353 Acked Read 1 Sent Done Done false
30646 8354 Acked Read 9 Sent Done Done false
30647 8355 Acked Write 6 Sent Done Done false
30648 8356 Acked Write 6 Sent Done Done false
30649 8357 Acked Write 5 Sent Done Done false
30650 8358 Acked Read 8 Sent Done Done false
30651 8359 Acked Flush 0 Sent Done Done false
30652 8360 Acked Read 6 Sent Done Done false
30653 8361 Acked Read 10 Sent Done Done false
30654 8362 Acked Write 3 Sent Done Done false
30655 8363 Acked Write 8 Sent Done Done false
30656 8364 Acked Read 2 Sent Done Done false
30657 8365 Acked Write 1 Sent Done Done false
30658 8366 Acked Read 8 Sent Done Done false
30659 8367 Acked Write 10 Sent Done Done false
30660 8368 Acked Read 4 Sent Done Done false
30661 8369 Acked Write 5 Sent Done Done false
30662 8370 Acked Write 3 Sent Done Done false
30663 8371 Acked Write 4 Sent Done Done false
30664 8372 Acked Write 10 Sent Done Done false
30665 8373 Acked Write 6 Sent Done Done false
30666 8374 Acked Read 5 Sent Done Done false
30667 8375 Acked Write 2 Sent Done Done false
30668 8376 Acked Write 8 Sent Done Done false
30669 8377 Acked Read 5 Sent Done Done false
30670 8378 Acked Flush 0 Sent Done Done false
30671 8379 Acked Read 1 Sent Done Done false
30672 8380 Acked Flush 0 Sent Done Done false
30673 8381 Acked Read 5 Sent Done Done false
30674 8382 Acked Read 4 Sent Done Done false
30675 8383 Acked Flush 0 Sent Done Done false
30676 8384 Acked Write 1 Sent Done Done false
30677 8385 Acked Read 10 Sent Done Done false
30678 8386 Acked Write 10 Sent Done Done false
30679 8387 Acked Read 7 Sent Done Done false
30680 8388 Acked Read 5 Sent Done Done false
30681 8389 Acked Write 6 Sent Done Done false
30682 8390 Acked Write 8 Sent Done Done false
30683 8391 Acked Write 9 Sent Done Done false
30684 8392 Acked Flush 0 Sent Done Done false
30685 8393 Acked Write 6 Sent Done Done false
30686 8394 Acked Flush 0 Sent Done Done false
30687 8395 Acked Write 10 Sent Done Done false
30688 8396 Acked Read 1 Sent Done Done false
30689 8397 Acked Write 5 Sent Done Done false
30690 8398 Acked Write 4 Sent Done Done false
30691 8399 Acked Read 9 Sent Done Done false
30692 8400 Acked Read 1 Sent Done Done false
30693 8401 Acked Read 3 Sent Done Done false
30694 8402 Acked Read 1 Sent Done Done false
30695 8403 Acked Write 4 Sent Done Done false
30696 8404 Acked Write 2 Sent Done Done false
30697 8405 Acked Write 1 Sent Done Done false
30698 8406 Acked Flush 0 Sent Done Done false
30699 8407 Acked Read 3 Sent Done Done false
30700 8408 Acked Read 4 Sent Done Done false
30701 8409 Acked Write 9 Sent Done Done false
30702 8410 Acked Write 8 Sent Done Done false
30703 8411 Acked Read 10 Sent Done Done false
30704 8412 Acked Read 10 Sent Done Done false
30705 8413 Acked Write 5 Sent Done Done false
30706 8414 Acked Write 9 Sent Done Done false
30707 8415 Acked Write 1 Sent Done Done false
30708 8416 Acked Write 10 Sent Done Done false
30709 8417 Acked Read 6 Sent Done Done false
30710 8418 Acked Read 10 Sent Done Done false
30711 8419 Acked Write 8 Sent Done Done false
30712 8420 Acked Read 10 Sent Done Done false
30713 8421 Acked Flush 0 Sent Done Done false
30714 8422 Acked Read 6 Sent Done Done false
30715 8423 Acked Flush 0 Sent Done Done false
30716 8424 Acked Read 5 Sent Done Done false
30717 8425 Acked Read 8 Sent Done Done false
30718 8426 Acked Read 4 Sent Done Done false
30719 8427 Acked Read 7 Sent Done Done false
30720 8428 Acked Write 7 Sent Done Done false
30721 8429 Acked Flush 0 Sent Done Done false
30722 8430 Acked Read 7 Sent Done Done false
30723 8431 Acked Read 8 Sent Done Done false
30724 8432 Acked Read 3 Sent Done Done false
30725 8433 Acked Write 9 Sent Done Done false
30726 8434 Acked Write 1 Sent Done Done false
30727 8435 Acked Write 1 Sent Done Done false
30728 8436 Acked Write 10 Sent Done Done false
30729 8437 Acked Write 6 Sent Done Done false
30730 8438 Acked Read 2 Sent Done Done false
30731 8439 Acked Write 9 Sent Done Done false
30732 8440 Acked Read 9 Sent Done Done false
30733 8441 Acked Read 2 Sent Done Done false
30734 8442 Acked Flush 0 Sent Done Done false
30735 8443 Acked Flush 0 Sent Done Done false
30736 8444 Acked Read 10 Sent Done Done false
30737 8445 Acked Flush 0 Sent Done Done false
30738 8446 Acked Write 5 Sent Done Done false
30739 8447 Acked Read 4 Sent Done Done false
30740 8448 Acked Read 6 Sent Done Done false
30741 8449 Acked Write 9 Sent Done Done false
30742 8450 Acked Write 6 Sent Done Done false
30743 8451 Acked Read 4 Sent Done Done false
30744 8452 Acked Write 6 Sent Done Done false
30745 8453 Acked Write 5 Sent Done Done false
30746 8454 Acked Flush 0 Sent Done Done false
30747 8455 Acked Write 10 Sent Done Done false
30748 8456 Acked Write 8 Sent Done Done false
30749 8457 Acked Read 2 Sent Done Done false
30750 8458 Acked Read 1 Sent Done Done false
30751 8459 Acked Write 1 Sent Done Done false
30752 8460 Acked Write 7 Sent Done Done false
30753 8461 Acked Flush 0 Sent Done Done false
30754 8462 Acked Read 7 Sent Done Done false
30755 8463 Acked Write 6 Sent Done Done false
30756 8464 Acked Read 6 Sent Done Done false
30757 8465 Acked Read 1 Sent Done Done false
30758 8466 Acked Write 10 Sent Done Done false
30759 8467 Acked Flush 0 Sent Done Done false
30760 8468 Acked Read 3 Sent Done Done false
30761 8469 Acked Read 4 Sent Done Done false
30762 8470 Acked Read 5 Sent Done Done false
30763 8471 Acked Flush 0 Sent Done Done false
30764 8472 Acked Write 9 Sent Done Done false
30765 8473 Acked Read 10 Sent Done Done false
30766 8474 Acked Read 3 Sent Done Done false
30767 8475 Acked Write 1 Sent Done Done false
30768 8476 Acked Write 7 Sent Done Done false
30769 8477 Acked Write 2 Sent Done Done false
30770 8478 Acked Read 1 Sent Done Done false
30771 8479 Acked Read 1 Sent Done Done false
30772 8480 Acked Read 3 Sent Done Done false
30773 8481 Acked Read 6 Sent Done Done false
30774 8482 Acked Flush 0 Sent Done Done false
30775 8483 Acked Flush 0 Sent Done Done false
30776 8484 Acked Read 7 Sent Done Done false
30777 8485 Acked Write 4 Sent Done Done false
30778 8486 Acked Write 9 Sent Done Done false
30779 8487 Acked Write 9 Sent Done Done false
30780 8488 Acked Read 1 Sent Done Done false
30781 8489 Acked Write 6 Sent Done Done false
30782 8490 Acked Read 4 Sent Done Done false
30783 8491 Acked Flush 0 Sent Done Done false
30784 8492 Acked Write 1 Sent Done Done false
30785 8493 Acked Write 7 Sent Done Done false
30786 8494 Acked Flush 0 Sent Done Done false
30787 8495 Acked Write 9 Sent Done Done false
30788 8496 Acked Write 4 Sent Done Done false
30789 8497 Acked Write 4 Sent Done Done false
30790 8498 Acked Write 2 Sent Done Done false
30791 8499 Acked Read 8 Sent Done Done false
30792 8500 Acked Read 2 Sent Done Done false
30793 8501 Acked Flush 0 Sent Done Done false
30794 8502 Acked Write 5 Sent Done Done false
30795 8503 Acked Write 3 Sent Done Done false
30796 8504 Acked Write 6 Sent Done Done false
30797 8505 Acked Write 7 Sent Done Done false
30798 8506 Acked Write 5 Sent Done Done false
30799 8507 Acked Flush 0 Sent Done Done false
30800 8508 Acked Write 10 Sent Done Done false
30801 8509 Acked Write 6 Sent Done Done false
30802 8510 Acked Read 1 Sent Done Done false
30803 8511 Acked Write 7 Sent Done Done false
30804 8512 Acked Read 9 Sent Done Done false
30805 8513 Acked Read 8 Sent Done Done false
30806 8514 Acked Read 4 Sent Done Done false
30807 8515 Acked Write 2 Sent Done Done false
30808 8516 Acked Read 10 Sent Done Done false
30809 8517 Acked Write 6 Sent Done Done false
30810 8518 Acked Read 3 Sent Done Done false
30811 8519 Acked Read 10 Sent Done Done false
30812 8520 Acked Write 8 Sent Done Done false
30813 8521 Acked Write 3 Sent Done Done false
30814 8522 Acked Read 5 Sent Done Done false
30815 8523 Acked Read 8 Sent Done Done false
30816 8524 Acked Write 2 Sent Done Done false
30817 8525 Acked Read 9 Sent Done Done false
30818 8526 Acked Flush 0 Sent Done Done false
30819 8527 Acked Write 10 Sent Done Done false
30820 8528 Acked Read 5 Sent Done Done false
30821 8529 Acked Flush 0 Sent Done Done false
30822 8530 Acked Read 2 Sent Done Done false
30823 8531 Acked Write 8 Sent Done Done false
30824 8532 Acked Write 8 Sent Done Done false
30825 8533 Acked Read 8 Sent Done Done false
30826 8534 Acked Read 9 Sent Done Done false
30827 8535 Acked Write 6 Sent Done Done false
30828 8536 Acked Read 1 Sent Done Done false
30829 8537 Acked Read 1 Sent Done Done false
30830 8538 Acked Write 6 Sent Done Done false
30831 8539 Acked Write 4 Sent Done Done false
30832 8540 Acked Read 5 Sent Done Done false
30833 8541 Acked Write 4 Sent Done Done false
30834 8542 Acked Write 3 Sent Done Done false
30835 8543 Acked Read 8 Sent Done Done false
30836 8544 Acked Write 3 Sent Done Done false
30837 8545 Acked Read 5 Sent Done Done false
30838 8546 Acked Write 6 Sent Done Done false
30839 STATES DS:0 DS:1 DS:2 TOTAL
30840 Sent 301 0 0 301
30841 Done 0 301 301 602
30842 Skipped 0 0 0 0
30843 Error 0 0 0 0
30844 Last Flush: Some(JobId(8245)) Some(JobId(8529)) Some(JobId(8529))
30845 Downstairs last five completed: 8245 8244 8243 8242 8241
30846 Upstairs last five completed: 8546 8545 8544 8543 8542
30847 CLIENT: Up:0 ds:301 act:2
308482025-03-17T22:25:49.235ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
30849 [8810] got start action
30850 [8810] I am going to restart
30851 Starting downstairs at port 8810
30852 Make output file at "/tmp/dsc/downstairs-8810.txt"
30853 [8810][0] reports Starting
30854 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1236)
30855 [8810][0] reports Running
308562025-03-17T22:25:50.237ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
308572025-03-17T22:25:51.239ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
308582025-03-17T22:25:51.740ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
30859 {"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible"----------------------------------------------------------------
30860 ," Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30861 level"States:: Offline40 Active Active
30862 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30863 8246 Acked Write 10 Sent Done Done false
30864 8247 Acked Write 3 Sent Done Done false
30865 8248 Acked Read 5 Sent Done Done false
30866 8249 Acked Write 8 Sent Done Done false
30867 8250 Acked Read 8 Sent Done Done false
30868 8251 Acked Read 8 Sent Done Done false
30869 8252 Acked Write 2, Sent" Donetime Done" false:"
30870 2025-03-17T22:25:52.445198739Z" 8253 Acked Write 9," Senthostname":" Done Done false
30871 w-01JPK2XSVRF7343QRZWA5N22R2" 8254 Acked Write 8," Sentpid" Done: Done false
30872 1211 8255 Acked Read 5 Sent Done Done false
30873 ," 8256 Acked Write 5session_id Sent": Done" Done false
30874 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf" 8257 Acked Write 5} Sent
30875 Done Done false
30876 8258 Acked Read 3 Sent Done Done false
30877 8259 Acked Read 6 Sent Done Done false
30878 8260 Acked Read 8 Sent Done Done false
30879 8261 Acked Write 5 Sent Done Done false
30880 8262 Acked Read 9 Sent Done Done false
30881 8263 Acked Read 2 Sent Done Done false
30882 8264 Acked Read 7 Sent Done Done false
30883 8265 Acked Write 2 Sent Done Done false
30884 8266 Acked Flush 0 Sent Done Done false
30885 8267 Acked Read 8 Sent Done Done false
30886 8268 Acked Read 8 Sent Done Done false
30887 8269 Acked Read 6 Sent Done Done false
30888 8270 Acked Write 7 Sent Done Done false
30889 8271 Acked Write 10 Sent Done Done false
30890 8272 Acked Read 7 Sent Done Done false
30891 8273 Acked Write 6 Sent Done Done false
30892 8274 Acked Write 10 Sent Done Done false
30893 8275 Acked Read 8 Sent Done Done false
30894 8276 Acked Write 3 Sent Done Done false
30895 8277 Acked Read 4 Sent Done Done false
30896 8278 Acked Read 7 Sent Done Done false
30897 8279 Acked Read 1 Sent Done Done false
30898 8280 Acked Read 9 Sent Done Done false
30899 8281 Acked Write 4 Sent Done Done false
30900 8282 Acked Write 9 Sent Done Done false
30901 8283 Acked Read 9 Sent Done Done false
30902 8284 Acked Read 10 Sent Done Done false
30903 8285 Acked Read 9 Sent Done Done false
30904 8286 Acked Write 6 Sent Done Done false
30905 8287 Acked Flush 0 Sent Done Done false
30906 8288 Acked Read 7 Sent Done Done false
30907 8289 Acked Read 3 Sent Done Done false
30908 8290 Acked Read 9 Sent Done Done false
30909 8291 Acked Flush 0 Sent Done Done false
30910 8292 Acked Read 9 Sent Done Done false
30911 8293 Acked Read 1 Sent Done Done false
30912 8294 Acked Write 3 Sent Done Done false
30913 8295 Acked Write 1 Sent Done Done false
30914 8296 Acked Write 1 Sent Done Done false
30915 8297 Acked Write 1 Sent Done Done false
30916 8298 Acked Read 6 Sent Done Done false
30917 8299 Acked Write 10 Sent Done Done false
30918 8300 Acked Write 9 Sent Done Done false
30919 8301 Acked Read 10 Sent Done Done false
30920 8302 Acked Write 6 Sent Done Done false
30921 8303 Acked Flush 0 Sent Done Done false
30922 8304 Acked Read 7 Sent Done Done false
30923 8305 Acked Read 1 Sent Done Done false
30924 8306 Acked Read 4 Sent Done Done false
30925 8307 Acked Write 1 Sent Done Done false
30926 8308 Acked Read 6 Sent Done Done false
30927 8309 Acked Flush 0 Sent Done Done false
30928 8310 Acked Write 8 Sent Done Done false
30929 8311 Acked Read 6 Sent Done Done false
30930 8312 Acked Write 5 Sent Done Done false
30931 8313 Acked Write 8 Sent Done Done false
30932 8314 Acked Read 9 Sent Done Done false
30933 8315 Acked Write 8 Sent Done Done false
30934 8316 Acked Write 8 Sent Done Done false
30935 8317 Acked Read 2 Sent Done Done false
30936 8318 Acked Flush 0 Sent Done Done false
30937 8319 Acked Read 8 Sent Done Done false
30938 8320 Acked Read 3 Sent Done Done false
30939 8321 Acked Read 10 Sent Done Done false
30940 8322 Acked Write 10 Sent Done Done false
30941 8323 Acked Read 1 Sent Done Done false
30942 8324 Acked Write 4 Sent Done Done false
30943 8325 Acked Read 4 Sent Done Done false
30944 8326 Acked Read 7 Sent Done Done false
30945 8327 Acked Read 1 Sent Done Done false
30946 8328 Acked Read 7 Sent Done Done false
30947 8329 Acked Write 7 Sent Done Done false
30948 8330 Acked Write 8 Sent Done Done false
30949 8331 Acked Write 5 Sent Done Done false
30950 8332 Acked Read 1 Sent Done Done false
30951 8333 Acked Flush 0 Sent Done Done false
30952 8334 Acked Write 6 Sent Done Done false
30953 8335 Acked Read 4 Sent Done Done false
30954 8336 Acked Write 2 Sent Done Done false
30955 8337 Acked Write 4 Sent Done Done false
30956 8338 Acked Write 8 Sent Done Done false
30957 8339 Acked Write 6 Sent Done Done false
30958 8340 Acked Read 9 Sent Done Done false
30959 8341 Acked Read 3 Sent Done Done false
30960 8342 Acked Read 7 Sent Done Done false
30961 8343 Acked Flush 0 Sent Done Done false
30962 8344 Acked Flush 0 Sent Done Done false
30963 8345 Acked Read 6 Sent Done Done false
30964 8346 Acked Read 8 Sent Done Done false
30965 8347 Acked Read 4 Sent Done Done false
30966 8348 Acked Read 10 Sent Done Done false
30967 8349 Acked Write 5 Sent Done Done false
30968 8350 Acked Read 3 Sent Done Done false
30969 8351 Acked Read 8 Sent Done Done false
30970 8352 Acked Write 4 Sent Done Done false
30971 8353 Acked Read 1 Sent Done Done false
30972 8354 Acked Read 9 Sent Done Done false
30973 8355 Acked Write 6 Sent Done Done false
30974 8356 Acked Write 6 Sent Done Done false
30975 8357 Acked Write 5 Sent Done Done false
30976 8358 Acked Read 8 Sent Done Done false
30977 8359 Acked Flush 0 Sent Done Done false
30978 8360 Acked Read 6 Sent Done Done false
30979 8361 Acked Read 10 Sent Done Done false
30980 8362 Acked Write 3 Sent Done Done false
30981 8363 Acked Write 8 Sent Done Done false
30982 8364 Acked Read 2 Sent Done Done false
30983 8365 Acked Write 1 Sent Done Done false
30984 8366 Acked Read 8 Sent Done Done false
30985 8367 Acked Write 10 Sent Done Done false
30986 8368 Acked Read 4 Sent Done Done false
30987 8369 Acked Write 5 Sent Done Done false
30988 8370 Acked Write 3 Sent Done Done false
30989 8371 Acked Write 4 Sent Done Done false
30990 8372 Acked Write 10 Sent Done Done false
30991 8373 Acked Write 6 Sent Done Done false
30992 8374 Acked Read 5 Sent Done Done false
30993 8375 Acked Write 2 Sent Done Done false
30994 8376 Acked Write 8 Sent Done Done false
30995 8377 Acked Read 5 Sent Done Done false
30996 8378 Acked Flush 0 Sent Done Done false
30997 8379 Acked Read 1 Sent Done Done false
30998 8380 Acked Flush 0 Sent Done Done false
30999 8381 Acked Read 5 Sent Done Done false
31000 8382 Acked Read 4 Sent Done Done false
31001 8383 Acked Flush 0 Sent Done Done false
31002 8384 Acked Write 1 Sent Done Done false
31003 8385 Acked Read 10 Sent Done Done false
31004 8386 Acked Write 10 Sent Done Done false
31005 8387 Acked Read 7 Sent Done Done false
31006 8388 Acked Read 5 Sent Done Done false
31007 8389 Acked Write 6 Sent Done Done false
31008 8390 Acked Write 8 Sent Done Done false
31009 8391 Acked Write 9 Sent Done Done false
31010 8392 Acked Flush 0 Sent Done Done false
31011 8393 Acked Write 6 Sent Done Done false
31012 8394 Acked Flush 0 Sent Done Done false
31013 8395 Acked Write 10 Sent Done Done false
31014 8396 Acked Read 1 Sent Done Done false
31015 8397 Acked Write 5 Sent Done Done false
31016 8398 Acked Write 4 Sent Done Done false
31017 8399 Acked Read 9 Sent Done Done false
31018 8400 Acked Read 1 Sent Done Done false
31019 8401 Acked Read 3 Sent Done Done false
31020 8402 Acked Read 1 Sent Done Done false
31021 8403 Acked Write 4 Sent Done Done false
31022 8404 Acked Write 2 Sent Done Done false
31023 8405 Acked Write 1 Sent Done Done false
31024 8406 Acked Flush 0 Sent Done Done false
31025 8407 Acked Read 3 Sent Done Done false
31026 8408 Acked Read 4 Sent Done Done false
31027 8409 Acked Write 9 Sent Done Done false
31028 8410 Acked Write 8 Sent Done Done false
31029 8411 Acked Read 10 Sent Done Done false
31030 8412 Acked Read 10 Sent Done Done false
31031 8413 Acked Write 5 Sent Done Done false
31032 8414 Acked Write 9 Sent Done Done false
31033 8415 Acked Write 1 Sent Done Done false
31034 8416 Acked Write 10 Sent Done Done false
31035 8417 Acked Read 6 Sent Done Done false
31036 8418 Acked Read 10 Sent Done Done false
31037 8419 Acked Write 8 Sent Done Done false
31038 8420 Acked Read 10 Sent Done Done false
31039 8421 Acked Flush 0 Sent Done Done false
31040 8422 Acked Read 6 Sent Done Done false
31041 8423 Acked Flush 0 Sent Done Done false
31042 8424 Acked Read 5 Sent Done Done false
31043 8425 Acked Read 8 Sent Done Done false
31044 8426 Acked Read 4 Sent Done Done false
31045 8427 Acked Read 7 Sent Done Done false
31046 8428 Acked Write 7 Sent Done Done false
31047 8429 Acked Flush 0 Sent Done Done false
31048 8430 Acked Read 7 Sent Done Done false
31049 8431 Acked Read 8 Sent Done Done false
31050 8432 Acked Read 3 Sent Done Done false
31051 8433 Acked Write 9 Sent Done Done false
31052 8434 Acked Write 1 Sent Done Done false
31053 8435 Acked Write 1 Sent Done Done false
31054 8436 Acked Write 10 Sent Done Done false
31055 8437 Acked Write 6 Sent Done Done false
31056 8438 Acked Read 2 Sent Done Done false
31057 8439 Acked Write 9 Sent Done Done false
31058 8440 Acked Read 9 Sent Done Done false
31059 8441 Acked Read 2 Sent Done Done false
31060 8442 Acked Flush 0 Sent Done Done false
31061 8443 Acked Flush 0 Sent Done Done false
31062 8444 Acked Read 10 Sent Done Done false
31063 8445 Acked Flush 0 Sent Done Done false
31064 8446 Acked Write 5 Sent Done Done false
31065 8447 Acked Read 4 Sent Done Done false
31066 8448 Acked Read 6 Sent Done Done false
31067 8449 Acked Write 9 Sent Done Done false
31068 8450 Acked Write 6 Sent Done Done false
31069 8451 Acked Read 4 Sent Done Done false
31070 8452 Acked Write 6 Sent Done Done false
31071 8453 Acked Write 5 Sent Done Done false
31072 8454 Acked Flush 0 Sent Done Done false
31073 8455 Acked Write 10 Sent Done Done false
31074 8456 Acked Write 8 Sent Done Done false
31075 8457 Acked Read 2 Sent Done Done false
31076 8458 Acked Read 1 Sent Done Done false
31077 8459 Acked Write 1 Sent Done Done false
31078 8460 Acked Write 7 Sent Done Done false
31079 8461 Acked Flush 0 Sent Done Done false
31080 8462 Acked Read 7 Sent Done Done false
31081 8463 Acked Write 6 Sent Done Done false
31082 8464 Acked Read 6 Sent Done Done false
31083 8465 Acked Read 1 Sent Done Done false
31084 8466 Acked Write 10 Sent Done Done false
31085 8467 Acked Flush 0 Sent Done Done false
31086 8468 Acked Read 3 Sent Done Done false
31087 8469 Acked Read 4 Sent Done Done false
31088 8470 Acked Read 5 Sent Done Done false
31089 8471 Acked Flush 0 Sent Done Done false
31090 8472 Acked Write 9 Sent Done Done false
31091 8473 Acked Read 10 Sent Done Done false
31092 8474 Acked Read 3 Sent Done Done false
31093 8475 Acked Write 1 Sent Done Done false
31094 8476 Acked Write 7 Sent Done Done false
31095 8477 Acked Write 2 Sent Done Done false
31096 8478 Acked Read 1 Sent Done Done false
31097 8479 Acked Read 1 Sent Done Done false
31098 8480 Acked Read 3 Sent Done Done false
31099 8481 Acked Read 6 Sent Done Done false
31100 8482 Acked Flush 0 Sent Done Done false
31101 8483 Acked Flush 0 Sent Done Done false
31102 8484 Acked Read 7 Sent Done Done false
31103 8485 Acked Write 4 Sent Done Done false
31104 8486 Acked Write 9 Sent Done Done false
31105 8487 Acked Write 9 Sent Done Done false
31106 8488 Acked Read 1 Sent Done Done false
31107 8489 Acked Write 6 Sent Done Done false
31108 8490 Acked Read 4 Sent Done Done false
31109 8491 Acked Flush 0 Sent Done Done false
31110 8492 Acked Write 1 Sent Done Done false
31111 8493 Acked Write 7 Sent Done Done false
31112 8494 Acked Flush 0 Sent Done Done false
31113 8495 Acked Write 9 Sent Done Done false
31114 8496 Acked Write 4 Sent Done Done false
31115 8497 Acked Write 4 Sent Done Done false
31116 8498 Acked Write 2 Sent Done Done false
31117 8499 Acked Read 8 Sent Done Done false
31118 8500 Acked Read 2 Sent Done Done false
31119 8501 Acked Flush 0 Sent Done Done false
31120 8502 Acked Write 5 Sent Done Done false
31121 8503 Acked Write 3 Sent Done Done false
31122 8504 Acked Write 6 Sent Done Done false
31123 8505 Acked Write 7 Sent Done Done false
31124 8506 Acked Write 5 Sent Done Done false
31125 8507 Acked Flush 0 Sent Done Done false
31126 8508 Acked Write 10 Sent Done Done false
31127 8509 Acked Write 6 Sent Done Done false
31128 8510 Acked Read 1 Sent Done Done false
31129 8511 Acked Write 7 Sent Done Done false
31130 8512 Acked Read 9 Sent Done Done false
31131 8513 Acked Read 8 Sent Done Done false
31132 8514 Acked Read 4 Sent Done Done false
31133 8515 Acked Write 2 Sent Done Done false
31134 8516 Acked Read 10 Sent Done Done false
31135 8517 Acked Write 6 Sent Done Done false
31136 8518 Acked Read 3 Sent Done Done false
31137 8519 Acked Read 10 Sent Done Done false
31138 8520 Acked Write 8 Sent Done Done false
31139 8521 Acked Write 3 Sent Done Done false
31140 8522 Acked Read 5 Sent Done Done false
31141 8523 Acked Read 8 Sent Done Done false
31142 8524 Acked Write 2 Sent Done Done false
31143 8525 Acked Read 9 Sent Done Done false
31144 8526 Acked Flush 0 Sent Done Done false
31145 8527 Acked Write 10 Sent Done Done false
31146 8528 Acked Read 5 Sent Done Done false
31147 8529 Acked Flush 0 Sent Done Done false
31148 8530 Acked Read 2 Sent Done Done false
31149 8531 Acked Write 8 Sent Done Done false
31150 8532 Acked Write 8 Sent Done Done false
31151 8533 Acked Read 8 Sent Done Done false
31152 8534 Acked Read 9 Sent Done Done false
31153 8535 Acked Write 6 Sent Done Done false
31154 8536 Acked Read 1 Sent Done Done false
31155 8537 Acked Read 1 Sent Done Done false
31156 8538 Acked Write 6 Sent Done Done false
31157 8539 Acked Write 4 Sent Done Done false
31158 8540 Acked Read 5 Sent Done Done false
31159 8541 Acked Write 4 Sent Done Done false
31160 8542 Acked Write 3 Sent Done Done false
31161 8543 Acked Read 8 Sent Done Done false
31162 8544 Acked Write 3 Sent Done Done false
31163 8545 Acked Read 5 Sent Done Done false
31164 8546 Acked Write 6 Sent Done Done false
31165 STATES DS:0 DS:1 DS:2 TOTAL
31166 Sent 301 0 0 301
31167 Done 0 301 301 602
31168 Skipped 0 0 0 0
31169 Error 0 0 0 0
31170 Last Flush: Some(JobId(8245)) Some(JobId(8529)) Some(JobId(8529))
31171 Downstairs last five completed: 8245 8244 8243 8242 8241
31172 Upstairs last five completed: 8546 8545 8544 8543 8542
31173 CLIENT: Up:0 ds:301 act:2
311742025-03-17T22:25:53.242ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
311752025-03-17T22:25:54.202ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
311762025-03-17T22:25:54.202ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:62496) = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
311772025-03-17T22:25:54.203ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
311782025-03-17T22:25:54.204ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
311792025-03-17T22:25:54.204ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
311802025-03-17T22:25:54.204ZINFOcrucible: send last flush ID to this DS: Some(JobId(8245)) = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
311812025-03-17T22:25:54.204ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
311822025-03-17T22:25:54.204ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(8245)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
311832025-03-17T22:25:54.204ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(8245)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
31184 ----------------------------------------------------------------
31185 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
31186 Downstairs last five completed: 8547 8546 8545 8544 8543
31187 Upstairs last five completed: 8547 8546 8545 8544 8543
31188 CLIENT: Up:0 ds:0 act:3
31189 Replay: All jobs finished
31190 Main task has work to do, go find it
31191 got dsc Stop(0)
31192 stop 0
31193 [8810] Got stop action so:false kr:false
31194 Mar 17 22:25:56.448 INFO request completed, latency_us: 371, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 95187d43-fa19-483b-a504-5fce8bfa0049, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
311952025-03-17T22:25:56.449ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
311962025-03-17T22:25:56.449ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
31197 Mar 17 22:25:56.449 INFO request completed, latency_us: 154, response_code: 200, uri: /state/cid/0, method{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:25:56.449638575Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
31198 : GET, req_id: 5ef8b04c-f1e1-4e65-927c-9728d407e886, remote_addr: 127.0.0.1{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:25:56.44968612Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"":"io task","client":"0","":"downstairs","session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
31199 :58317, local_addr: 127.0.0.1:9998
31200 [8810] Exited with: ExitStatus(unix_wait_status(9))
31201 [8810][0] reports Exit
31202 Mar 17 22:26:00.451 INFO request completed, latency_us: 214, response_code: 200, uri: /state/cid/0, method: GET, req_id: d50e073c-b286-4016-809c-7a0773a35b39, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
31203 001/300 Write block 3640 len 28672 data: 7 5 4 4 4 4 4
31204 002/300 Flush
31205 003/300 Read block 4240 len 4096
31206 004/300 Read block 4306 len 36864
31207 005/300 Flush
31208 006/300 Write block 3895 len 12288 data: 4 4 3
31209 007/300 Read block 4850 len 32768
31210 008/300 Write block 977 len 36864 data: 7 6 5 5 6 6 5 5 5
31211 009/300 Write block 2547 len 32768 data: 5 6 8 8 6 7 7 7
31212 010/300 Write block 3700 len 32768 data: 6 6 6 4 3 3 2 2
31213 011/300 Flush
31214 012/300 Write block 2200 len 24576 data: 3 3 5 7 5 5
31215 013/300 Flush
31216 014/300 Write block 1384 len 20480 data: 4 4 4 4 3
31217 015/300 Read block 753 len 36864
31218 016/300 Flush
31219 017/300 Write block 4119 len 16384 data: 5 6 6 5
31220 018/300 Read block 2930 len 40960
31221 019/300 Flush
31222 020/300 Flush
31223 021/300 Read block 3245 len 32768
31224 022/300 Write block 3648 len 32768 data: 5 4 4 5 8 10 10 9
31225 023/300 Write block 1106 len 28672 data: 5 5 6 7 8 8 7
31226 024/300 Read block 701 len 20480
31227 025/300 Read block 2546 len 36864
31228 026/300 Read block 4390 len 8192
31229 027/300 Read block 3490 len 24576
31230 028/300 Read block 1998 len 36864
31231 029/300 Read block 3243 len 40960
31232 030/300 Read block 1675 len 32768
31233 031/300 Write block 2734 len 32768 data: 5 7 7 7 6 7 6 5
31234 032/300 Write block 2827 len 28672 data: 7 6 4 4 5 4 3
31235 033/300 Write block 340 len 24576 data: 6 7 6 6 6 5
31236 034/300 Read block 3852 len 20480
31237 035/300 Write block 2404 len 12288 data: 6 7 6
31238 036/300 Write block 239 len 32768 data: 3 4 5 5 5 5 6 7
31239 037/300 Write block 2687 len 8192 data: 9 9
31240 038/300 Read block 2863 len 28672
31241 039/300 Write block 1270 len 8192 data: 4 5
31242 040/300 Flush
31243 041/300 Write block 3990 len 28672 data: 3 3 3 4 5 4 4
31244 042/300 Read block 2788 len 12288
31245 043/300 Flush
31246 044/300 Read block 1971 len 40960
31247 045/300 Write block 1298 len 24576 data: 5 4 4 3 3 2
31248 046/300 Write block 386 len 24576 data: 4 4 6 7 6 6
31249 047/300 Read block 1093 len 28672
31250 048/300 Read block 343 len 8192
31251 049/300 Write block 735 len 24576 data: 4 4 5 7 7 11
31252 050/300 Write block 4636 len 24576 data: 4 5 5 5 6 8
31253 051/300 Read block 1804 len 36864
31254 052/300 Flush
31255 053/300 Write block 4147 len 12288 data: 5 5 6
31256 054/300 Read block 2983 len 4096
31257 055/300 Read block 1130 len 20480
31258 056/300 Read block 3609 len 12288
31259 057/300 Read block 2512 len 36864
31260 058/300 Read block 2241 len 40960
31261 059/300 Read block 228 len 24576
31262 060/300 Read block 2459 len 28672
31263 061/300 Read block 3149 len 16384
31264 062/300 Read block 1569 len 20480
31265 063/300 Write block 4201 len 36864 data: 5 5 5 6 4 4 4 7 7
31266 064/300 Write block 3232 len 20480 data: 6 5 5 5 4
31267 065/300 Write block 4317 len 8192 data: 4 5
31268 066/300 Write block 1454 len 40960 data: 4 4 4 4 6 5 4 3 4 4
31269 067/300 Read block 1359 len 12288
31270 068/300 Write block 1368 len 4096 data: 4
31271 069/300 Write block 3407 len 20480 data: 6 7 6 6 7
31272 070/300 Write block 2375 len 12288 data: 5 5 5
31273 071/300 Read block 863 len 28672
31274 072/300 Write block 751 len 16384 data: 4 5 6 7
31275 073/300 Write block 2986 len 20480 data: 4 3 3 3 4
31276 074/300 Read block 1266 len 20480
31277 075/300 Flush
31278 076/300 Read block 378 len 20480
31279 077/300 Read block 4545 len 16384
31280 078/300 Write block 1355 len 24576 data: 6 6 6 8 7 6
31281 079/300 Read block 2874 len 40960
31282 080/300 Write block 3349 len 20480 data: 8 8 5 6 6
31283 081/300 Write block 4856 len 36864 data: 5 6 7 6 7 7 8 7 8
31284 082/300 Read block 2905 len 16384
31285 083/300 Write block 4267 len 36864 data: 4 5 6 4 5 6 7 6 6
31286 084/300 Read block 1791 len 28672
31287 085/300 Read block 2919 len 40960
31288 086/300 Write block 1168 len 20480 data: 5 4 3 3 3
31289 087/300 Flush
31290 088/300 Write block 2534 len 12288 data: 6 7 6
31291 089/300 Read block 1057 len 40960
31292 090/300 Write block 441 len 4096 data: 2
31293 091/300 Read block 275 len 36864
31294 092/300 Read block 3500 len 36864
31295 093/300 Flush
31296 094/300 Read block 3983 len 4096
31297 095/300 Write block 604 len 28672 data: 6 7 7 7 7 10 10
31298 096/300 Write block 4343 len 12288 data: 7 6 8
31299 097/300 Write block 2195 len 12288 data: 5 5 6
31300 098/300 Write block 1621 len 12288 data: 4 5 4
31301 099/300 Flush
31302 100/300 Write block 4426 len 8192 data: 5 5
31303 101/300 Write block 2459 len 24576 data: 4 5 4 3 3 3
31304 102/300 Read block 3126 len 28672
31305 103/300 Read block 2134 len 20480
31306 104/300 Write block 2948 len 36864 data: 8 8 8 9 9 9 8 8 7
31307 105/300 Read block 2849 len 28672
31308 106/300 Read block 1720 len 20480
31309 107/300 Write block 4251 len 28672 data: 5 7 5 7 6 7 7
31310 108/300 Write block 2035 len 4096 data: 4
31311 109/300 Write block 2601 len 12288 data: 6 9 9
31312 110/300 Read block 2897 len 12288
31313 111/300 Read block 4548 len 4096
31314 112/300 Read block 4745 len 12288
31315 113/300 Read block 1243 len 24576
31316 114/300 Read block 2142 len 20480
31317 115/300 Read block 1707 len 28672
31318 116/300 Write block 4786 len 32768 data: 5 5 5 5 4 3 3 4
31319 117/300 Read block 2593 len 24576
31320 118/300 Flush
31321 119/300 Read block 1222 len 20480
31322 120/300 Read block 4459 len 16384
31323 121/300 Flush
31324 122/300 Write block 1219 len 36864 data: 3 3 3 3 2 2 3 3 3
31325 123/300 Write block 1857 len 40960 data: 5 5 4 4 4 5 5 5 5 7
31326 124/300 Flush
31327 125/300 Write block 1466 len 32768 data: 3 3 3 4 5 8 9 9
31328 126/300 Flush
31329 127/300 Read block 2081 len 4096
31330 128/300 Read block 3534 len 12288
31331 129/300 Write block 3894 len 32768 data: 4 5 5 4 3 4 3 3
31332 130/300 Read block 949 len 28672
31333 131/300 Read block 1605 len 24576
31334 132/300 Read block 2261 len 36864
31335 133/300 Read block 141 len 24576
31336 134/300 Flush
31337 135/300 Flush
31338 136/300 Read block 2763 len 28672
31339 137/300 Write block 1901 len 24576 data: 5 4 5 5 8 9
31340 138/300 Write block 1524 len 12288 data: 5 4 4
31341 139/300 Write block 3298 len 16384 data: 3 4 4 4
31342 140/300 Read block 1366 len 24576
31343 141/300 Write block 442 len 8192 data: 4 6
31344 142/300 Read block 334 len 12288
31345 143/300 Read block 2705 len 20480
31346 144/300 Read block 3694 len 32768
31347 145/300 Read block 928 len 16384
31348 146/300 Write block 1326 len 36864 data: 6 8 6 6 9 9 8 8 8
31349 147/300 Read block 2348 len 4096
31350 148/300 Flush
31351 149/300 Write block 2953 len 24576 data: 10 9 9 8 6 6
31352 150/300 Read block 3881 len 24576
31353 151/300 Write block 2014 len 32768 data: 7 8 7 6 7 7 5 5
31354 152/300 Read block 2188 len 24576
31355 153/300 Flush
31356 154/300 Write block 2774 len 28672 data: 4 4 3 3 3 4 4
31357 155/300 Write block 4772 len 20480 data: 5 4 3 4 5
31358 156/300 Read block 1502 len 28672
31359 157/300 Write block 4100 len 4096 data: 4
31360 158/300 Write block 2954 len 36864 data: 10 10 9 7 7 7 8 6 7
31361 159/300 Write block 2318 len 28672 data: 7 6 5 4 4 4 3
31362 160/300 Write block 2166 len 36864 data: 6 6 7 7 8 6 7 8 7
31363 161/300 Read block 3406 len 12288
31364 162/300 Read block 456 len 4096
31365 163/300 Write block 2217 len 20480 data: 3 3 4 4 6
31366 164/300 Flush
31367 165/300 Read block 2578 len 4096
31368 166/300 Write block 3305 len 24576 data: 4 4 4 4 4 4
31369 167/300 Write block 3111 len 32768 data: 8 7 8 9 8 8 8 7
31370 168/300 Read block 1955 len 32768
31371 169/300 Read block 4891 len 8192
31372 170/300 Write block 2329 len 12288 data: 4 4 4
31373 171/300 Read block 4073 len 12288
31374 172/300 Read block 4230 len 16384
31375 173/300 Flush
31376 174/300 Flush
31377 175/300 Read block 2848 len 16384
31378 176/300 Write block 3683 len 40960 data: 8 7 7 6 6 6 6 6 6 7
31379 177/300 Read block 1655 len 40960
31380 178/300 Read block 2909 len 40960
31381 179/300 Write block 2574 len 16384 data: 7 7 7 8
31382 180/300 Write block 1055 len 4096 data: 9
31383 181/300 Write block 4789 len 32768 data: 6 5 4 4 5 3 4 4
31384 182/300 Flush
31385 183/300 Write block 4042 len 4096 data: 3
31386 184/300 Read block 3421 len 4096
31387 185/300 Read block 864 len 8192
31388 186/300 Write block 3755 len 24576 data: 8 8 7 7 8 5
31389 187/300 Write block 3319 len 28672 data: 6 6 8 7 7 5 5
31390 188/300 Write block 1032 len 4096 data: 5
31391 189/300 Write block 4889 len 24576 data: 5 5 4 4 4 5
31392 190/300 Flush
31393 191/300 Write block 753 len 40960 data: 7 8 7 8 8 7 8 7 7 7
31394 192/300 Write block 316 len 12288 data: 8 8 9
31395 193/300 Write block 786 len 32768 data: 8 7 7 7 6 6 5 4
31396 194/300 Read block 3690 len 20480
31397 195/300 Write block 3537 len 12288 data: 8 8 6
31398 196/300 Write block 2450 len 16384 data: 7 4 3 3
31399 197/300 Read block 167 len 24576
31400 198/300 Flush
31401 199/300 Read block 3874 len 20480
31402 200/300 Flush
31403 201/300 Read block 140 len 4096
31404 202/300 Read block 4462 len 28672
31405 203/300 Write block 405 len 20480 data: 4 5 6 6 5
31406 204/300 Read block 3341 len 20480
31407 205/300 Read block 4605 len 20480
31408 206/300 Flush
31409 207/300 Write block 689 len 32768 data: 6 4 4 5 5 5 6 6
31410 208/300 Read block 2443 len 16384
31411 209/300 Write block 2898 len 20480 data: 9 9 11 11 10
31412 210/300 Write block 2400 len 32768 data: 8 7 6 5 7 8 7 5
31413 211/300 Write block 317 len 28672 data: 9 10 7 7 8 6 6
31414 212/300 Read block 4922 len 36864
31415 213/300 Read block 328 len 40960
31416 214/300 Write block 3081 len 4096 data: 8
31417 215/300 Flush
31418 216/300 Write block 1252 len 24576 data: 11 10 8 8 7 6
31419 217/300 Read block 2934 len 4096
31420 218/300 Read block 8 len 28672
31421 219/300 Read block 87 len 8192
31422 220/300 Flush
31423 221/300 Write block 3816 len 32768 data: 4 3 4 4 5 6 6 6
31424 222/300 Write block 2105 len 40960 data: 3 4 3 4 6 6 7 6 5 6
31425 223/300 Read block 4311 len 36864
31426 224/300 Read block 2581 len 20480
31427 225/300 Read block 1672 len 16384
31428 226/300 Read block 4054 len 16384
31429 227/300 Write block 3716 len 28672 data: 5 6 7 6 7 7 8
31430 228/300 Read block 4451 len 32768
31431 229/300 Read block 420 len 4096
31432 230/300 Read block 1809 len 24576
31433 231/300 Write block 2063 len 20480 data: 4 6 7 9 8
31434 232/300 Read block 1823 len 28672
31435 233/300 Read block 3766 len 8192
31436 234/300 Write block 3917 len 8192 data: 5 4
31437 235/300 Write block 2533 len 40960 data: 8 7 8 7 6 6 6 7 7 6
31438 236/300 Read block 4138 len 28672
31439 237/300 Write block 1171 len 24576 data: 4 4 3 3 3 3
31440 238/300 Read block 1661 len 32768
31441 239/300 Write block 2952 len 36864 data: 10 11 11 11 10 8 8 8 9
31442 240/300 Read block 2579 len 12288
31443 241/300 Write block 3011 len 40960 data: 4 4 4 4 5 6 9 8 8 8
31444 242/300 Read block 103 len 16384
31445 243/300 Write block 1269 len 12288 data: 5 5 6
31446 244/300 Write block 1784 len 28672 data: 6 5 4 3 2 4 4
31447 245/300 Write block 185 len 8192 data: 6 5
31448 246/300 Read block 4450 len 32768
31449 247/300 Write block 2457 len 24576 data: 4 4 5 6 5 4
31450 248/300 Read block 3606 len 4096
31451 249/300 Read block 1342 len 8192
31452 250/300 Read block 1758 len 4096
31453 251/300 Read block 3029 len 16384
31454 252/300 Write block 416 len 20480 data: 9 9 8 9 10
31455 253/300 Read block 4040 len 36864
31456 254/300 Read block 4193 len 16384
31457 255/300 Read block 3693 len 12288
31458 256/300 Read block 1515 len 20480
31459 257/300 Write block 1793 len 16384 data: 6 6 6 6
31460 258/300 Read block 4707 len 32768
31461 259/300 Read block 4819 len 40960
31462 260/300 Read block 4559 len 28672
31463 261/300 Write block 2207 len 8192 data: 4 4
31464 262/300 Read block 3714 len 20480
31465 263/300 Write block 1778 len 24576 data: 5 3 3 4 6 6
31466 264/300 Read block 752 len 36864
31467 265/300 Read block 3165 len 16384
31468 266/300 Write block 2834 len 4096 data: 3
31469 267/300 Read block 3601 len 24576
31470 268/300 Flush
31471 269/300 Write block 3183 len 4096 data: 8
31472 270/300 Write block 2292 len 32768 data: 6 7 5 5 5 6 7 6
31473 271/300 Write block 729 len 36864 data: 7 6 4 5 5 5 5 5 6
31474 272/300 Read block 2197 len 8192
31475 273/300 Read block 1028 len 12288
31476 274/300 Write block 3402 len 16384 data: 7 8 6 4
31477 275/300 Read block 1840 len 4096
31478 276/300 Write block 2151 len 16384 data: 4 4 4 4
31479 277/300 Write block 3835 len 36864 data: 6 5 4 3 3 3 3 3 4
31480 278/300 Read block 771 len 36864
31481 279/300 Flush
31482 280/300 Read block 3853 len 16384
31483 281/300 Write block 246 len 12288 data: 8 4 3
31484 282/300 Write block 2489 len 20480 data: 4 5 6 6 8
31485 283/300 Write block 4583 len 4096 data: 6
31486 284/300 Write block 3434 len 8192 data: 7 8
31487 285/300 Read block 308 len 8192
31488 286/300 Read block 4280 len 4096
31489 287/300 Read block 254 len 32768
31490 288/300 Read block 916 len 4096
31491 289/300 Write block 1694 len 20480 data: 4 6 5 5 5
31492 290/300 Write block 1315 len 12288 data: 3 4 4
31493 291/300 Read block 343 len 8192
31494 292/300 Read block 3845 len 8192
31495 293/300 Read block 2639 len 28672
31496 294/300 Read block 159 len 24576
31497 295/300 Write block 3522 len 24576 data: 7 8 6 9 6 6
31498 296/300 Write block 4457 len 36864 data: 5 4 4 3 3 3 3 3 3
31499 297/300 Read block 494 len 4096
31500 298/300 Read block 3872 len 4096
31501 299/300 Write block 1107 len 4096 data: 6
31502 300/300 Write block 2242 len 16384 data: 7 8 7 8
31503 301/300 Read block 662 len 4096
31504 Main task has work to do, go find it
31505 got dsc Start(0)
31506 start 0
31507 Mar 17 22:26:00.687 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/0, method: POST, req_id: bc41bf8d-f9b0-4073-aad0-50e54c17f6d2, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
31508 Replay: started 0, returned:Ok(())
31509 ----------------------------------------------------------------
31510 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
31511 States: Offline Active Active
31512 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31513 8548 Acked Write 7 Sent Done Done false
31514 8549 Acked Flush 0 Sent Done Done false
31515 8550 Acked Read 1 Sent Done Done false
31516 8551 Acked Read 9 Sent Done Done false
31517 8552 Acked Flush 0 Sent Done Done false
31518 8553 Acked Write 3 Sent Done Done false
31519 8554 Acked Read 8 Sent Done Done false
31520 8555 Acked Write 9 Sent Done Done false
31521 8556 Acked Write 8 Sent Done Done false
31522 8557 Acked Write 8 Sent Done Done false
31523 8558 Acked Flush 0 Sent Done Done false
31524 8559 Acked Write 6 Sent Done Done false
31525 8560 Acked Flush 0 Sent Done Done false
31526 8561 Acked Write 5 Sent Done Done false
31527 8562 Acked Read 9 Sent Done Done false
31528 8563 Acked Flush 0 Sent Done Done false
31529 8564 Acked Write 4 Sent Done Done false
31530 8565 Acked Read 10 Sent Done Done false
31531 8566 Acked Flush 0 Sent Done Done false
31532 8567 Acked Flush 0 Sent Done Done false
31533 8568 Acked Read 8 Sent Done Done false
31534 8569 Acked Write 8 Sent Done Done false
31535 8570 Acked Write 7 Sent Done Done false
31536 8571 Acked Read 5 Sent Done Done false
31537 8572 Acked Read 9 Sent Done Done false
31538 8573 Acked Read 2 Sent Done Done false
31539 8574 Acked Read 6 Sent Done Done false
31540 8575 Acked Read 9 Sent Done Done false
31541 8576 Acked Read 10 Sent Done Done false
31542 8577 Acked Read 8 Sent Done Done false
31543 8578 Acked Write 8 Sent Done Done false
31544 8579 Acked Write 7 Sent Done Done false
31545 8580 Acked Write 6 Sent Done Done false
31546 8581 Acked Read 5 Sent Done Done false
31547 8582 Acked Write 3 Sent Done Done false
31548 8583 Acked Write 8 Sent Done Done false
31549 8584 Acked Write 2 Sent Done Done false
31550 8585 Acked Read 7 Sent Done Done false
31551 8586 Acked Write 2 Sent Done Done false
31552 8587 Acked Flush 0 Sent Done Done false
31553 8588 Acked Write 7 Sent Done Done false
31554 8589 Acked Read 3 Sent Done Done false
31555 8590 Acked Flush 0 Sent Done Done false
31556 8591 Acked Read 10 Sent Done Done false
31557 8592 Acked Write 6 Sent Done Done false
31558 8593 Acked Write 6 Sent Done Done false
31559 8594 Acked Read 7 Sent Done Done false
31560 8595 Acked Read 2 Sent Done Done false
31561 8596 Acked Write 6 Sent Done Done false
31562 8597 Acked Write 6 Sent Done Done false
31563 8598 Acked Read 9 Sent Done Done false
31564 8599 Acked Flush 0 Sent Done Done false
31565 8600 Acked Write 3 Sent Done Done false
31566 8601 Acked Read 1 Sent Done Done false
31567 8602 Acked Read 5 Sent Done Done false
31568 8603 Acked Read 3 Sent Done Done false
31569 8604 Acked Read 9 Sent Done Done false
31570 8605 Acked Read 10 Sent Done Done false
31571 8606 Acked Read 6 Sent Done Done false
31572 8607 Acked Read 7 Sent Done Done false
31573 8608 Acked Read 4 Sent Done Done false
31574 8609 Acked Read 5 Sent Done Done false
31575 8610 Acked Write 9 Sent Done Done false
31576 8611 Acked Write 5 Sent Done Done false
31577 8612 Acked Write 2 Sent Done Done false
31578 8613 Acked Write 10 Sent Done Done false
31579 8614 Acked Read 3 Sent Done Done false
31580 8615 Acked Write 1 Sent Done Done false
31581 8616 Acked Write 5 Sent Done Done false
31582 8617 Acked Write 3 Sent Done Done false
31583 8618 Acked Read 7 Sent Done Done false
31584 8619 Acked Write 4 Sent Done Done false
31585 8620 Acked Write 5 Sent Done Done false
31586 8621 Acked Read 5 Sent Done Done false
31587 8622 Acked Flush 0 Sent Done Done false
31588 8623 Acked Read 5 Sent Done Done false
31589 8624 Acked Read 4 Sent Done Done false
31590 8625 Acked Write 6 Sent Done Done false
31591 8626 Acked Read 10 Sent Done Done false
31592 8627 Acked Write 5 Sent Done Done false
31593 8628 Acked Write 9 Sent Done Done false
31594 8629 Acked Read 4 Sent Done Done false
31595 8630 Acked Write 9 Sent Done Done false
31596 8631 Acked Read 7 Sent Done Done false
31597 8632 Acked Read 10 Sent Done Done false
31598 8633 Acked Write 5 Sent Done Done false
31599 8634 Acked Flush 0 Sent Done Done false
31600 8635 Acked Write 3 Sent Done Done false
31601 8636 Acked Read 10 Sent Done Done false
31602 8637 Acked Write 1 Sent Done Done false
31603 8638 Acked Read 9 Sent Done Done false
31604 8639 Acked Read 9 Sent Done Done false
31605 8640 Acked Flush 0 Sent Done Done false
31606 8641 Acked Read 1 Sent Done Done false
31607 8642 Acked Write 7 Sent Done Done false
31608 8643 Acked Write 3 Sent Done Done false
31609 8644 Acked Write 3 Sent Done Done false
31610 8645 Acked Write 3 Sent Done Done false
31611 8646 Acked Flush 0 Sent Done Done false
31612 8647 Acked Write 2 Sent Done Done false
31613 8648 Acked Write 6 Sent Done Done false
31614 8649 Acked Read 7 Sent Done Done false
31615 8650 Acked Read 5 Sent Done Done false
31616 8651 Acked Write 9 Sent Done Done false
31617 8652 Acked Read 7 Sent Done Done false
31618 8653 Acked Read 5 Sent Done Done false
31619 8654 Acked Write 7 Sent Done Done false
31620 8655 Acked Write 1 Sent Done Done false
31621 8656 Acked Write 3 Sent Done Done false
31622 8657 Acked Read 3 Sent Done Done false
31623 8658 Acked Read 1 Sent Done Done false
31624 8659 Acked Read 3 Sent Done Done false
31625 8660 Acked Read 6 Sent Done Done false
31626 8661 Acked Read 5 Sent Done Done false
31627 8662 Acked Read 7 Sent Done Done false
31628 8663 Acked Write 8 Sent Done Done false
31629 8664 Acked Read 6 Sent Done Done false
31630 8665 Acked Flush 0 Sent Done Done false
31631 8666 Acked Read 5 Sent Done Done false
31632 8667 Acked Read 4 Sent Done Done false
31633 8668 Acked Flush 0 Sent Done Done false
31634 8669 Acked Write 9 Sent Done Done false
31635 8670 Acked Write 10 Sent Done Done false
31636 8671 Acked Flush 0 Sent Done Done false
31637 8672 Acked Write 8 Sent Done Done false
31638 8673 Acked Flush 0 Sent Done Done false
31639 8674 Acked Read 1 Sent Done Done false
31640 8675 Acked Read 3 Sent Done Done false
31641 8676 Acked Write 8 Sent Done Done false
31642 8677 Acked Read 7 Sent Done Done false
31643 8678 Acked Read 6 Sent Done Done false
31644 8679 Acked Read 9 Sent Done Done false
31645 8680 Acked Read 6 Sent Done Done false
31646 8681 Acked Flush 0 Sent Done Done false
31647 8682 Acked Flush 0 Sent Done Done false
31648 8683 Acked Read 7 Sent Done Done false
31649 8684 Acked Write 6 Sent Done Done false
31650 8685 Acked Write 3 Sent Done Done false
31651 8686 Acked Write 4 Sent Done Done false
31652 8687 Acked Read 6 Sent Done Done false
31653 8688 Acked Write 2 Sent Done Done false
31654 8689 Acked Read 3 Sent Done Done false
31655 8690 Acked Read 5 Sent Done Done false
31656 8691 Acked Read 8 Sent Done Done false
31657 8692 Acked Read 4 Sent Done Done false
31658 8693 Acked Write 9 Sent Done Done false
31659 8694 Acked Read 1 Sent Done Done false
31660 8695 Acked Flush 0 Sent Done Done false
31661 8696 Acked Write 6 Sent Done Done false
31662 8697 Acked Read 6 Sent Done Done false
31663 8698 Acked Write 8 Sent Done Done false
31664 8699 Acked Read 6 Sent Done Done false
31665 8700 Acked Flush 0 Sent Done Done false
31666 8701 Acked Write 7 Sent Done Done false
31667 8702 Acked Write 5 Sent Done Done false
31668 8703 Acked Read 7 Sent Done Done false
31669 8704 Acked Write 1 Sent Done Done false
31670 8705 Acked Write 9 Sent Done Done false
31671 8706 Acked Write 7 Sent Done Done false
31672 8707 Acked Write 9 Sent Done Done false
31673 8708 Acked Read 3 Sent Done Done false
31674 8709 Acked Read 1 Sent Done Done false
31675 8710 Acked Write 5 Sent Done Done false
31676 8711 Acked Flush 0 Sent Done Done false
31677 8712 Acked Read 1 Sent Done Done false
31678 8713 Acked Write 6 Sent Done Done false
31679 8714 Acked Write 8 Sent Done Done false
31680 8715 Acked Read 8 Sent Done Done false
31681 8716 Acked Read 2 Sent Done Done false
31682 8717 Acked Write 3 Sent Done Done false
31683 8718 Acked Read 3 Sent Done Done false
31684 8719 Acked Read 4 Sent Done Done false
31685 8720 Acked Flush 0 Sent Done Done false
31686 8721 Acked Flush 0 Sent Done Done false
31687 8722 Acked Read 4 Sent Done Done false
31688 8723 Acked Write 10 Sent Done Done false
31689 8724 Acked Read 10 Sent Done Done false
31690 8725 Acked Read 10 Sent Done Done false
31691 8726 Acked Write 4 Sent Done Done false
31692 8727 Acked Write 1 Sent Done Done false
31693 8728 Acked Write 8 Sent Done Done false
31694 8729 Acked Flush 0 Sent Done Done false
31695 8730 Acked Write 1 Sent Done Done false
31696 8731 Acked Read 1 Sent Done Done false
31697 8732 Acked Read 2 Sent Done Done false
31698 8733 Acked Write 6 Sent Done Done false
31699 8734 Acked Write 7 Sent Done Done false
31700 8735 Acked Write 1 Sent Done Done false
31701 8736 Acked Write 6 Sent Done Done false
31702 8737 Acked Flush 0 Sent Done Done false
31703 8738 Acked Write 10 Sent Done Done false
31704 8739 Acked Write 3 Sent Done Done false
31705 8740 Acked Write 8 Sent Done Done false
31706 8741 Acked Read 5 Sent Done Done false
31707 8742 Acked Write 3 Sent Done Done false
31708 8743 Acked Write 4 Sent Done Done false
31709 8744 Acked Read 6 Sent Done Done false
31710 8745 Acked Flush 0 Sent Done Done false
31711 8746 Acked Read 5 Sent Done Done false
31712 8747 Acked Flush 0 Sent Done Done false
31713 8748 Acked Read 1 Sent Done Done false
31714 8749 Acked Read 7 Sent Done Done false
31715 8750 Acked Write 5 Sent Done Done false
31716 8751 Acked Read 5 Sent Done Done false
31717 8752 Acked Read 5 Sent Done Done false
31718 8753 Acked Flush 0 Sent Done Done false
31719 8754 Acked Write 8 Sent Done Done false
31720 8755 Acked Read 4 Sent Done Done false
31721 8756 Acked Write 5 Sent Done Done false
31722 8757 Acked Write 8 Sent Done Done false
31723 8758 Acked Write 7 Sent Done Done false
31724 8759 Acked Read 9 Sent Done Done false
31725 8760 Acked Read 10 Sent Done Done false
31726 8761 Acked Write 1 Sent Done Done false
31727 8762 Acked Flush 0 Sent Done Done false
31728 8763 Acked Write 6 Sent Done Done false
31729 8764 Acked Read 1 Sent Done Done false
31730 8765 Acked Read 7 Sent Done Done false
31731 8766 Acked Read 2 Sent Done Done false
31732 8767 Acked Flush 0 Sent Done Done false
31733 8768 Acked Write 8 Sent Done Done false
31734 8769 Acked Write 10 Sent Done Done false
31735 8770 Acked Read 9 Sent Done Done false
31736 8771 Acked Read 5 Sent Done Done false
31737 8772 Acked Read 4 Sent Done Done false
31738 8773 Acked Read 4 Sent Done Done false
31739 8774 Acked Write 7 Sent Done Done false
31740 8775 Acked Read 8 Sent Done Done false
31741 8776 Acked Read 1 Sent Done Done false
31742 8777 Acked Read 6 Sent Done Done false
31743 8778 Acked Write 5 Sent Done Done false
31744 8779 Acked Read 7 Sent Done Done false
31745 8780 Acked Read 2 Sent Done Done false
31746 8781 Acked Write 2 Sent Done Done false
31747 8782 Acked Write 10 Sent Done Done false
31748 8783 Acked Read 7 Sent Done Done false
31749 8784 Acked Write 6 Sent Done Done false
31750 8785 Acked Read 8 Sent Done Done false
31751 8786 Acked Write 9 Sent Done Done false
31752 8787 Acked Read 3 Sent Done Done false
31753 8788 Acked Write 10 Sent Done Done false
31754 8789 Acked Read 4 Sent Done Done false
31755 8790 Acked Write 3 Sent Done Done false
31756 8791 Acked Write 7 Sent Done Done false
31757 8792 Acked Write 2 Sent Done Done false
31758 8793 Acked Read 8 Sent Done Done false
31759 8794 Acked Write 6 Sent Done Done false
31760 8795 Acked Read 1 Sent Done Done false
31761 8796 Acked Read 2 Sent Done Done false
31762 8797 Acked Read 1 Sent Done Done false
31763 8798 Acked Read 4 Sent Done Done false
31764 8799 Acked Write 5 Sent Done Done false
31765 8800 Acked Read 9 Sent Done Done false
31766 8801 Acked Read 4 Sent Done Done false
31767 8802 Acked Read 3 Sent Done Done false
31768 8803 Acked Read 5 Sent Done Done false
31769 8804 Acked Write 4 Sent Done Done false
31770 8805 Acked Read 8 Sent Done Done false
31771 8806 Acked Read 10 Sent Done Done false
31772 8807 Acked Read 7 Sent Done Done false
31773 8808 Acked Write 2 Sent Done Done false
31774 8809 Acked Read 5 Sent Done Done false
31775 8810 Acked Write 6 Sent Done Done false
31776 8811 Acked Read 9 Sent Done Done false
31777 8812 Acked Read 4 Sent Done Done false
31778 8813 Acked Write 1 Sent Done Done false
31779 8814 Acked Read 6 Sent Done Done false
31780 8815 Acked Flush 0 Sent Done Done false
31781 8816 Acked Write 1 Sent Done Done false
31782 8817 Acked Write 8 Sent Done Done false
31783 8818 Acked Write 9 Sent Done Done false
31784 8819 Acked Read 2 Sent Done Done false
31785 8820 Acked Read 3 Sent Done Done false
31786 8821 Acked Write 4 Sent Done Done false
31787 8822 Acked Read 1 Sent Done Done false
31788 8823 Acked Write 4 Sent Done Done false
31789 8824 Acked Write 9 Sent Done Done false
31790 8825 Acked Read 9 Sent Done Done false
31791 8826 Acked Flush 0 Sent Done Done false
31792 8827 Acked Read 4 Sent Done Done false
31793 8828 Acked Write 3 Sent Done Done false
31794 8829 Acked Write 5 Sent Done Done false
31795 8830 Acked Write 1 Sent Done Done false
31796 8831 Acked Write 2 Sent Done Done false
31797 8832 Acked Read 2 Sent Done Done false
31798 8833 Acked Read 1 Sent Done Done false
31799 8834 Acked Read 8 Sent Done Done false
31800 8835 Acked Read 1 Sent Done Done false
31801 8836 Acked Write 5 Sent Done Done false
31802 8837 Acked Write 3 Sent Done Done false
31803 8838 Acked Read 2 Sent Done Done false
31804 8839 Acked Read 2 Sent Done Done false
31805 8840 Acked Read 7 Sent Done Done false
31806 8841 Acked Read 6 Sent Done Done false
31807 8842 Acked Write 6 Sent Done Done false
31808 8843 Acked Write 9 Sent Done Done false
31809 8844 Acked Read 1 Sent Done Done false
31810 8845 Acked Read 1 Sent Done Done false
31811 8846 Acked Write 1 Sent Done Done false
31812 8847 Acked Write 4 Sent Done Done false
31813 8848 Acked Read 1 Sent Done Done false
31814 STATES DS:0 DS:1 DS:2 TOTAL
31815 Sent 301 0 0 301
31816 Done 0 301 301 602
31817 Skipped 0 0 0 0
31818 Error 0 0 0 0
31819 Last Flush: Some(JobId(8547)) Some(JobId(8826)) Some(JobId(8826))
31820 Downstairs last five completed: 8547 8546 8545 8544 8543
31821 Upstairs last five completed: 8848 8847 8846 8845 8844
31822 CLIENT: Up:0 ds:301 act:2
318232025-03-17T22:26:01.255ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
318242025-03-17T22:26:02.447ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
318252025-03-17T22:26:03.259ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
318262025-03-17T22:26:04.261ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
31827 ----------------------------------------------------------------
31828 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
31829 States: Offline Active Active
31830 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31831 8548 Acked Write 7 Sent Done Done false
31832 8549 Acked Flush 0 Sent Done Done false
31833 8550 Acked Read 1 Sent Done Done false
31834 8551 Acked Read 9 Sent Done Done false
31835 8552 Acked Flush 0 Sent Done Done false
31836 8553 Acked Write 3 Sent Done Done false
31837 8554 Acked Read 8 Sent Done Done false
31838 8555 Acked Write 9 Sent Done Done false
31839 8556 Acked Write 8 Sent Done Done false
31840 8557 Acked Write 8 Sent Done Done false
31841 8558 Acked Flush 0 Sent Done Done false
31842 8559 Acked Write 6 Sent Done Done false
31843 8560 Acked Flush 0 Sent Done Done false
31844 8561 Acked Write 5 Sent Done Done false
31845 8562 Acked Read 9 Sent Done Done false
31846 8563 Acked Flush 0 Sent Done Done false
31847 8564 Acked Write 4 Sent Done Done false
31848 8565 Acked Read 10 Sent Done Done false
31849 8566 Acked Flush 0 Sent Done Done false
31850 8567 Acked Flush 0 Sent Done Done false
31851 8568 Acked Read 8 Sent Done Done false
31852 8569 Acked Write 8 Sent Done Done false
31853 8570 Acked Write 7 Sent Done Done false
31854 8571 Acked Read 5 Sent Done Done false
31855 8572 Acked Read 9 Sent Done Done false
31856 8573 Acked Read 2 Sent Done Done false
31857 8574 Acked Read 6 Sent Done Done false
31858 8575 Acked Read 9 Sent Done Done false
31859 8576 Acked Read 10 Sent Done Done false
31860 8577 Acked Read 8 Sent Done Done false
31861 8578 Acked Write 8 Sent Done Done false
31862 8579 Acked Write 7 Sent Done Done false
31863 8580 Acked Write 6 Sent Done Done false
31864 8581 Acked Read 5 Sent Done Done false
31865 8582 Acked Write 3 Sent Done Done false
31866 8583 Acked Write 8 Sent Done Done false
31867 8584 Acked Write 2 Sent Done Done false
31868 8585 Acked Read 7 Sent Done Done false
31869 8586 Acked Write 2 Sent Done Done false
31870 8587 Acked Flush 0 Sent Done Done false
31871 8588 Acked Write 7 Sent Done Done false
31872 8589 Acked Read 3 Sent Done Done false
31873 8590 Acked Flush 0 Sent Done Done false
31874 8591 Acked Read 10 Sent Done Done false
31875 8592 Acked Write 6 Sent Done Done false
31876 8593 Acked Write 6 Sent Done Done false
31877 8594 Acked Read 7 Sent Done Done false
31878 8595 Acked Read 2 Sent Done Done false
31879 8596 Acked Write 6 Sent Done Done false
31880 8597 Acked Write 6 Sent Done Done false
31881 8598 Acked Read 9 Sent Done Done false
31882 8599 Acked Flush 0 Sent Done Done false
31883 8600 Acked Write 3 Sent Done Done false
31884 8601 Acked Read 1 Sent Done Done false
31885 8602 Acked Read 5 Sent Done Done false
31886 8603 Acked Read 3 Sent Done Done false
31887 8604 Acked Read 9 Sent Done Done false
31888 8605 Acked Read 10 Sent Done Done false
31889 8606 Acked Read 6 Sent Done Done false
31890 8607 Acked Read 7 Sent Done Done false
31891 8608 Acked Read 4 Sent Done Done false
31892 8609 Acked Read 5 Sent Done Done false
31893 8610 Acked Write 9 Sent Done Done false
31894 8611 Acked Write 5 Sent Done Done false
31895 8612 Acked Write 2 Sent Done Done false
31896 8613 Acked Write 10 Sent Done Done false
31897 8614 Acked Read 3 Sent Done Done false
31898 8615 Acked Write 1 Sent Done Done false
31899 8616 Acked Write 5 Sent Done Done false
31900 8617 Acked Write 3 Sent Done Done false
31901 8618 Acked Read 7 Sent Done Done false
31902 8619 Acked Write 4 Sent Done Done false
31903 8620 Acked Write 5 Sent Done Done false
31904 8621 Acked Read 5 Sent Done Done false
31905 8622 Acked Flush 0 Sent Done Done false
31906 8623 Acked Read 5 Sent Done Done false
31907 8624 Acked Read 4 Sent Done Done false
31908 8625 Acked Write 6 Sent Done Done false
31909 8626 Acked Read 10 Sent Done Done false
31910 8627 Acked Write 5 Sent Done Done false
31911 8628 Acked Write 9 Sent Done Done false
31912 8629 Acked Read 4 Sent Done Done false
31913 8630 Acked Write 9 Sent Done Done false
31914 8631 Acked Read 7 Sent Done Done false
31915 8632 Acked Read 10 Sent Done Done false
31916 8633 Acked Write 5 Sent Done Done false
31917 8634 Acked Flush 0 Sent Done Done false
31918 8635 Acked Write 3 Sent Done Done false
31919 8636 Acked Read 10 Sent Done Done false
31920 8637 Acked Write 1 Sent Done Done false
31921 8638 Acked Read 9 Sent Done Done false
31922 8639 Acked Read 9 Sent Done Done false
31923 8640 Acked Flush 0 Sent Done Done false
31924 8641 Acked Read 1 Sent Done Done false
31925 8642 Acked Write 7 Sent Done Done false
31926 8643 Acked Write 3 Sent Done Done false
31927 8644 Acked Write 3 Sent Done Done false
31928 8645 Acked Write 3 Sent Done Done false
31929 8646 Acked Flush 0 Sent Done Done false
31930 8647 Acked Write 2 Sent Done Done false
31931 8648 Acked Write 6 Sent Done Done false
31932 8649 Acked Read 7 Sent Done Done false
31933 8650 Acked Read 5 Sent Done Done false
31934 8651 Acked Write 9 Sent Done Done false
31935 8652 Acked Read 7 Sent Done Done false
31936 8653 Acked Read 5 Sent Done Done false
31937 8654 Acked Write 7 Sent Done Done false
31938 8655 Acked Write 1 Sent Done Done false
31939 8656 Acked Write 3 Sent Done Done false
31940 8657 Acked Read 3 Sent Done Done false
31941 8658 Acked Read 1 Sent Done Done false
31942 8659 Acked Read 3 Sent Done Done false
31943 8660 Acked Read 6 Sent Done Done false
31944 8661 Acked Read 5 Sent Done Done false
31945 8662 Acked Read 7 Sent Done Done false
31946 8663 Acked Write 8 Sent Done Done false
31947 8664 Acked Read 6 Sent Done Done false
31948 8665 Acked Flush 0 Sent Done Done false
31949 8666 Acked Read 5 Sent Done Done false
31950 8667 Acked Read 4 Sent Done Done false
31951 8668 Acked Flush 0 Sent Done Done false
31952 8669 Acked Write 9 Sent Done Done false
31953 8670 Acked Write 10 Sent Done Done false
31954 8671 Acked Flush 0 Sent Done Done false
31955 8672 Acked Write 8 Sent Done Done false
31956 8673 Acked Flush 0 Sent Done Done false
31957 8674 Acked Read 1 Sent Done Done false
31958 8675 Acked Read 3 Sent Done Done false
31959 8676 Acked Write 8 Sent Done Done false
31960 8677 Acked Read 7 Sent Done Done false
31961 8678 Acked Read 6 Sent Done Done false
31962 8679 Acked Read 9 Sent Done Done false
31963 8680 Acked Read 6 Sent Done Done false
31964 8681 Acked Flush 0 Sent Done Done false
31965 8682 Acked Flush 0 Sent Done Done false
31966 8683 Acked Read 7 Sent Done Done false
31967 8684 Acked Write 6 Sent Done Done false
31968 8685 Acked Write 3 Sent Done Done false
31969 8686 Acked Write 4 Sent Done Done false
31970 8687 Acked Read 6 Sent Done Done false
31971 8688 Acked Write 2 Sent Done Done false
31972 8689 Acked Read 3 Sent Done Done false
31973 8690 Acked Read 5 Sent Done Done false
31974 8691 Acked Read 8 Sent Done Done false
31975 8692 Acked Read 4 Sent Done Done false
31976 8693 Acked Write 9 Sent Done Done false
31977 8694 Acked Read 1 Sent Done Done false
31978 8695 Acked Flush 0 Sent Done Done false
31979 8696 Acked Write 6 Sent Done Done false
31980 8697 Acked Read 6 Sent Done Done false
31981 8698 Acked Write 8 Sent Done Done false
31982 8699 Acked Read 6 Sent Done Done false
31983 8700 Acked Flush 0 Sent Done Done false
31984 8701 Acked Write 7 Sent Done Done false
31985 8702 Acked Write 5 Sent Done Done false
31986 8703 Acked Read 7 Sent Done Done false
31987 8704 Acked Write 1 Sent Done Done false
31988 8705 Acked Write 9 Sent Done Done false
31989 8706 Acked Write 7 Sent Done Done false
31990 8707 Acked Write 9 Sent Done Done false
31991 8708 Acked Read 3 Sent Done Done false
31992 8709 Acked Read 1 Sent Done Done false
31993 8710 Acked Write 5 Sent Done Done false
31994 8711 Acked Flush 0 Sent Done Done false
31995 8712 Acked Read 1 Sent Done Done false
31996 8713 Acked Write 6 Sent Done Done false
31997 8714 Acked Write 8 Sent Done Done false
31998 8715 Acked Read 8 Sent Done Done false
31999 8716 Acked Read 2 Sent Done Done false
32000 8717 Acked Write 3 Sent Done Done false
32001 8718 Acked Read 3 Sent Done Done false
32002 8719 Acked Read 4 Sent Done Done false
32003 8720 Acked Flush 0 Sent Done Done false
32004 8721 Acked Flush 0 Sent Done Done false
32005 8722 Acked Read 4 Sent Done Done false
32006 8723 Acked Write 10 Sent Done Done false
32007 8724 Acked Read 10 Sent Done Done false
32008 8725 Acked Read 10 Sent Done Done false
32009 8726 Acked Write 4 Sent Done Done false
32010 8727 Acked Write 1 Sent Done Done false
32011 8728 Acked Write 8 Sent Done Done false
32012 8729 Acked Flush 0 Sent Done Done false
32013 8730 Acked Write 1 Sent Done Done false
32014 8731 Acked Read 1 Sent Done Done false
32015 8732 Acked Read 2 Sent Done Done false
32016 8733 Acked Write 6 Sent Done Done false
32017 8734 Acked Write 7 Sent Done Done false
32018 8735 Acked Write 1 Sent Done Done false
32019 8736 Acked Write 6 Sent Done Done false
32020 8737 Acked Flush 0 Sent Done Done false
32021 8738 Acked Write 10 Sent Done Done false
32022 8739 Acked Write 3 Sent Done Done false
32023 8740 Acked Write 8 Sent Done Done false
32024 8741 Acked Read 5 Sent Done Done false
32025 8742 Acked Write 3 Sent Done Done false
32026 8743 Acked Write 4 Sent Done Done false
32027 8744 Acked Read 6 Sent Done Done false
32028 8745 Acked Flush 0 Sent Done Done false
32029 8746 Acked Read 5 Sent Done Done false
32030 8747 Acked Flush 0 Sent Done Done false
32031 8748 Acked Read 1 Sent Done Done false
32032 8749 Acked Read 7 Sent Done Done false
32033 8750 Acked Write 5 Sent Done Done false
32034 8751 Acked Read 5 Sent Done Done false
32035 8752 Acked Read 5 Sent Done Done false
32036 8753 Acked Flush 0 Sent Done Done false
32037 8754 Acked Write 8 Sent Done Done false
32038 8755 Acked Read 4 Sent Done Done false
32039 8756 Acked Write 5 Sent Done Done false
32040 8757 Acked Write 8 Sent Done Done false
32041 8758 Acked Write 7 Sent Done Done false
32042 8759 Acked Read 9 Sent Done Done false
32043 8760 Acked Read 10 Sent Done Done false
32044 8761 Acked Write 1 Sent Done Done false
32045 8762 Acked Flush 0 Sent Done Done false
32046 8763 Acked Write 6 Sent Done Done false
32047 8764 Acked Read 1 Sent Done Done false
32048 8765 Acked Read 7 Sent Done Done false
32049 8766 Acked Read 2 Sent Done Done false
32050 8767 Acked Flush 0 Sent Done Done false
32051 8768 Acked Write 8 Sent Done Done false
32052 8769 Acked Write 10 Sent Done Done false
32053 8770 Acked Read 9 Sent Done Done false
32054 8771 Acked Read 5 Sent Done Done false
32055 8772 Acked Read 4 Sent Done Done false
32056 8773 Acked Read 4 Sent Done Done false
32057 8774 Acked Write 7 Sent Done Done false
32058 8775 Acked Read 8 Sent Done Done false
32059 8776 Acked Read 1 Sent Done Done false
32060 8777 Acked Read 6 Sent Done Done false
32061 8778 Acked Write 5 Sent Done Done false
32062 8779 Acked Read 7 Sent Done Done false
32063 8780 Acked Read 2 Sent Done Done false
32064 8781 Acked Write 2 Sent Done Done false
32065 8782 Acked Write 10 Sent Done Done false
32066 8783 Acked Read 7 Sent Done Done false
32067 8784 Acked Write 6 Sent Done Done false
32068 8785 Acked Read 8 Sent Done Done false
32069 8786 Acked Write 9 Sent Done Done false
32070 8787 Acked Read 3 Sent Done Done false
32071 8788 Acked Write 10 Sent Done Done false
32072 8789 Acked Read 4 Sent Done Done false
32073 8790 Acked Write 3 Sent Done Done false
32074 8791 Acked Write 7 Sent Done Done false
32075 8792 Acked Write 2 Sent Done Done false
32076 8793 Acked Read 8 Sent Done Done false
32077 8794 Acked Write 6 Sent Done Done false
32078 8795 Acked Read 1 Sent Done Done false
32079 8796 Acked Read 2 Sent Done Done false
32080 8797 Acked Read 1 Sent Done Done false
32081 8798 Acked Read 4 Sent Done Done false
32082 8799 Acked Write 5 Sent Done Done false
32083 8800 Acked Read 9 Sent Done Done false
32084 8801 Acked Read 4 Sent Done Done false
32085 8802 Acked Read 3 Sent Done Done false
32086 8803 Acked Read 5 Sent Done Done false
32087 8804 Acked Write 4 Sent Done Done false
32088 8805 Acked Read 8 Sent Done Done false
32089 8806 Acked Read 10 Sent Done Done false
32090 8807 Acked Read 7 Sent Done Done false
32091 8808 Acked Write 2 Sent Done Done false
32092 8809 Acked Read 5 Sent Done Done false
32093 8810 Acked Write 6 Sent Done Done false
32094 8811 Acked Read 9 Sent Done Done false
32095 8812 Acked Read 4 Sent Done Done false
32096 8813 Acked Write 1 Sent Done Done false
32097 8814 Acked Read 6 Sent Done Done false
32098 8815 Acked Flush 0 Sent Done Done false
32099 8816 Acked Write 1 Sent Done Done false
32100 8817 Acked Write 8 Sent Done Done false
32101 8818 Acked Write 9 Sent Done Done false
32102 8819 Acked Read 2 Sent Done Done false
32103 8820 Acked Read 3 Sent Done Done false
32104 8821 Acked Write 4 Sent Done Done false
32105 8822 Acked Read 1 Sent Done Done false
32106 8823 Acked Write 4 Sent Done Done false
32107 8824 Acked Write 9 Sent Done Done false
32108 8825 Acked Read 9 Sent Done Done false
32109 8826 Acked Flush 0 Sent Done Done false
32110 8827 Acked Read 4 Sent Done Done false
32111 8828 Acked Write 3 Sent Done Done false
32112 8829 Acked Write 5 Sent Done Done false
32113 8830 Acked Write 1 Sent Done Done false
32114 8831 Acked Write 2 Sent Done Done false
32115 8832 Acked Read 2 Sent Done Done false
32116 8833 Acked Read 1 Sent Done Done false
32117 8834 Acked Read 8 Sent Done Done false
32118 8835 Acked Read 1 Sent Done Done false
32119 8836 Acked Write 5 Sent Done Done false
32120 8837 Acked Write 3 Sent Done Done false
32121 8838 Acked Read 2 Sent Done Done false
32122 8839 Acked Read 2 Sent Done Done false
32123 8840 Acked Read 7 Sent Done Done false
32124 8841 Acked Read 6 Sent Done Done false
32125 8842 Acked Write 6 Sent Done Done false
32126 8843 Acked Write 9 Sent Done Done false
32127 8844 Acked Read 1 Sent Done Done false
32128 8845 Acked Read 1 Sent Done Done false
32129 8846 Acked Write 1 Sent Done Done false
32130 8847 Acked Write 4 Sent Done Done false
32131 8848 Acked Read 1 Sent Done Done false
32132 STATES DS:0 DS:1 DS:2 TOTAL
32133 Sent 301 0 0 301
32134 Done 0 301 301 602
32135 Skipped 0 0 0 0
32136 Error 0 0 0 0
32137 Last Flush: Some(JobId(8547)) Some(JobId(8826)) Some(JobId(8826))
32138 Downstairs last five completed: 8547 8546 8545 8544 8543
32139 Upstairs last five completed: 8848 8847 8846 8845 8844
32140 CLIENT: Up:0 ds:301 act:2
321412025-03-17T22:26:05.264ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
321422025-03-17T22:26:06.264ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
321432025-03-17T22:26:06.450ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
321442025-03-17T22:26:06.450ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
321452025-03-17T22:26:06.450ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
321462025-03-17T22:26:06.450ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
321472025-03-17T22:26:06.450ZINFOcrucible: client task is exiting = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
321482025-03-17T22:26:07.448ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
321492025-03-17T22:26:08.268ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
32150 [8810] got start action
32151 [8810] I am going to restart
32152 Starting downstairs at port 8810
32153 Make output file at "/tmp/dsc/downstairs-8810.txt"
32154 [8810][0] reports Starting
32155 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1237)
32156 [8810][0] reports Running
32157 ----------------------------------------------------------------
32158 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32159 States: Offline Active Active
32160 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32161 8548 Acked Write 7 Sent Done Done false
32162 8549 Acked Flush 0 Sent Done Done false
32163 8550 Acked Read 1 Sent Done Done false
32164 8551 Acked Read 9 Sent Done Done false
32165 8552 Acked Flush 0 Sent Done Done false
32166 8553 Acked Write 3 Sent Done Done false
32167 8554 Acked Read 8 Sent Done Done false
32168 8555 Acked Write 9 Sent Done Done false
32169 8556 Acked Write 8 Sent Done Done false
32170 8557 Acked Write 8 Sent Done Done false
32171 8558 Acked Flush 0 Sent Done Done false
32172 8559 Acked Write 6 Sent Done Done false
32173 8560 Acked Flush 0 Sent Done Done false
32174 8561 Acked Write 5 Sent Done Done false
32175 8562 Acked Read 9 Sent Done Done false
32176 8563 Acked Flush 0 Sent Done Done false
32177 8564 Acked Write 4 Sent Done Done false
32178 8565 Acked Read 10 Sent Done Done false
32179 8566 Acked Flush 0 Sent Done Done false
32180 8567 Acked Flush 0 Sent Done Done false
32181 8568 Acked Read 8 Sent Done Done false
32182 8569 Acked Write 8 Sent Done Done false
32183 8570 Acked Write 7 Sent Done Done false
32184 8571 Acked Read 5 Sent Done Done false
32185 8572 Acked Read 9 Sent Done Done false
32186 8573 Acked Read 2 Sent Done Done false
32187 8574 Acked Read 6 Sent Done Done false
32188 8575 Acked Read 9 Sent Done Done false
32189 8576 Acked Read 10 Sent Done Done false
32190 8577 Acked Read 8 Sent Done Done false
32191 8578 Acked Write 8 Sent Done Done false
32192 8579 Acked Write 7 Sent Done Done false
32193 8580 Acked Write 6 Sent Done Done false
32194 8581 Acked Read 5 Sent Done Done false
32195 8582 Acked Write 3 Sent Done Done false
32196 8583 Acked Write 8 Sent Done Done false
32197 8584 Acked Write 2 Sent Done Done false
32198 8585 Acked Read 7 Sent Done Done false
32199 8586 Acked Write 2 Sent Done Done false
32200 8587 Acked Flush 0 Sent Done Done false
32201 8588 Acked Write 7 Sent Done Done false
32202 8589 Acked Read 3 Sent Done Done false
32203 8590 Acked Flush 0 Sent Done Done false
32204 8591 Acked Read 10 Sent Done Done false
32205 8592 Acked Write 6 Sent Done Done false
32206 8593 Acked Write 6 Sent Done Done false
32207 8594 Acked Read 7 Sent Done Done false
32208 8595 Acked Read 2 Sent Done Done false
32209 8596 Acked Write 6 Sent Done Done false
32210 8597 Acked Write 6 Sent Done Done false
32211 8598 Acked Read 9 Sent Done Done false
32212 8599 Acked Flush 0 Sent Done Done false
32213 8600 Acked Write 3 Sent Done Done false
32214 8601 Acked Read 1 Sent Done Done false
32215 8602 Acked Read 5 Sent Done Done false
32216 8603 Acked Read 3 Sent Done Done false
32217 8604 Acked Read 9 Sent Done Done false
32218 8605 Acked Read 10 Sent Done Done false
32219 8606 Acked Read 6 Sent Done Done false
32220 8607 Acked Read 7 Sent Done Done false
32221 8608 Acked Read 4 Sent Done Done false
32222 8609 Acked Read 5 Sent Done Done false
32223 8610 Acked Write 9 Sent Done Done false
32224 8611 Acked Write 5 Sent Done Done false
32225 8612 Acked Write 2 Sent Done Done false
32226 8613 Acked Write 10 Sent Done Done false
32227 8614 Acked Read 3 Sent Done Done false
32228 8615 Acked Write 1 Sent Done Done false
32229 8616 Acked Write 5 Sent Done Done false
32230 8617 Acked Write 3 Sent Done Done false
32231 8618 Acked Read 7 Sent Done Done false
32232 8619 Acked Write 4 Sent Done Done false
32233 8620 Acked Write 5 Sent Done Done false
32234 8621 Acked Read 5 Sent Done Done false
32235 8622 Acked Flush 0 Sent Done Done false
32236 8623 Acked Read 5 Sent Done Done false
32237 8624 Acked Read 4 Sent Done Done false
32238 8625 Acked Write 6 Sent Done Done false
32239 8626 Acked Read 10 Sent Done Done false
32240 8627 Acked Write 5 Sent Done Done false
32241 8628 Acked Write 9 Sent Done Done false
32242 8629 Acked Read 4 Sent Done Done false
32243 8630 Acked Write 9 Sent Done Done false
32244 8631 Acked Read 7 Sent Done Done false
32245 8632 Acked Read 10 Sent Done Done false
32246 8633 Acked Write 5 Sent Done Done false
32247 8634 Acked Flush 0 Sent Done Done false
32248 8635 Acked Write 3 Sent Done Done false
32249 8636 Acked Read 10 Sent Done Done false
32250 8637 Acked Write 1 Sent Done Done false
32251 8638 Acked Read 9 Sent Done Done false
32252 8639 Acked Read 9 Sent Done Done false
32253 8640 Acked Flush 0 Sent Done Done false
32254 8641 Acked Read 1 Sent Done Done false
32255 8642 Acked Write 7 Sent Done Done false
32256 8643 Acked Write 3 Sent Done Done false
32257 8644 Acked Write 3 Sent Done Done false
32258 8645 Acked Write 3 Sent Done Done false
32259 8646 Acked Flush 0 Sent Done Done false
32260 8647 Acked Write 2 Sent Done Done false
32261 8648 Acked Write 6 Sent Done Done false
32262 8649 Acked Read 7 Sent Done Done false
32263 8650 Acked Read 5 Sent Done Done false
32264 8651 Acked Write 9 Sent Done Done false
32265 8652 Acked Read 7 Sent Done Done false
32266 8653 Acked Read 5 Sent Done Done false
32267 8654 Acked Write 7 Sent Done Done false
32268 8655 Acked Write 1 Sent Done Done false
32269 8656 Acked Write 3 Sent Done Done false
32270 8657 Acked Read 3 Sent Done Done false
32271 8658 Acked Read 1 Sent Done Done false
32272 8659 Acked Read 3 Sent Done Done false
32273 8660 Acked Read 6 Sent Done Done false
32274 8661 Acked Read 5 Sent Done Done false
32275 8662 Acked Read 7 Sent Done Done false
32276 8663 Acked Write 8 Sent Done Done false
32277 8664 Acked Read 6 Sent Done Done false
32278 8665 Acked Flush 0 Sent Done Done false
32279 8666 Acked Read 5 Sent Done Done false
32280 8667 Acked Read 4 Sent Done Done false
32281 8668 Acked Flush 0 Sent Done Done false
32282 8669 Acked Write 9 Sent Done Done false
32283 8670 Acked Write 10 Sent Done Done false
32284 8671 Acked Flush 0 Sent Done Done false
32285 8672 Acked Write 8 Sent Done Done false
32286 8673 Acked Flush 0 Sent Done Done false
32287 8674 Acked Read 1 Sent Done Done false
32288 8675 Acked Read 3 Sent Done Done false
32289 8676 Acked Write 8 Sent Done Done false
32290 8677 Acked Read 7 Sent Done Done false
32291 8678 Acked Read 6 Sent Done Done false
32292 8679 Acked Read 9 Sent Done Done false
32293 8680 Acked Read 6 Sent Done Done false
32294 8681 Acked Flush 0 Sent Done Done false
32295 8682 Acked Flush 0 Sent Done Done false
32296 8683 Acked Read 7 Sent Done Done false
32297 8684 Acked Write 6 Sent Done Done false
32298 8685 Acked Write 3 Sent Done Done false
32299 8686 Acked Write 4 Sent Done Done false
32300 8687 Acked Read 6 Sent Done Done false
32301 8688 Acked Write 2 Sent Done Done false
32302 8689 Acked Read 3 Sent Done Done false
32303 8690 Acked Read 5 Sent Done Done false
32304 8691 Acked Read 8 Sent Done Done false
32305 8692 Acked Read 4 Sent Done Done false
32306 8693 Acked Write 9 Sent Done Done false
32307 8694 Acked Read 1 Sent Done Done false
32308 8695 Acked Flush 0 Sent Done Done false
32309 8696 Acked Write 6 Sent Done Done false
32310 8697 Acked Read 6 Sent Done Done false
32311 8698 Acked Write 8 Sent Done Done false
32312 8699 Acked Read 6 Sent Done Done false
32313 8700 Acked Flush 0 Sent Done Done false
32314 8701 Acked Write 7 Sent Done Done false
32315 8702 Acked Write 5 Sent Done Done false
32316 8703 Acked Read 7 Sent Done Done false
32317 8704 Acked Write 1 Sent Done Done false
32318 8705 Acked Write 9 Sent Done Done false
32319 8706 Acked Write 7 Sent Done Done false
32320 8707 Acked Write 9 Sent Done Done false
32321 8708 Acked Read 3 Sent Done Done false
32322 8709 Acked Read 1 Sent Done Done false
32323 8710 Acked Write 5 Sent Done Done false
32324 8711 Acked Flush 0 Sent Done Done false
32325 8712 Acked Read 1 Sent Done Done false
32326 8713 Acked Write 6 Sent Done Done false
32327 8714 Acked Write 8 Sent Done Done false
32328 8715 Acked Read 8 Sent Done Done false
32329 8716 Acked Read 2 Sent Done Done false
32330 8717 Acked Write 3 Sent Done Done false
32331 8718 Acked Read 3 Sent Done Done false
32332 8719 Acked Read 4 Sent Done Done false
32333 8720 Acked Flush 0 Sent Done Done false
32334 8721 Acked Flush 0 Sent Done Done false
32335 8722 Acked Read 4 Sent Done Done false
32336 8723 Acked Write 10 Sent Done Done false
32337 8724 Acked Read 10 Sent Done Done false
32338 8725 Acked Read 10 Sent Done Done false
32339 8726 Acked Write 4 Sent Done Done false
32340 8727 Acked Write 1 Sent Done Done false
32341 8728 Acked Write 8 Sent Done Done false
32342 8729 Acked Flush 0 Sent Done Done false
32343 8730 Acked Write 1 Sent Done Done false
32344 8731 Acked Read 1 Sent Done Done false
32345 8732 Acked Read 2 Sent Done Done false
32346 8733 Acked Write 6 Sent Done Done false
32347 8734 Acked Write 7 Sent Done Done false
32348 8735 Acked Write 1 Sent Done Done false
32349 8736 Acked Write 6 Sent Done Done false
32350 8737 Acked Flush 0 Sent Done Done false
32351 8738 Acked Write 10 Sent Done Done false
32352 8739 Acked Write 3 Sent Done Done false
32353 8740 Acked Write 8 Sent Done Done false
32354 8741 Acked Read 5 Sent Done Done false
32355 8742 Acked Write 3 Sent Done Done false
32356 8743 Acked Write 4 Sent Done Done false
32357 8744 Acked Read 6 Sent Done Done false
32358 8745 Acked Flush 0 Sent Done Done false
32359 8746 Acked Read 5 Sent Done Done false
32360 8747 Acked Flush 0 Sent Done Done false
32361 8748 Acked Read 1 Sent Done Done false
32362 8749 Acked Read 7 Sent Done Done false
32363 8750 Acked Write 5 Sent Done Done false
32364 8751 Acked Read 5 Sent Done Done false
32365 8752 Acked Read 5 Sent Done Done false
32366 8753 Acked Flush 0 Sent Done Done false
32367 8754 Acked Write 8 Sent Done Done false
32368 8755 Acked Read 4 Sent Done Done false
32369 8756 Acked Write 5 Sent Done Done false
32370 8757 Acked Write 8 Sent Done Done false
32371 8758 Acked Write 7 Sent Done Done false
32372 8759 Acked Read 9 Sent Done Done false
32373 8760 Acked Read 10 Sent Done Done false
32374 8761 Acked Write 1 Sent Done Done false
32375 8762 Acked Flush 0 Sent Done Done false
32376 8763 Acked Write 6 Sent Done Done false
32377 8764 Acked Read 1 Sent Done Done false
32378 8765 Acked Read 7 Sent Done Done false
32379 8766 Acked Read 2 Sent Done Done false
32380 8767 Acked Flush 0 Sent Done Done false
32381 8768 Acked Write 8 Sent Done Done false
32382 8769 Acked Write 10 Sent Done Done false
32383 8770 Acked Read 9 Sent Done Done false
32384 8771 Acked Read 5 Sent Done Done false
32385 8772 Acked Read 4 Sent Done Done false
32386 8773 Acked Read 4 Sent Done Done false
32387 8774 Acked Write 7 Sent Done Done false
32388 8775 Acked Read 8 Sent Done Done false
32389 8776 Acked Read 1 Sent Done Done false
32390 8777 Acked Read 6 Sent Done Done false
32391 8778 Acked Write 5 Sent Done Done false
32392 8779 Acked Read 7 Sent Done Done false
32393 8780 Acked Read 2 Sent Done Done false
32394 8781 Acked Write 2 Sent Done Done false
32395 8782 Acked Write 10 Sent Done Done false
32396 8783 Acked Read 7 Sent Done Done false
32397 8784 Acked Write 6 Sent Done Done false
32398 8785 Acked Read 8 Sent Done Done false
32399 8786 Acked Write 9 Sent Done Done false
32400 8787 Acked Read 3 Sent Done Done false
32401 8788 Acked Write 10 Sent Done Done false
32402 8789 Acked Read 4 Sent Done Done false
32403 8790 Acked Write 3 Sent Done Done false
32404 8791 Acked Write 7 Sent Done Done false
32405 8792 Acked Write 2 Sent Done Done false
32406 8793 Acked Read 8 Sent Done Done false
32407 8794 Acked Write 6 Sent Done Done false
32408 8795 Acked Read 1 Sent Done Done false
32409 8796 Acked Read 2 Sent Done Done false
32410 8797 Acked Read 1 Sent Done Done false
32411 8798 Acked Read 4 Sent Done Done false
32412 8799 Acked Write 5 Sent Done Done false
32413 8800 Acked Read 9 Sent Done Done false
32414 8801 Acked Read 4 Sent Done Done false
32415 8802 Acked Read 3 Sent Done Done false
32416 8803 Acked Read 5 Sent Done Done false
32417 8804 Acked Write 4 Sent Done Done false
32418 8805 Acked Read 8 Sent Done Done false
32419 8806 Acked Read 10 Sent Done Done false
32420 8807 Acked Read 7 Sent Done Done false
32421 8808 Acked Write 2 Sent Done Done false
32422 8809 Acked Read 5 Sent Done Done false
32423 8810 Acked Write 6 Sent Done Done false
32424 8811 Acked Read 9 Sent Done Done false
32425 8812 Acked Read 4 Sent Done Done false
32426 8813 Acked Write 1 Sent Done Done false
32427 8814 Acked Read 6 Sent Done Done false
32428 8815 Acked Flush 0 Sent Done Done false
32429 8816 Acked Write 1 Sent Done Done false
32430 8817 Acked Write 8 Sent Done Done false
32431 8818 Acked Write 9 Sent Done Done false
32432 8819 Acked Read 2 Sent Done Done false
32433 8820 Acked Read 3 Sent Done Done false
32434 8821 Acked Write 4 Sent Done Done false
32435 8822 Acked Read 1 Sent Done Done false
32436 8823 Acked Write 4 Sent Done Done false
32437 8824 Acked Write 9 Sent Done Done false
32438 8825 Acked Read 9 Sent Done Done false
32439 8826 Acked Flush 0 Sent Done Done false
32440 8827 Acked Read 4 Sent Done Done false
32441 8828 Acked Write 3 Sent Done Done false
32442 8829 Acked Write 5 Sent Done Done false
32443 8830 Acked Write 1 Sent Done Done false
32444 8831 Acked Write 2 Sent Done Done false
32445 8832 Acked Read 2 Sent Done Done false
32446 8833 Acked Read 1 Sent Done Done false
32447 8834 Acked Read 8 Sent Done Done false
32448 8835 Acked Read 1 Sent Done Done false
32449 8836 Acked Write 5 Sent Done Done false
32450 8837 Acked Write 3 Sent Done Done false
32451 8838 Acked Read 2 Sent Done Done false
32452 8839 Acked Read 2 Sent Done Done false
32453 8840 Acked Read 7 Sent Done Done false
32454 8841 Acked Read 6 Sent Done Done false
32455 8842 Acked Write 6 Sent Done Done false
32456 8843 Acked Write 9 Sent Done Done false
32457 8844 Acked Read 1 Sent Done Done false
32458 8845 Acked Read 1 Sent Done Done false
32459 8846 Acked Write 1 Sent Done Done false
32460 8847 Acked Write 4 Sent Done Done false
32461 8848 Acked Read 1 Sent Done Done false
32462 STATES DS:0 DS:1 DS:2 TOTAL
32463 Sent 301 0 0 301
32464 Done 0 301 301 602
32465 Skipped 0 0 0 0
32466 Error 0 0 0 0
32467 Last Flush: Some(JobId(8547)) Some(JobId(8826)) Some(JobId(8826))
32468 Downstairs last five completed: 8547 8546 8545 8544 8543
32469 Upstairs last five completed: 8848 8847 8846 8845 8844
32470 CLIENT: Up:0 ds:301 act:2
324712025-03-17T22:26:09.270ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
324722025-03-17T22:26:10.270ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
324732025-03-17T22:26:11.272ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
324742025-03-17T22:26:12.449ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
32475 ----------------------------------------------------------------
32476 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32477 States: Offline Active Active
32478 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32479 8548 Acked Write 7 Sent Done Done false
32480 8549 Acked Flush 0 Sent Done Done false
32481 8550 Acked Read 1 Sent Done Done false
32482 8551 Acked Read 9 Sent Done Done false
32483 8552 Acked Flush 0 Sent Done Done false
32484 8553 Acked Write 3 Sent Done Done false
32485 8554 Acked Read 8 Sent Done Done false
32486 8555 Acked Write 9 Sent Done Done false
32487 8556 Acked Write 8 Sent Done Done false
32488 8557 Acked Write 8 Sent Done Done false
32489 8558 Acked Flush 0 Sent Done Done false
32490 8559 Acked Write 6 Sent Done Done false
32491 8560 Acked Flush 0 Sent Done Done false
32492 8561 Acked Write 5 Sent Done Done false
32493 8562 Acked Read 9 Sent Done Done false
32494 8563 Acked Flush 0 Sent Done Done false
32495 8564 Acked Write 4 Sent Done Done false
32496 8565 Acked Read 10 Sent Done Done false
32497 8566 Acked Flush 0 Sent Done Done false
32498 8567 Acked Flush 0 Sent Done Done false
32499 8568 Acked Read 8 Sent Done Done false
32500 8569 Acked Write 8 Sent Done Done false
32501 8570 Acked Write 7 Sent Done Done false
32502 8571 Acked Read 5 Sent Done Done false
32503 8572 Acked Read 9 Sent Done Done false
32504 8573 Acked Read 2 Sent Done Done false
32505 8574 Acked Read 6 Sent Done Done false
32506 8575 Acked Read 9 Sent Done Done false
32507 8576 Acked Read 10 Sent Done Done false
32508 8577 Acked Read 8 Sent Done Done false
32509 8578 Acked Write 8 Sent Done Done false
32510 8579 Acked Write 7 Sent Done Done false
32511 8580 Acked Write 6 Sent Done Done false
32512 8581 Acked Read 5 Sent Done Done false
32513 8582 Acked Write 3 Sent Done Done false
32514 8583 Acked Write 8 Sent Done Done false
32515 8584 Acked Write 2 Sent Done Done false
32516 8585 Acked Read 7 Sent Done Done false
32517 8586 Acked Write 2 Sent Done Done false
32518 8587 Acked Flush 0 Sent Done Done false
32519 8588 Acked Write 7 Sent Done Done false
32520 8589 Acked Read 3 Sent Done Done false
32521 8590 Acked Flush 0 Sent Done Done false
32522 8591 Acked Read 10 Sent Done Done false
32523 8592 Acked Write 6 Sent Done Done false
32524 8593 Acked Write 6 Sent Done Done false
32525 8594 Acked Read 7 Sent Done Done false
32526 8595 Acked Read 2 Sent Done Done false
32527 8596 Acked Write 6 Sent Done Done false
32528 8597 Acked Write 6 Sent Done Done false
32529 8598 Acked Read 9 Sent Done Done false
32530 8599 Acked Flush 0 Sent Done Done false
32531 8600 Acked Write 3 Sent Done Done false
32532 8601 Acked Read 1 Sent Done Done false
32533 8602 Acked Read 5 Sent Done Done false
32534 8603 Acked Read 3 Sent Done Done false
32535 8604 Acked Read 9 Sent Done Done false
32536 8605 Acked Read 10 Sent Done Done false
32537 8606 Acked Read 6 Sent Done Done false
32538 8607 Acked Read 7 Sent Done Done false
32539 8608 Acked Read 4 Sent Done Done false
32540 8609 Acked Read 5 Sent Done Done false
32541 8610 Acked Write 9 Sent Done Done false
32542 8611 Acked Write 5 Sent Done Done false
32543 8612 Acked Write 2 Sent Done Done false
32544 8613 Acked Write 10 Sent Done Done false
32545 8614 Acked Read 3 Sent Done Done false
32546 8615 Acked Write 1 Sent Done Done false
32547 8616 Acked Write 5 Sent Done Done false
32548 8617 Acked Write 3 Sent Done Done false
32549 8618 Acked Read 7 Sent Done Done false
32550 8619 Acked Write 4 Sent Done Done false
32551 8620 Acked Write 5 Sent Done Done false
32552 8621 Acked Read 5 Sent Done Done false
32553 8622 Acked Flush 0 Sent Done Done false
32554 8623 Acked Read 5 Sent Done Done false
32555 8624 Acked Read 4 Sent Done Done false
32556 8625 Acked Write 6 Sent Done Done false
32557 8626 Acked Read 10 Sent Done Done false
32558 8627 Acked Write 5 Sent Done Done false
32559 8628 Acked Write 9 Sent Done Done false
32560 8629 Acked Read 4 Sent Done Done false
32561 8630 Acked Write 9 Sent Done Done false
32562 8631 Acked Read 7 Sent Done Done false
32563 8632 Acked Read 10 Sent Done Done false
32564 8633 Acked Write 5 Sent Done Done false
32565 8634 Acked Flush 0 Sent Done Done false
32566 8635 Acked Write 3 Sent Done Done false
32567 8636 Acked Read 10 Sent Done Done false
32568 8637 Acked Write 1 Sent Done Done false
32569 8638 Acked Read 9 Sent Done Done false
32570 8639 Acked Read 9 Sent Done Done false
32571 8640 Acked Flush 0 Sent Done Done false
32572 8641 Acked Read 1 Sent Done Done false
32573 8642 Acked Write 7 Sent Done Done false
32574 8643 Acked Write 3 Sent Done Done false
32575 8644 Acked Write 3 Sent Done Done false
32576 8645 Acked Write 3 Sent Done Done false
32577 8646 Acked Flush 0 Sent Done Done false
32578 8647 Acked Write 2 Sent Done Done false
32579 8648 Acked Write 6 Sent Done Done false
32580 8649 Acked Read 7 Sent Done Done false
32581 8650 Acked Read 5 Sent Done Done false
32582 8651 Acked Write 9 Sent Done Done false
32583 8652 Acked Read 7 Sent Done Done false
32584 8653 Acked Read 5 Sent Done Done false
32585 8654 Acked Write 7 Sent Done Done false
32586 8655 Acked Write 1 Sent Done Done false
32587 8656 Acked Write 3 Sent Done Done false
32588 8657 Acked Read 3 Sent Done Done false
32589 8658 Acked Read 1 Sent Done Done false
32590 8659 Acked Read 3 Sent Done Done false
32591 8660 Acked Read 6 Sent Done Done false
32592 8661 Acked Read 5 Sent Done Done false
32593 8662 Acked Read 7 Sent Done Done false
32594 8663 Acked Write 8 Sent Done Done false
32595 8664 Acked Read 6 Sent Done Done false
32596 8665 Acked Flush 0 Sent Done Done false
32597 8666 Acked Read 5 Sent Done Done false
32598 8667 Acked Read 4 Sent Done Done false
32599 8668 Acked Flush 0 Sent Done Done false
32600 8669 Acked Write 9 Sent Done Done false
32601 8670 Acked Write 10 Sent Done Done false
32602 8671 Acked Flush 0 Sent Done Done false
32603 8672 Acked Write 8 Sent Done Done false
32604 8673 Acked Flush 0 Sent Done Done false
32605 8674 Acked Read 1 Sent Done Done false
32606 8675 Acked Read 3 Sent Done Done false
32607 8676 Acked Write 8 Sent Done Done false
32608 8677 Acked Read 7 Sent Done Done false
32609 8678 Acked Read 6 Sent Done Done false
32610 8679 Acked Read 9 Sent Done Done false
32611 8680 Acked Read 6 Sent Done Done false
32612 8681 Acked Flush 0 Sent Done Done false
32613 8682 Acked Flush 0 Sent Done Done false
32614 8683 Acked Read 7 Sent Done Done false
32615 8684 Acked Write 6 Sent Done Done false
32616 8685 Acked Write 3 Sent Done Done false
32617 8686 Acked Write 4 Sent Done Done false
32618 8687 Acked Read 6 Sent Done Done false
32619 8688 Acked Write 2 Sent Done Done false
32620 8689 Acked Read 3 Sent Done Done false
32621 8690 Acked Read 5 Sent Done Done false
32622 8691 Acked Read 8 Sent Done Done false
32623 8692 Acked Read 4 Sent Done Done false
32624 8693 Acked Write 9 Sent Done Done false
32625 8694 Acked Read 1 Sent Done Done false
32626 8695 Acked Flush 0 Sent Done Done false
32627 8696 Acked Write 6 Sent Done Done false
32628 8697 Acked Read 6 Sent Done Done false
32629 8698 Acked Write 8 Sent Done Done false
32630 8699 Acked Read 6 Sent Done Done false
32631 8700 Acked Flush 0 Sent Done Done false
32632 8701 Acked Write 7 Sent Done Done false
32633 8702 Acked Write 5 Sent Done Done false
32634 8703 Acked Read 7 Sent Done Done false
32635 8704 Acked Write 1 Sent Done Done false
32636 8705 Acked Write 9 Sent Done Done false
32637 8706 Acked Write 7 Sent Done Done false
32638 8707 Acked Write 9 Sent Done Done false
32639 8708 Acked Read 3 Sent Done Done false
32640 8709 Acked Read 1 Sent Done Done false
32641 8710 Acked Write 5 Sent Done Done false
32642 8711 Acked Flush 0 Sent Done Done false
32643 8712 Acked Read 1 Sent Done Done false
32644 8713 Acked Write 6 Sent Done Done false
32645 8714 Acked Write 8 Sent Done Done false
32646 8715 Acked Read 8 Sent Done Done false
32647 8716 Acked Read 2 Sent Done Done false
32648 8717 Acked Write 3 Sent Done Done false
32649 8718 Acked Read 3 Sent Done Done false
32650 8719 Acked Read 4 Sent Done Done false
32651 8720 Acked Flush 0 Sent Done Done false
32652 8721 Acked Flush 0 Sent Done Done false
32653 8722 Acked Read 4 Sent Done Done false
32654 8723 Acked Write 10 Sent Done Done false
32655 8724 Acked Read 10 Sent Done Done false
32656 8725 Acked Read 10 Sent Done Done false
32657 8726 Acked Write 4 Sent Done Done false
32658 8727 Acked Write 1 Sent Done Done false
32659 8728 Acked Write 8 Sent Done Done false
32660 8729 Acked Flush 0 Sent Done Done false
32661 8730 Acked Write 1 Sent Done Done false
32662 8731 Acked Read 1 Sent Done Done false
32663 8732 Acked Read 2 Sent Done Done false
32664 8733 Acked Write 6 Sent Done Done false
32665 8734 Acked Write 7 Sent Done Done false
32666 8735 Acked Write 1 Sent Done Done false
32667 8736 Acked Write 6 Sent Done Done false
32668 8737 Acked Flush 0 Sent Done Done false
32669 8738 Acked Write 10 Sent Done Done false
32670 8739 Acked Write 3 Sent Done Done false
32671 8740 Acked Write 8 Sent Done Done false
32672 8741 Acked Read 5 Sent Done Done false
32673 8742 Acked Write 3 Sent Done Done false
32674 8743 Acked Write 4 Sent Done Done false
32675 8744 Acked Read 6 Sent Done Done false
32676 8745 Acked Flush 0 Sent Done Done false
32677 8746 Acked Read 5 Sent Done Done false
32678 8747 Acked Flush 0 Sent Done Done false
32679 8748 Acked Read 1 Sent Done Done false
32680 8749 Acked Read 7 Sent Done Done false
32681 8750 Acked Write 5 Sent Done Done false
32682 8751 Acked Read 5 Sent Done Done false
32683 8752 Acked Read 5 Sent Done Done false
32684 8753 Acked Flush 0 Sent Done Done false
32685 8754 Acked Write 8 Sent Done Done false
32686 8755 Acked Read 4 Sent Done Done false
32687 8756 Acked Write 5 Sent Done Done false
32688 8757 Acked Write 8 Sent Done Done false
32689 8758 Acked Write 7 Sent Done Done false
32690 8759 Acked Read 9 Sent Done Done false
32691 8760 Acked Read 10 Sent Done Done false
32692 8761 Acked Write 1 Sent Done Done false
32693 8762 Acked Flush 0 Sent Done Done false
32694 8763 Acked Write 6 Sent Done Done false
32695 8764 Acked Read 1 Sent Done Done false
32696 8765 Acked Read 7 Sent Done Done false
32697 8766 Acked Read 2 Sent Done Done false
32698 8767 Acked Flush 0 Sent Done Done false
32699 8768 Acked Write 8 Sent Done Done false
32700 8769 Acked Write 10 Sent Done Done false
32701 8770 Acked Read 9 Sent Done Done false
32702 8771 Acked Read 5 Sent Done Done false
32703 8772 Acked Read 4 Sent Done Done false
32704 8773 Acked Read 4 Sent Done Done false
32705 8774 Acked Write 7 Sent Done Done false
32706 8775 Acked Read 8 Sent Done Done false
32707 8776 Acked Read 1 Sent Done Done false
32708 8777 Acked Read 6 Sent Done Done false
32709 8778 Acked Write 5 Sent Done Done false
32710 8779 Acked Read 7 Sent Done Done false
32711 8780 Acked Read 2 Sent Done Done false
32712 8781 Acked Write 2 Sent Done Done false
32713 8782 Acked Write 10 Sent Done Done false
32714 8783 Acked Read 7 Sent Done Done false
32715 8784 Acked Write 6 Sent Done Done false
32716 8785 Acked Read 8 Sent Done Done false
32717 8786 Acked Write 9 Sent Done Done false
32718 8787 Acked Read 3 Sent Done Done false
32719 8788 Acked Write 10 Sent Done Done false
32720 8789 Acked Read 4 Sent Done Done false
32721 8790 Acked Write 3 Sent Done Done false
32722 8791 Acked Write 7 Sent Done Done false
32723 8792 Acked Write 2 Sent Done Done false
32724 8793 Acked Read 8 Sent Done Done false
32725 8794 Acked Write 6 Sent Done Done false
32726 8795 Acked Read 1 Sent Done Done false
32727 8796 Acked Read 2 Sent Done Done false
32728 8797 Acked Read 1 Sent Done Done false
32729 8798 Acked Read 4 Sent Done Done false
32730 8799 Acked Write 5 Sent Done Done false
32731 8800 Acked Read 9 Sent Done Done false
32732 8801 Acked Read 4 Sent Done Done false
32733 8802 Acked Read 3 Sent Done Done false
32734 8803 Acked Read 5 Sent Done Done false
32735 8804 Acked Write 4 Sent Done Done false
32736 8805 Acked Read 8 Sent Done Done false
32737 8806 Acked Read 10 Sent Done Done false
32738 8807 Acked Read 7 Sent Done Done false
32739 8808 Acked Write 2 Sent Done Done false
32740 8809 Acked Read 5 Sent Done Done false
32741 8810 Acked Write 6 Sent Done Done false
32742 8811 Acked Read 9 Sent Done Done false
32743 8812 Acked Read 4 Sent Done Done false
32744 8813 Acked Write 1 Sent Done Done false
32745 8814 Acked Read 6 Sent Done Done false
32746 8815 Acked Flush 0 Sent Done Done false
32747 8816 Acked Write 1 Sent Done Done false
32748 8817 Acked Write 8 Sent Done Done false
32749 8818 Acked Write 9 Sent Done Done false
32750 8819 Acked Read 2 Sent Done Done false
32751 8820 Acked Read 3 Sent Done Done false
32752 8821 Acked Write 4 Sent Done Done false
32753 8822 Acked Read 1 Sent Done Done false
32754 8823 Acked Write 4 Sent Done Done false
32755 8824 Acked Write 9 Sent Done Done false
32756 8825 Acked Read 9 Sent Done Done false
32757 8826 Acked Flush 0 Sent Done Done false
32758 8827 Acked Read 4 Sent Done Done false
32759 8828 Acked Write 3 Sent Done Done false
32760 8829 Acked Write 5 Sent Done Done false
32761 8830 Acked Write 1 Sent Done Done false
32762 8831 Acked Write 2 Sent Done Done false
32763 8832 Acked Read 2 Sent Done Done false
32764 8833 Acked Read 1 Sent Done Done false
32765 8834 Acked Read 8 Sent Done Done false
32766 8835 Acked Read 1 Sent Done Done false
32767 8836 Acked Write 5 Sent Done Done false
32768 8837 Acked Write 3 Sent Done Done false
32769 8838 Acked Read 2 Sent Done Done false
32770 8839 Acked Read 2 Sent Done Done false
32771 8840 Acked Read 7 Sent Done Done false
32772 8841 Acked Read 6 Sent Done Done false
32773 8842 Acked Write 6 Sent Done Done false
32774 8843 Acked Write 9 Sent Done Done false
32775 8844 Acked Read 1 Sent Done Done false
32776 8845 Acked Read 1 Sent Done Done false
32777 8846 Acked Write 1 Sent Done Done false
32778 8847 Acked Write 4 Sent Done Done false
32779 8848 Acked Read 1 Sent Done Done false
32780 STATES DS:0 DS:1 DS:2 TOTAL
32781 Sent 301 0 0 301
32782 Done 0 301 301 602
32783 Skipped 0 0 0 0
32784 Error 0 0 0 0
32785 Last Flush: Some(JobId(8547)) Some(JobId(8826)) Some(JobId(8826))
32786 Downstairs last five completed: 8547 8546 8545 8544 8543
32787 Upstairs last five completed: 8848 8847 8846 8845 8844
32788 CLIENT: Up:0 ds:301 act:2
327892025-03-17T22:26:13.276ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
327902025-03-17T22:26:14.279ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
327912025-03-17T22:26:15.280ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
327922025-03-17T22:26:16.281ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
327932025-03-17T22:26:16.451ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
327942025-03-17T22:26:16.451ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:39765) = io task client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
327952025-03-17T22:26:16.452ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
327962025-03-17T22:26:16.453ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
327972025-03-17T22:26:16.453ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
327982025-03-17T22:26:16.453ZINFOcrucible: send last flush ID to this DS: Some(JobId(8547)) = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
327992025-03-17T22:26:16.453ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
328002025-03-17T22:26:16.453ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(8547)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
328012025-03-17T22:26:16.453ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(8547)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
32802 ----------------------------------------------------------------
32803 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:22
32804 States: Active Active Active
32805 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32806 8827 Acked Read 4 Done Done Done true
32807 8828 Acked Write 3 Done Done Done true
32808 8829 Acked Write 5 Done Done Done true
32809 8830 Acked Write 1 Done Done Done true
32810 8831 Acked Write 2 Done Done Done true
32811 8832 Acked Read 2 Done Done Done true
32812 8833 Acked Read 1 Done Done Done true
32813 8834 Acked Read 8 Done Done Done true
32814 8835 Acked Read 1 Done Done Done true
32815 8836 Acked Write 5 Done Done Done true
32816 8837 Acked Write 3 Done Done Done true
32817 8838 Acked Read 2 Done Done Done true
32818 8839 Acked Read 2 Done Done Done true
32819 8840 Acked Read 7 Done Done Done true
32820 8841 Acked Read 6 Done Done Done true
32821 8842 Acked Write 6 Done Done Done true
32822 8843 Acked Write 9 Done Done Done true
32823 8844 Acked Read 1 Done Done Done true
32824 8845 Acked Read 1 Done Done Done true
32825 8846 Acked Write 1 Done Done Done true
32826 8847 Acked Write 4 Done Done Done true
32827 8848 Acked Read 1 Done Done Done true
32828 STATES DS:0 DS:1 DS:2 TOTAL
32829 Sent 0 0 0 0
32830 Done 22 22 22 66
32831 Skipped 0 0 0 0
32832 Error 0 0 0 0
32833 Last Flush: Some(JobId(8826)) Some(JobId(8826)) Some(JobId(8826))
32834 Downstairs last five completed: 8826 8825 8824 8823 8822
32835 Upstairs last five completed: 8848 8847 8846 8845 8844
32836 CLIENT: Up:0 ds:22 act:3
32837 ----------------------------------------------------------------
32838 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
32839 Downstairs last five completed: 8849 8848 8847 8846 8845
32840 Upstairs last five completed: 8849 8848 8847 8846 8845
32841 CLIENT: Up:0 ds:0 act:3
32842 Replay: All jobs finished
32843 Main task has work to do, go find it
32844 got dsc Stop(1)
32845 stop 1
32846 Mar 17 22:26:20.698 INFO request completed, latency_us: 347, response_code: 204, uri:[8820] Got stop action so:false kr:false
32847 /stop/cid/1, method: POST, req_id: f16ebbd8-39e5-4bfc-bc27-72f1467f864d, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
328482025-03-17T22:26:20.699ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
328492025-03-17T22:26:20.699ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
328502025-03-17T22:26:20.699ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
32851 Mar 17 22:26:20.699 INFO request completed, latency_us: 137, response_code: 200, uri: /state/cid/1, method: GET, req_id: 181746fa-7410-4cde-ad9d-f3adbda90734, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
328522025-03-17T22:26:20.699ZINFOcrucible: client task is exiting = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
32853 [8820] Exited with: ExitStatus(unix_wait_status(9))
32854 [8820][1] reports Exit
32855 Mar 17 22:26:24.701 INFO request completed, latency_us: 230, response_code: 200, uri: /state/cid/1, method: GET, req_id: 15671ed2-d902-41a6-a544-bd05a826b608, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
32856 001/300 Write block 4030 len 36864 data: 7 8 8 8 8 9 4 3 2
32857 002/300 Write block 1775 len 40960 data: 5 6 6 6 4 4 5 7 7 7
32858 003/300 Write block 152 len 40960 data: 6 7 10 8 7 7 7 7 6 6
32859 004/300 Read block 1563 len 8192
32860 005/300 Read block 1323 len 20480
32861 006/300 Write block 108 len 12288 data: 5 6 5
32862 007/300 Write block 504 len 8192 data: 6 7
32863 008/300 Flush
32864 009/300 Read block 1254 len 12288
32865 010/300 Write block 4721 len 32768 data: 5 6 5 4 3 4 4 5
32866 011/300 Write block 1603 len 12288 data: 7 7 7
32867 012/300 Read block 2811 len 16384
32868 013/300 Write block 2273 len 36864 data: 3 4 5 5 7 7 7 7 9
32869 014/300 Write block 1360 len 40960 data: 7 6 7 5 4 5 5 4 5 3
32870 015/300 Write block 2657 len 40960 data: 4 5 5 4 5 5 5 5 5 5
32871 016/300 Write block 2470 len 12288 data: 6 6 6
32872 017/300 Read block 1093 len 36864
32873 018/300 Write block 483 len 20480 data: 7 7 7 7 6
32874 019/300 Read block 2953 len 4096
32875 020/300 Read block 4861 len 4096
32876 021/300 Write block 4651 len 32768 data: 7 6 4 5 5 4 5 6
32877 022/300 Read block 3454 len 16384
32878 023/300 Read block 899 len 36864
32879 024/300 Write block 1220 len 24576 data: 4 4 4 3 3 4
32880 025/300 Flush
32881 026/300 Read block 2818 len 40960
32882 027/300 Write block 1989 len 20480 data: 6 6 6 6 4
32883 028/300 Read block 3332 len 36864
32884 029/300 Read block 2433 len 12288
32885 030/300 Read block 1087 len 24576
32886 031/300 Write block 1539 len 32768 data: 3 3 4 3 3 2 2 4
32887 032/300 Write block 2950 len 20480 data: 9 10 11 12 12
32888 033/300 Write block 657 len 4096 data: 6
32889 034/300 Write block 244 len 20480 data: 6 7 9 5 4
32890 035/300 Flush
32891 036/300 Write block 485 len 28672 data: 8 8 7 5 5 6 5
32892 037/300 Read block 787 len 32768
32893 038/300 Read block 1647 len 32768
32894 039/300 Write block 3720 len 28672 data: 8 8 9 10 9 6 6
32895 040/300 Read block 3165 len 12288
32896 041/300 Read block 2314 len 12288
32897 042/300 Read block 778 len 12288
32898 043/300 Read block 4558 len 40960
32899 044/300 Flush
32900 045/300 Read block 3945 len 40960
32901 046/300 Read block 4948 len 16384
32902 047/300 Write block 2596 len 36864 data: 5 8 8 7 8 7 10 10 8
32903 048/300 Read block 2938 len 16384
32904 049/300 Flush
32905 050/300 Write block 2572 len 24576 data: 5 6 8 8 8 9
32906 051/300 Read block 287 len 28672
32907 052/300 Read block 3784 len 16384
32908 053/300 Write block 2816 len 4096 data: 2
32909 054/300 Flush
32910 055/300 Write block 4922 len 24576 data: 5 6 6 7 7 7
32911 056/300 Read block 3606 len 4096
32912 057/300 Flush
32913 058/300 Write block 1857 len 12288 data: 6 6 5
32914 059/300 Read block 1576 len 12288
32915 060/300 Write block 4086 len 24576 data: 4 4 3 3 5 6
32916 061/300 Read block 873 len 4096
32917 062/300 Write block 4000 len 16384 data: 5 5 4 3
32918 063/300 Read block 2424 len 4096
32919 064/300 Write block 4081 len 32768 data: 4 4 5 5 6 5 5 4
32920 065/300 Flush
32921 066/300 Flush
32922 067/300 Read block 2947 len 24576
32923 068/300 Write block 782 len 32768 data: 9 9 8 8 9 8 8 8
32924 069/300 Read block 1147 len 40960
32925 070/300 Read block 2461 len 12288
32926 071/300 Write block 4164 len 4096 data: 8
32927 072/300 Read block 477 len 40960
32928 073/300 Read block 1781 len 32768
32929 074/300 Write block 1858 len 28672 data: 7 6 5 5 6 6 6
32930 075/300 Read block 44 len 24576
32931 076/300 Write block 4193 len 16384 data: 4 5 6 8
32932 077/300 Write block 707 len 40960 data: 5 6 6 6 7 7 5 4 4 4
32933 078/300 Write block 1230 len 20480 data: 3 4 4 7 8
32934 079/300 Read block 1315 len 40960
32935 080/300 Read block 1962 len 40960
32936 081/300 Write block 4921 len 8192 data: 4 6
32937 082/300 Read block 3006 len 32768
32938 083/300 Read block 2836 len 8192
32939 084/300 Write block 4495 len 36864 data: 6 6 7 6 5 6 6 6 7
32940 085/300 Read block 901 len 40960
32941 086/300 Read block 2014 len 4096
32942 087/300 Write block 554 len 8192 data: 4 4
32943 088/300 Read block 2584 len 16384
32944 089/300 Read block 1350 len 28672
32945 090/300 Read block 4706 len 40960
32946 091/300 Flush
32947 092/300 Flush
32948 093/300 Read block 1859 len 16384
32949 094/300 Read block 3238 len 4096
32950 095/300 Read block 2476 len 24576
32951 096/300 Write block 1863 len 8192 data: 7 7
32952 097/300 Flush
32953 098/300 Write block 3268 len 20480 data: 6 5 6 5 6
32954 099/300 Write block 444 len 8192 data: 6 6
32955 100/300 Read block 2116 len 24576
32956 101/300 Write block 1794 len 12288 data: 7 7 7
32957 102/300 Read block 1589 len 12288
32958 103/300 Write block 2509 len 24576 data: 5 5 3 3 4 4
32959 104/300 Read block 4228 len 20480
32960 105/300 Read block 3 len 32768
32961 106/300 Write block 2881 len 4096 data: 7
32962 107/300 Write block 1710 len 20480 data: 13 10 8 9 7
32963 108/300 Read block 671 len 20480
32964 109/300 Read block 4859 len 12288
32965 110/300 Read block 1584 len 20480
32966 111/300 Read block 2440 len 12288
32967 112/300 Read block 3015 len 8192
32968 113/300 Read block 2486 len 4096
32969 114/300 Write block 1684 len 4096 data: 6
32970 115/300 Flush
32971 116/300 Write block 2867 len 4096 data: 6
32972 117/300 Write block 1112 len 4096 data: 8
32973 118/300 Flush
32974 119/300 Write block 4218 len 24576 data: 8 7 8 8 8 7
32975 120/300 Read block 3277 len 4096
32976 121/300 Read block 546 len 28672
32977 122/300 Write block 3745 len 20480 data: 7 7 7 6 5
32978 123/300 Read block 1066 len 20480
32979 124/300 Write block 3376 len 28672 data: 5 4 4 3 3 5 6
32980 125/300 Read block 2018 len 12288
32981 126/300 Write block 4701 len 4096 data: 7
32982 127/300 Flush
32983 128/300 Flush
32984 129/300 Read block 2665 len 28672
32985 130/300 Flush
32986 131/300 Write block 846 len 36864 data: 5 6 5 6 6 5 7 6 6
32987 132/300 Write block 280 len 4096 data: 5
32988 133/300 Write block 955 len 28672 data: 4 6 8 8 8 9 6
32989 134/300 Flush
32990 135/300 Write block 4659 len 24576 data: 8 8 7 8 8 8
32991 136/300 Read block 2904 len 40960
32992 137/300 Flush
32993 138/300 Flush
32994 139/300 Read block 1887 len 24576
32995 140/300 Read block 2493 len 40960
32996 141/300 Flush
32997 142/300 Write block 4744 len 32768 data: 4 4 5 4 4 4 5 6
32998 143/300 Write block 1218 len 12288 data: 3 4 5
32999 144/300 Write block 2403 len 12288 data: 6 8 9
33000 145/300 Read block 1946 len 28672
33001 146/300 Read block 4378 len 24576
33002 147/300 Read block 3190 len 24576
33003 148/300 Write block 2616 len 12288 data: 4 3 4
33004 149/300 Read block 1243 len 4096
33005 150/300 Read block 3310 len 4096
33006 151/300 Flush
33007 152/300 Write block 893 len 40960 data: 5 5 6 4 4 5 5 5 4 4
33008 153/300 Write block 4493 len 28672 data: 5 7 7 7 8 7 6
33009 154/300 Flush
33010 155/300 Write block 4402 len 40960 data: 3 3 4 6 6 6 5 6 5 5
33011 156/300 Read block 9 len 40960
33012 157/300 Read block 247 len 16384
33013 158/300 Write block 1986 len 24576 data: 5 6 6 7 7 7
33014 159/300 Flush
33015 160/300 Read block 2045 len 4096
33016 161/300 Read block 4846 len 16384
33017 162/300 Write block 1899 len 8192 data: 5 4
33018 163/300 Read block 3725 len 32768
33019 164/300 Write block 3861 len 24576 data: 5 4 4 3 3 3
33020 165/300 Read block 4635 len 28672
33021 166/300 Read block 1653 len 24576
33022 167/300 Write block 1728 len 32768 data: 8 7 7 8 6 6 4 2
33023 168/300 Read block 4105 len 16384
33024 169/300 Read block 2046 len 36864
33025 170/300 Write block 4787 len 4096 data: 6
33026 171/300 Write block 3685 len 8192 data: 8 7
33027 172/300 Flush
33028 173/300 Read block 1555 len 8192
33029 174/300 Read block 1128 len 20480
33030 175/300 Read block 1212 len 24576
33031 176/300 Write block 3627 len 28672 data: 4 4 4 5 5 5 5
33032 177/300 Read block 303 len 4096
33033 178/300 Write block 1380 len 20480 data: 5 5 5 5 5
33034 179/300 Read block 4147 len 20480
33035 180/300 Write block 108 len 36864 data: 6 7 6 4 4 4 3 4 3
33036 181/300 Read block 1764 len 8192
33037 182/300 Write block 3436 len 20480 data: 7 9 8 7 8
33038 183/300 Read block 3409 len 28672
33039 184/300 Read block 2926 len 28672
33040 185/300 Read block 1483 len 16384
33041 186/300 Write block 3743 len 32768 data: 6 6 8 8 8 7 6 6
33042 187/300 Read block 953 len 24576
33043 188/300 Read block 3707 len 32768
33044 189/300 Write block 2490 len 24576 data: 6 7 7 9 9 8
33045 190/300 Flush
33046 191/300 Write block 328 len 40960 data: 3 4 3 4 3 3 3 5 5 5
33047 192/300 Write block 1135 len 12288 data: 5 7 6
33048 193/300 Write block 4809 len 32768 data: 5 5 6 5 4 7 7 7
33049 194/300 Read block 4248 len 16384
33050 195/300 Write block 3024 len 32768 data: 9 5 5 4 4 5 4 4
33051 196/300 Read block 4921 len 36864
33052 197/300 Read block 3556 len 28672
33053 198/300 Read block 849 len 8192
33054 199/300 Write block 4489 len 4096 data: 5
33055 200/300 Read block 3705 len 4096
33056 201/300 Read block 460 len 36864
33057 202/300 Write block 383 len 32768 data: 4 4 4 5 5 7 8 7
33058 203/300 Write block 2789 len 8192 data: 8 8
33059 204/300 Read block 4859 len 24576
33060 205/300 Write block 2998 len 28672 data: 4 3 4 5 6 6 7
33061 206/300 Write block 2467 len 20480 data: 3 3 5 7 7
33062 207/300 Read block 1597 len 20480
33063 208/300 Read block 83 len 40960
33064 209/300 Read block 2983 len 4096
33065 210/300 Read block 498 len 20480
33066 211/300 Read block 1665 len 20480
33067 212/300 Read block 707 len 24576
33068 213/300 Write block 4148 len 40960 data: 6 7 5 7 7 7 5 7 8 9
33069 214/300 Read block 1629 len 8192
33070 215/300 Read block 3252 len 36864
33071 216/300 Write block 1763 len 12288 data: 4 3 4
33072 217/300 Read block 3079 len 28672
33073 218/300 Write block 3619 len 8192 data: 4 4
33074 219/300 Read block 3855 len 32768
33075 220/300 Read block 3485 len 20480
33076 221/300 Write block 4290 len 12288 data: 5 6 5
33077 222/300 Read block 2458 len 40960
33078 223/300 Write block 3685 len 12288 data: 9 8 7
33079 224/300 Write block 866 len 36864 data: 3 3 4 4 5 5 4 5 6
33080 225/300 Write block 2343 len 24576 data: 6 6 8 8 5 5
33081 226/300 Read block 2028 len 40960
33082 227/300 Read block 4523 len 8192
33083 228/300 Read block 351 len 36864
33084 229/300 Write block 3441 len 16384 data: 7 5 6 7
33085 230/300 Read block 2686 len 24576
33086 231/300 Write block 717 len 4096 data: 4
33087 232/300 Read block 3817 len 4096
33088 233/300 Write block 833 len 28672 data: 3 4 3 4 5 6 6
33089 234/300 Write block 3949 len 40960 data: 6 7 7 7 6 5 6 7 6 6
33090 235/300 Write block 4931 len 24576 data: 7 9 9 6 7 7
33091 236/300 Write block 3667 len 8192 data: 3 3
33092 237/300 Read block 2652 len 4096
33093 238/300 Write block 3317 len 40960 data: 5 6 7 7 9 8 8 6 6 6
33094 239/300 Write block 508 len 16384 data: 5 6 6 7
33095 240/300 Flush
33096 241/300 Write block 1045 len 28672 data: 6 8 7 8 8 8 6
33097 242/300 Read block 3854 len 40960
33098 243/300 Read block 4088 len 40960
33099 244/300 Read block 935 len 24576
33100 245/300 Write block 2169 len 40960 data: 8 9 7 8 9 8 8 7 7 6
33101 246/300 Read block 4852 len 24576
33102 247/300 Read block 3803 len 32768
33103 248/300 Flush
33104 249/300 Write block 2046 len 28672 data: 9 7 7 7 4 5 5
33105 250/300 Read block 1269 len 16384
33106 251/300 Read block 3859 len 36864
33107 252/300 Read block 2831 len 12288
33108 253/300 Write block 1845 len 8192 data: 3 2
33109 254/300 Read block 3689 len 40960
33110 255/300 Read block 1211 len 28672
33111 256/300 Write block 1585 len 40960 data: 9 7 10 11 11 9 7 7 7 5
33112 257/300 Write block 3096 len 40960 data: 2 5 5 5 5 4 5 5 4 5
33113 258/300 Write block 3745 len 24576 data: 9 9 9 8 7 7
33114 259/300 Read block 768 len 16384
33115 260/300 Write block 4604 len 40960 data: 2 2 2 2 3 4 4 4 4 4
33116 261/300 Read block 1665 len 16384
33117 262/300 Write block 3908 len 40960 data: 2 3 3 3 3 3 3 5 5 6
33118 263/300 Read block 2283 len 40960
33119 264/300 Read block 1769 len 8192
33120 265/300 Write block 3294 len 20480 data: 3 3 4 3 4
33121 266/300 Read block 2646 len 8192
33122 267/300 Write block 2447 len 16384 data: 9 8 8 8
33123 268/300 Read block 3118 len 28672
33124 269/300 Write block 3304 len 32768 data: 4 5 5 5 5 5 5 3
33125 270/300 Read block 4741 len 8192
33126 271/300 Read block 4090 len 16384
33127 272/300 Write block 3899 len 8192 data: 5 4
33128 273/300 Read block 3829 len 16384
33129 274/300 Read block 611 len 12288
33130 275/300 Write block 80 len 20480 data: 5 6 4 6 6
33131 276/300 Write block 4694 len 12288 data: 6 5 5
33132 277/300 Write block 2172 len 36864 data: 9 10 9 9 8 8 7 6 5
33133 278/300 Write block 2764 len 12288 data: 5 6 6
33134 279/300 Read block 671 len 40960
33135 280/300 Read block 4383 len 4096
33136 281/300 Flush
33137 282/300 Read block 4280 len 8192
33138 283/300 Write block 4967 len 16384 data: 6 4 3 3
33139 284/300 Write block 2090 len 20480 data: 3 3 4 4 2
33140 285/300 Read block 2085 len 40960
33141 286/300 Write block 3554 len 12288 data: 3 4 4
33142 287/300 Read block 4847 len 24576
33143 288/300 Read block 1619 len 36864
33144 289/300 Flush
33145 290/300 Read block 4020 len 36864
33146 291/300 Read block 943 len 28672
33147 292/300 Read block 3677 len 20480
33148 293/300 Read block 878 len 40960
33149 294/300 Read block 2133 len 32768
33150 295/300 Flush
33151 296/300 Write block 32 len 12288 data: 7 5 4
33152 297/300 Read block 4900 len 4096
33153 298/300 Read block 2036 len 20480
33154 299/300 Write block 2743 len 24576 data: 7 7 6 8 6 6
33155 300/300 Write block 4916 len 36864 data: 2 3 3 3 3 5 7 7 7
33156 301/300 Write block 2799 len 20480 data: 4 4 4 6 6
33157 Main task has work to do, go find it
33158 got dsc Start(1)
33159 start 1
33160 Mar 17 22:26:24.945 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/1, method: POST, req_id: d3e48006-b87d-4855-b675-2d85b3385f16, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
33161 Replay: started 1, returned:Ok(())
33162 ----------------------------------------------------------------
33163 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
33164 States: Active Offline Active
33165 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33166 8850 Acked Write 9 Done Sent Done false
33167 8851 Acked Write 10 Done Sent Done false
33168 8852 Acked Write 10 Done Sent Done false
33169 8853 Acked Read 2 Done Sent Done false
33170 8854 Acked Read 5 Done Sent Done false
33171 8855 Acked Write 3 Done Sent Done false
33172 8856 Acked Write 2 Done Sent Done false
33173 8857 Acked Flush 0 Done Sent Done false
33174 8858 Acked Read 3 Done Sent Done false
33175 8859 Acked Write 8 Done Sent Done false
33176 8860 Acked Write 3 Done Sent Done false
33177 8861 Acked Read 4 Done Sent Done false
33178 8862 Acked Write 9 Done Sent Done false
33179 8863 Acked Write 10 Done Sent Done false
33180 8864 Acked Write 10 Done Sent Done false
33181 8865 Acked Write 3 Done Sent Done false
33182 8866 Acked Read 9 Done Sent Done false
33183 8867 Acked Write 5 Done Sent Done false
33184 8868 Acked Read 1 Done Sent Done false
33185 8869 Acked Read 1 Done Sent Done false
33186 8870 Acked Write 8 Done Sent Done false
33187 8871 Acked Read 4 Done Sent Done false
33188 8872 Acked Read 9 Done Sent Done false
33189 8873 Acked Write 6 Done Sent Done false
33190 8874 Acked Flush 0 Done Sent Done false
33191 8875 Acked Read 10 Done Sent Done false
33192 8876 Acked Write 5 Done Sent Done false
33193 8877 Acked Read 9 Done Sent Done false
33194 8878 Acked Read 3 Done Sent Done false
33195 8879 Acked Read 6 Done Sent Done false
33196 8880 Acked Write 8 Done Sent Done false
33197 8881 Acked Write 5 Done Sent Done false
33198 8882 Acked Write 1 Done Sent Done false
33199 8883 Acked Write 5 Done Sent Done false
33200 8884 Acked Flush 0 Done Sent Done false
33201 8885 Acked Write 7 Done Sent Done false
33202 8886 Acked Read 8 Done Sent Done false
33203 8887 Acked Read 8 Done Sent Done false
33204 8888 Acked Write 7 Done Sent Done false
33205 8889 Acked Read 3 Done Sent Done false
33206 8890 Acked Read 3 Done Sent Done false
33207 8891 Acked Read 3 Done Sent Done false
33208 8892 Acked Read 10 Done Sent Done false
33209 8893 Acked Flush 0 Done Sent Done false
33210 8894 Acked Read 10 Done Sent Done false
33211 8895 Acked Read 4 Done Sent Done false
33212 8896 Acked Write 9 Done Sent Done false
33213 8897 Acked Read 4 Done Sent Done false
33214 8898 Acked Flush 0 Done Sent Done false
33215 8899 Acked Write 6 Done Sent Done false
33216 8900 Acked Read 7 Done Sent Done false
33217 8901 Acked Read 4 Done Sent Done false
33218 8902 Acked Write 1 Done Sent Done false
33219 8903 Acked Flush 0 Done Sent Done false
33220 8904 Acked Write 6 Done Sent Done false
33221 8905 Acked Read 1 Done Sent Done false
33222 8906 Acked Flush 0 Done Sent Done false
33223 8907 Acked Write 3 Done Sent Done false
33224 8908 Acked Read 3 Done Sent Done false
33225 8909 Acked Write 6 Done Sent Done false
33226 8910 Acked Read 1 Done Sent Done false
33227 8911 Acked Write 4 Done Sent Done false
33228 8912 Acked Read 1 Done Sent Done false
33229 8913 Acked Write 8 Done Sent Done false
33230 8914 Acked Flush 0 Done Sent Done false
33231 8915 Acked Flush 0 Done Sent Done false
33232 8916 Acked Read 6 Done Sent Done false
33233 8917 Acked Write 8 Done Sent Done false
33234 8918 Acked Read 10 Done Sent Done false
33235 8919 Acked Read 3 Done Sent Done false
33236 8920 Acked Write 1 Done Sent Done false
33237 8921 Acked Read 10 Done Sent Done false
33238 8922 Acked Read 8 Done Sent Done false
33239 8923 Acked Write 7 Done Sent Done false
33240 8924 Acked Read 6 Done Sent Done false
33241 8925 Acked Write 4 Done Sent Done false
33242 8926 Acked Write 10 Done Sent Done false
33243 8927 Acked Write 5 Done Sent Done false
33244 8928 Acked Read 10 Done Sent Done false
33245 8929 Acked Read 10 Done Sent Done false
33246 8930 Acked Write 2 Done Sent Done false
33247 8931 Acked Read 8 Done Sent Done false
33248 8932 Acked Read 2 Done Sent Done false
33249 8933 Acked Write 9 Done Sent Done false
33250 8934 Acked Read 10 Done Sent Done false
33251 8935 Acked Read 1 Done Sent Done false
33252 8936 Acked Write 2 Done Sent Done false
33253 8937 Acked Read 4 Done Sent Done false
33254 8938 Acked Read 7 Done Sent Done false
33255 8939 Acked Read 10 Done Sent Done false
33256 8940 Acked Flush 0 Done Sent Done false
33257 8941 Acked Flush 0 Done Sent Done false
33258 8942 Acked Read 4 Done Sent Done false
33259 8943 Acked Read 1 Done Sent Done false
33260 8944 Acked Read 6 Done Sent Done false
33261 8945 Acked Write 2 Done Sent Done false
33262 8946 Acked Flush 0 Done Sent Done false
33263 8947 Acked Write 5 Done Sent Done false
33264 8948 Acked Write 2 Done Sent Done false
33265 8949 Acked Read 6 Done Sent Done false
33266 8950 Acked Write 3 Done Sent Done false
33267 8951 Acked Read 3 Done Sent Done false
33268 8952 Acked Write 6 Done Sent Done false
33269 8953 Acked Read 5 Done Sent Done false
33270 8954 Acked Read 8 Done Sent Done false
33271 8955 Acked Write 1 Done Sent Done false
33272 8956 Acked Write 5 Done Sent Done false
33273 8957 Acked Read 5 Done Sent Done false
33274 8958 Acked Read 3 Done Sent Done false
33275 8959 Acked Read 5 Done Sent Done false
33276 8960 Acked Read 3 Done Sent Done false
33277 8961 Acked Read 2 Done Sent Done false
33278 8962 Acked Read 1 Done Sent Done false
33279 8963 Acked Write 1 Done Sent Done false
33280 8964 Acked Flush 0 Done Sent Done false
33281 8965 Acked Write 1 Done Sent Done false
33282 8966 Acked Write 1 Done Sent Done false
33283 8967 Acked Flush 0 Done Sent Done false
33284 8968 Acked Write 6 Done Sent Done false
33285 8969 Acked Read 1 Done Sent Done false
33286 8970 Acked Read 7 Done Sent Done false
33287 8971 Acked Write 5 Done Sent Done false
33288 8972 Acked Read 5 Done Sent Done false
33289 8973 Acked Write 7 Done Sent Done false
33290 8974 Acked Read 3 Done Sent Done false
33291 8975 Acked Write 1 Done Sent Done false
33292 8976 Acked Flush 0 Done Sent Done false
33293 8977 Acked Flush 0 Done Sent Done false
33294 8978 Acked Read 7 Done Sent Done false
33295 8979 Acked Flush 0 Done Sent Done false
33296 8980 Acked Write 9 Done Sent Done false
33297 8981 Acked Write 1 Done Sent Done false
33298 8982 Acked Write 7 Done Sent Done false
33299 8983 Acked Flush 0 Done Sent Done false
33300 8984 Acked Write 6 Done Sent Done false
33301 8985 Acked Read 10 Done Sent Done false
33302 8986 Acked Flush 0 Done Sent Done false
33303 8987 Acked Flush 0 Done Sent Done false
33304 8988 Acked Read 6 Done Sent Done false
33305 8989 Acked Read 10 Done Sent Done false
33306 8990 Acked Flush 0 Done Sent Done false
33307 8991 Acked Write 8 Done Sent Done false
33308 8992 Acked Write 3 Done Sent Done false
33309 8993 Acked Write 3 Done Sent Done false
33310 8994 Acked Read 7 Done Sent Done false
33311 8995 Acked Read 6 Done Sent Done false
33312 8996 Acked Read 6 Done Sent Done false
33313 8997 Acked Write 3 Done Sent Done false
33314 8998 Acked Read 1 Done Sent Done false
33315 8999 Acked Read 1 Done Sent Done false
33316 9000 Acked Flush 0 Done Sent Done false
33317 9001 Acked Write 10 Done Sent Done false
33318 9002 Acked Write 7 Done Sent Done false
33319 9003 Acked Flush 0 Done Sent Done false
33320 9004 Acked Write 10 Done Sent Done false
33321 9005 Acked Read 10 Done Sent Done false
33322 9006 Acked Read 4 Done Sent Done false
33323 9007 Acked Write 6 Done Sent Done false
33324 9008 Acked Flush 0 Done Sent Done false
33325 9009 Acked Read 1 Done Sent Done false
33326 9010 Acked Read 4 Done Sent Done false
33327 9011 Acked Write 2 Done Sent Done false
33328 9012 Acked Read 8 Done Sent Done false
33329 9013 Acked Write 6 Done Sent Done false
33330 9014 Acked Read 7 Done Sent Done false
33331 9015 Acked Read 6 Done Sent Done false
33332 9016 Acked Write 8 Done Sent Done false
33333 9017 Acked Read 4 Done Sent Done false
33334 9018 Acked Read 9 Done Sent Done false
33335 9019 Acked Write 1 Done Sent Done false
33336 9020 Acked Write 2 Done Sent Done false
33337 9021 Acked Flush 0 Done Sent Done false
33338 9022 Acked Read 2 Done Sent Done false
33339 9023 Acked Read 5 Done Sent Done false
33340 9024 Acked Read 6 Done Sent Done false
33341 9025 Acked Write 7 Done Sent Done false
33342 9026 Acked Read 1 Done Sent Done false
33343 9027 Acked Write 5 Done Sent Done false
33344 9028 Acked Read 5 Done Sent Done false
33345 9029 Acked Write 9 Done Sent Done false
33346 9030 Acked Read 2 Done Sent Done false
33347 9031 Acked Write 5 Done Sent Done false
33348 9032 Acked Read 7 Done Sent Done false
33349 9033 Acked Read 7 Done Sent Done false
33350 9034 Acked Read 4 Done Sent Done false
33351 9035 Acked Write 8 Done Sent Done false
33352 9036 Acked Read 6 Done Sent Done false
33353 9037 Acked Read 8 Done Sent Done false
33354 9038 Acked Write 6 Done Sent Done false
33355 9039 Acked Flush 0 Done Sent Done false
33356 9040 Acked Write 10 Done Sent Done false
33357 9041 Acked Write 3 Done Sent Done false
33358 9042 Acked Write 8 Done Sent Done false
33359 9043 Acked Read 4 Done Sent Done false
33360 9044 Acked Write 8 Done Sent Done false
33361 9045 Acked Read 9 Done Sent Done false
33362 9046 Acked Read 7 Done Sent Done false
33363 9047 Acked Read 2 Done Sent Done false
33364 9048 Acked Write 1 Done Sent Done false
33365 9049 Acked Read 1 Done Sent Done false
33366 9050 Acked Read 9 Done Sent Done false
33367 9051 Acked Write 8 Done Sent Done false
33368 9052 Acked Write 2 Done Sent Done false
33369 9053 Acked Read 6 Done Sent Done false
33370 9054 Acked Write 7 Done Sent Done false
33371 9055 Acked Write 5 Done Sent Done false
33372 9056 Acked Read 5 Done Sent Done false
33373 9057 Acked Read 10 Done Sent Done false
33374 9058 Acked Read 1 Done Sent Done false
33375 9059 Acked Read 5 Done Sent Done false
33376 9060 Acked Read 5 Done Sent Done false
33377 9061 Acked Read 6 Done Sent Done false
33378 9062 Acked Write 10 Done Sent Done false
33379 9063 Acked Read 2 Done Sent Done false
33380 9064 Acked Read 9 Done Sent Done false
33381 9065 Acked Write 3 Done Sent Done false
33382 9066 Acked Read 7 Done Sent Done false
33383 9067 Acked Write 2 Done Sent Done false
33384 9068 Acked Read 8 Done Sent Done false
33385 9069 Acked Read 5 Done Sent Done false
33386 9070 Acked Write 3 Done Sent Done false
33387 9071 Acked Read 10 Done Sent Done false
33388 9072 Acked Write 3 Done Sent Done false
33389 9073 Acked Write 9 Done Sent Done false
33390 9074 Acked Write 6 Done Sent Done false
33391 9075 Acked Read 10 Done Sent Done false
33392 9076 Acked Read 2 Done Sent Done false
33393 9077 Acked Read 9 Done Sent Done false
33394 9078 Acked Write 4 Done Sent Done false
33395 9079 Acked Read 6 Done Sent Done false
33396 9080 Acked Write 1 Done Sent Done false
33397 9081 Acked Read 1 Done Sent Done false
33398 9082 Acked Write 7 Done Sent Done false
33399 9083 Acked Write 10 Done Sent Done false
33400 9084 Acked Write 6 Done Sent Done false
33401 9085 Acked Write 2 Done Sent Done false
33402 9086 Acked Read 1 Done Sent Done false
33403 9087 Acked Write 10 Done Sent Done false
33404 9088 Acked Write 4 Done Sent Done false
33405 9089 Acked Flush 0 Done Sent Done false
33406 9090 Acked Write 7 Done Sent Done false
33407 9091 Acked Read 10 Done Sent Done false
33408 9092 Acked Read 10 Done Sent Done false
33409 9093 Acked Read 6 Done Sent Done false
33410 9094 Acked Write 10 Done Sent Done false
33411 9095 Acked Read 6 Done Sent Done false
33412 9096 Acked Read 8 Done Sent Done false
33413 9097 Acked Flush 0 Done Sent Done false
33414 9098 Acked Write 7 Done Sent Done false
33415 9099 Acked Read 4 Done Sent Done false
33416 9100 Acked Read 9 Done Sent Done false
33417 9101 Acked Read 3 Done Sent Done false
33418 9102 Acked Write 2 Done Sent Done false
33419 9103 Acked Read 10 Done Sent Done false
33420 9104 Acked Read 7 Done Sent Done false
33421 9105 Acked Write 10 Done Sent Done false
33422 9106 Acked Write 10 Done Sent Done false
33423 9107 Acked Write 6 Done Sent Done false
33424 9108 Acked Read 4 Done Sent Done false
33425 9109 Acked Write 10 Done Sent Done false
33426 9110 Acked Read 4 Done Sent Done false
33427 9111 Acked Write 10 Done Sent Done false
33428 9112 Acked Read 10 Done Sent Done false
33429 9113 Acked Read 2 Done Sent Done false
33430 9114 Acked Write 5 Done Sent Done false
33431 9115 Acked Read 2 Done Sent Done false
33432 9116 Acked Write 4 Done Sent Done false
33433 9117 Acked Read 7 Done Sent Done false
33434 9118 Acked Write 8 Done Sent Done false
33435 9119 Acked Read 2 Done Sent Done false
33436 9120 Acked Read 4 Done Sent Done false
33437 9121 Acked Write 2 Done Sent Done false
33438 9122 Acked Read 4 Done Sent Done false
33439 9123 Acked Read 3 Done Sent Done false
33440 9124 Acked Write 5 Done Sent Done false
33441 9125 Acked Write 3 Done Sent Done false
33442 9126 Acked Write 9 Done Sent Done false
33443 9127 Acked Write 3 Done Sent Done false
33444 9128 Acked Read 10 Done Sent Done false
33445 9129 Acked Read 1 Done Sent Done false
33446 9130 Acked Flush 0 Done Sent Done false
33447 9131 Acked Read 2 Done Sent Done false
33448 9132 Acked Write 4 Done Sent Done false
33449 9133 Acked Write 5 Done Sent Done false
33450 9134 Acked Read 10 Done Sent Done false
33451 9135 Acked Write 3 Done Sent Done false
33452 9136 Acked Read 6 Done Sent Done false
33453 9137 Acked Read 9 Done Sent Done false
33454 9138 Acked Flush 0 Done Sent Done false
33455 9139 Acked Read 9 Done Sent Done false
33456 9140 Acked Read 7 Done Sent Done false
33457 9141 Acked Read 5 Done Sent Done false
33458 9142 Acked Read 10 Done Sent Done false
33459 9143 Acked Read 8 Done Sent Done false
33460 9144 Acked Flush 0 Done Sent Done false
33461 9145 Acked Write 3 Done Sent Done false
33462 9146 Acked Read 1 Done Sent Done false
33463 9147 Acked Read 5 Done Sent Done false
33464 9148 Acked Write 6 Done Sent Done false
33465 9149 Acked Write 9 Done Sent Done false
33466 9150 Acked Write 5 Done Sent Done false
33467 STATES DS:0 DS:1 DS:2 TOTAL
33468 Sent 0 301 0 301
33469 Done 301 0 301 602
33470 Skipped 0 0 0 0
33471 Error 0 0 0 0
33472 Last Flush: Some(JobId(9144)) Some(JobId(8849)) Some(JobId(9144))
33473 Downstairs last five completed: 8849 8848 8847 8846 8845
33474 Upstairs last five completed: 9150 9149 9148 9147 9146
33475 CLIENT: Up:0 ds:301 act:2
334762025-03-17T22:26:25.447ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
334772025-03-17T22:26:26.298ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
33478 [8820] got start action
33479 [8820] I am going to restart
33480 Starting downstairs at port 8820
33481 Make output file at "/tmp/dsc/downstairs-8820.txt"
33482 [8820][1] reports Starting
33483 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1238)
33484 [8820][1] reports Running
334852025-03-17T22:26:27.451ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
334862025-03-17T22:26:28.302ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
334872025-03-17T22:26:28.803ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
33488 ----------------------------------------------------------------
33489 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
33490 States: Active Offline Active
33491 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33492 8850 Acked Write 9 Done Sent Done false
33493 8851 Acked Write 10 Done Sent Done false
33494 8852 Acked Write 10 Done Sent Done false
33495 8853 Acked Read 2 Done Sent Done false
33496 8854 Acked Read 5 Done Sent Done false
33497 8855 Acked Write 3 Done Sent Done false
33498 8856 Acked Write 2 Done Sent Done false
33499 8857 Acked Flush 0 Done Sent Done false
33500 8858 Acked Read 3 Done Sent Done false
33501 8859 Acked Write 8 Done Sent Done false
33502 8860 Acked Write 3 Done Sent Done false
33503 8861 Acked Read 4 Done Sent Done false
33504 8862 Acked Write 9 Done Sent Done false
33505 8863 Acked Write 10 Done Sent Done false
33506 8864 Acked Write 10 Done Sent Done false
33507 8865 Acked Write 3 Done Sent Done false
33508 8866 Acked Read 9 Done Sent Done false
33509 8867 Acked Write 5 Done Sent Done false
33510 8868 Acked Read 1 Done Sent Done false
33511 8869 Acked Read 1 Done Sent Done false
33512 8870 Acked Write 8 Done Sent Done false
33513 8871 Acked Read 4 Done Sent Done false
33514 8872 Acked Read 9 Done Sent Done false
33515 8873 Acked Write 6 Done Sent Done false
33516 8874 Acked Flush 0 Done Sent Done false
33517 8875 Acked Read 10 Done Sent Done false
33518 8876 Acked Write 5 Done Sent Done false
33519 8877 Acked Read 9 Done Sent Done false
33520 8878 Acked Read 3 Done Sent Done false
33521 8879 Acked Read 6 Done Sent Done false
33522 8880 Acked Write 8 Done Sent Done false
33523 8881 Acked Write 5 Done Sent Done false
33524 8882 Acked Write 1 Done Sent Done false
33525 8883 Acked Write 5 Done Sent Done false
33526 8884 Acked Flush 0 Done Sent Done false
33527 8885 Acked Write 7 Done Sent Done false
33528 8886 Acked Read 8 Done Sent Done false
33529 8887 Acked Read 8 Done Sent Done false
33530 8888 Acked Write 7 Done Sent Done false
33531 8889 Acked Read 3 Done Sent Done false
33532 8890 Acked Read 3 Done Sent Done false
33533 8891 Acked Read 3 Done Sent Done false
33534 8892 Acked Read 10 Done Sent Done false
33535 8893 Acked Flush 0 Done Sent Done false
33536 8894 Acked Read 10 Done Sent Done false
33537 8895 Acked Read 4 Done Sent Done false
33538 8896 Acked Write 9 Done Sent Done false
33539 8897 Acked Read 4 Done Sent Done false
33540 8898 Acked Flush 0 Done Sent Done false
33541 8899 Acked Write 6 Done Sent Done false
33542 8900 Acked Read 7 Done Sent Done false
33543 8901 Acked Read 4 Done Sent Done false
33544 8902 Acked Write 1 Done Sent Done false
33545 8903 Acked Flush 0 Done Sent Done false
33546 8904 Acked Write 6 Done Sent Done false
33547 8905 Acked Read 1 Done Sent Done false
33548 8906 Acked Flush 0 Done Sent Done false
33549 8907 Acked Write 3 Done Sent Done false
33550 8908 Acked Read 3 Done Sent Done false
33551 8909 Acked Write 6 Done Sent Done false
33552 8910 Acked Read 1 Done Sent Done false
33553 8911 Acked Write 4 Done Sent Done false
33554 8912 Acked Read 1 Done Sent Done false
33555 8913 Acked Write 8 Done Sent Done false
33556 8914 Acked Flush 0 Done Sent Done false
33557 8915 Acked Flush 0 Done Sent Done false
33558 8916 Acked Read 6 Done Sent Done false
33559 8917 Acked Write 8 Done Sent Done false
33560 8918 Acked Read 10 Done Sent Done false
33561 8919 Acked Read 3 Done Sent Done false
33562 8920 Acked Write 1 Done Sent Done false
33563 8921 Acked Read 10 Done Sent Done false
33564 8922 Acked Read 8 Done Sent Done false
33565 8923 Acked Write 7 Done Sent Done false
33566 8924 Acked Read 6 Done Sent Done false
33567 8925 Acked Write 4 Done Sent Done false
33568 8926 Acked Write 10 Done Sent Done false
33569 8927 Acked Write 5 Done Sent Done false
33570 8928 Acked Read 10 Done Sent Done false
33571 8929 Acked Read 10 Done Sent Done false
33572 8930 Acked Write 2 Done Sent Done false
33573 8931 Acked Read 8 Done Sent Done false
33574 8932 Acked Read 2 Done Sent Done false
33575 8933 Acked Write 9 Done Sent Done false
33576 8934 Acked Read 10 Done Sent Done false
33577 8935 Acked Read 1 Done Sent Done false
33578 8936 Acked Write 2 Done Sent Done false
33579 8937 Acked Read 4 Done Sent Done false
33580 8938 Acked Read 7 Done Sent Done false
33581 8939 Acked Read 10 Done Sent Done false
33582 8940 Acked Flush 0 Done Sent Done false
33583 8941 Acked Flush 0 Done Sent Done false
33584 8942 Acked Read 4 Done Sent Done false
33585 8943 Acked Read 1 Done Sent Done false
33586 8944 Acked Read 6 Done Sent Done false
33587 8945 Acked Write 2 Done Sent Done false
33588 8946 Acked Flush 0 Done Sent Done false
33589 8947 Acked Write 5 Done Sent Done false
33590 8948 Acked Write 2 Done Sent Done false
33591 8949 Acked Read 6 Done Sent Done false
33592 8950 Acked Write 3 Done Sent Done false
33593 8951 Acked Read 3 Done Sent Done false
33594 8952 Acked Write 6 Done Sent Done false
33595 8953 Acked Read 5 Done Sent Done false
33596 8954 Acked Read 8 Done Sent Done false
33597 8955 Acked Write 1 Done Sent Done false
33598 8956 Acked Write 5 Done Sent Done false
33599 8957 Acked Read 5 Done Sent Done false
33600 8958 Acked Read 3 Done Sent Done false
33601 8959 Acked Read 5 Done Sent Done false
33602 8960 Acked Read 3 Done Sent Done false
33603 8961 Acked Read 2 Done Sent Done false
33604 8962 Acked Read 1 Done Sent Done false
33605 8963 Acked Write 1 Done Sent Done false
33606 8964 Acked Flush 0 Done Sent Done false
33607 8965 Acked Write 1 Done Sent Done false
33608 8966 Acked Write 1 Done Sent Done false
33609 8967 Acked Flush 0 Done Sent Done false
33610 8968 Acked Write 6 Done Sent Done false
33611 8969 Acked Read 1 Done Sent Done false
33612 8970 Acked Read 7 Done Sent Done false
33613 8971 Acked Write 5 Done Sent Done false
33614 8972 Acked Read 5 Done Sent Done false
33615 8973 Acked Write 7 Done Sent Done false
33616 8974 Acked Read 3 Done Sent Done false
33617 8975 Acked Write 1 Done Sent Done false
33618 8976 Acked Flush 0 Done Sent Done false
33619 8977 Acked Flush 0 Done Sent Done false
33620 8978 Acked Read 7 Done Sent Done false
33621 8979 Acked Flush 0 Done Sent Done false
33622 8980 Acked Write 9 Done Sent Done false
33623 8981 Acked Write 1 Done Sent Done false
33624 8982 Acked Write 7 Done Sent Done false
33625 8983 Acked Flush 0 Done Sent Done false
33626 8984 Acked Write 6 Done Sent Done false
33627 8985 Acked Read 10 Done Sent Done false
33628 8986 Acked Flush 0 Done Sent Done false
33629 8987 Acked Flush 0 Done Sent Done false
33630 8988 Acked Read 6 Done Sent Done false
33631 8989 Acked Read 10 Done Sent Done false
33632 8990 Acked Flush 0 Done Sent Done false
33633 8991 Acked Write 8 Done Sent Done false
33634 8992 Acked Write 3 Done Sent Done false
33635 8993 Acked Write 3 Done Sent Done false
33636 8994 Acked Read 7 Done Sent Done false
33637 8995 Acked Read 6 Done Sent Done false
33638 8996 Acked Read 6 Done Sent Done false
33639 8997 Acked Write 3 Done Sent Done false
33640 8998 Acked Read 1 Done Sent Done false
33641 8999 Acked Read 1 Done Sent Done false
33642 9000 Acked Flush 0 Done Sent Done false
33643 9001 Acked Write 10 Done Sent Done false
33644 9002 Acked Write 7 Done Sent Done false
33645 9003 Acked Flush 0 Done Sent Done false
33646 9004 Acked Write 10 Done Sent Done false
33647 9005 Acked Read 10 Done Sent Done false
33648 9006 Acked Read 4 Done Sent Done false
33649 9007 Acked Write 6 Done Sent Done false
33650 9008 Acked Flush 0 Done Sent Done false
33651 9009 Acked Read 1 Done Sent Done false
33652 9010 Acked Read 4 Done Sent Done false
33653 9011 Acked Write 2 Done Sent Done false
33654 9012 Acked Read 8 Done Sent Done false
33655 9013 Acked Write 6 Done Sent Done false
33656 9014 Acked Read 7 Done Sent Done false
33657 9015 Acked Read 6 Done Sent Done false
33658 9016 Acked Write 8 Done Sent Done false
33659 9017 Acked Read 4 Done Sent Done false
33660 9018 Acked Read 9 Done Sent Done false
33661 9019 Acked Write 1 Done Sent Done false
33662 9020 Acked Write 2 Done Sent Done false
33663 9021 Acked Flush 0 Done Sent Done false
33664 9022 Acked Read 2 Done Sent Done false
33665 9023 Acked Read 5 Done Sent Done false
33666 9024 Acked Read 6 Done Sent Done false
33667 9025 Acked Write 7 Done Sent Done false
33668 9026 Acked Read 1 Done Sent Done false
33669 9027 Acked Write 5 Done Sent Done false
33670 9028 Acked Read 5 Done Sent Done false
33671 9029 Acked Write 9 Done Sent Done false
33672 9030 Acked Read 2 Done Sent Done false
33673 9031 Acked Write 5 Done Sent Done false
33674 9032 Acked Read 7 Done Sent Done false
33675 9033 Acked Read 7 Done Sent Done false
33676 9034 Acked Read 4 Done Sent Done false
33677 9035 Acked Write 8 Done Sent Done false
33678 9036 Acked Read 6 Done Sent Done false
33679 9037 Acked Read 8 Done Sent Done false
33680 9038 Acked Write 6 Done Sent Done false
33681 9039 Acked Flush 0 Done Sent Done false
33682 9040 Acked Write 10 Done Sent Done false
33683 9041 Acked Write 3 Done Sent Done false
33684 9042 Acked Write 8 Done Sent Done false
33685 9043 Acked Read 4 Done Sent Done false
33686 9044 Acked Write 8 Done Sent Done false
33687 9045 Acked Read 9 Done Sent Done false
33688 9046 Acked Read 7 Done Sent Done false
33689 9047 Acked Read 2 Done Sent Done false
33690 9048 Acked Write 1 Done Sent Done false
33691 9049 Acked Read 1 Done Sent Done false
33692 9050 Acked Read 9 Done Sent Done false
33693 9051 Acked Write 8 Done Sent Done false
33694 9052 Acked Write 2 Done Sent Done false
33695 9053 Acked Read 6 Done Sent Done false
33696 9054 Acked Write 7 Done Sent Done false
33697 9055 Acked Write 5 Done Sent Done false
33698 9056 Acked Read 5 Done Sent Done false
33699 9057 Acked Read 10 Done Sent Done false
33700 9058 Acked Read 1 Done Sent Done false
33701 9059 Acked Read 5 Done Sent Done false
33702 9060 Acked Read 5 Done Sent Done false
33703 9061 Acked Read 6 Done Sent Done false
33704 9062 Acked Write 10 Done Sent Done false
33705 9063 Acked Read 2 Done Sent Done false
33706 9064 Acked Read 9 Done Sent Done false
33707 9065 Acked Write 3 Done Sent Done false
33708 9066 Acked Read 7 Done Sent Done false
33709 9067 Acked Write 2 Done Sent Done false
33710 9068 Acked Read 8 Done Sent Done false
33711 9069 Acked Read 5 Done Sent Done false
33712 9070 Acked Write 3 Done Sent Done false
33713 9071 Acked Read 10 Done Sent Done false
33714 9072 Acked Write 3 Done Sent Done false
33715 9073 Acked Write 9 Done Sent Done false
33716 9074 Acked Write 6 Done Sent Done false
33717 9075 Acked Read 10 Done Sent Done false
33718 9076 Acked Read 2 Done Sent Done false
33719 9077 Acked Read 9 Done Sent Done false
33720 9078 Acked Write 4 Done Sent Done false
33721 9079 Acked Read 6 Done Sent Done false
33722 9080 Acked Write 1 Done Sent Done false
33723 9081 Acked Read 1 Done Sent Done false
33724 9082 Acked Write 7 Done Sent Done false
33725 9083 Acked Write 10 Done Sent Done false
33726 9084 Acked Write 6 Done Sent Done false
33727 9085 Acked Write 2 Done Sent Done false
33728 9086 Acked Read 1 Done Sent Done false
33729 9087 Acked Write 10 Done Sent Done false
33730 9088 Acked Write 4 Done Sent Done false
33731 9089 Acked Flush 0 Done Sent Done false
33732 9090 Acked Write 7 Done Sent Done false
33733 9091 Acked Read 10 Done Sent Done false
33734 9092 Acked Read 10 Done Sent Done false
33735 9093 Acked Read 6 Done Sent Done false
33736 9094 Acked Write 10 Done Sent Done false
33737 9095 Acked Read 6 Done Sent Done false
33738 9096 Acked Read 8 Done Sent Done false
33739 9097 Acked Flush 0 Done Sent Done false
33740 9098 Acked Write 7 Done Sent Done false
33741 9099 Acked Read 4 Done Sent Done false
33742 9100 Acked Read 9 Done Sent Done false
33743 9101 Acked Read 3 Done Sent Done false
33744 9102 Acked Write 2 Done Sent Done false
33745 9103 Acked Read 10 Done Sent Done false
33746 9104 Acked Read 7 Done Sent Done false
33747 9105 Acked Write 10 Done Sent Done false
33748 9106 Acked Write 10 Done Sent Done false
33749 9107 Acked Write 6 Done Sent Done false
33750 9108 Acked Read 4 Done Sent Done false
33751 9109 Acked Write 10 Done Sent Done false
33752 9110 Acked Read 4 Done Sent Done false
33753 9111 Acked Write 10 Done Sent Done false
33754 9112 Acked Read 10 Done Sent Done false
33755 9113 Acked Read 2 Done Sent Done false
33756 9114 Acked Write 5 Done Sent Done false
33757 9115 Acked Read 2 Done Sent Done false
33758 9116 Acked Write 4 Done Sent Done false
33759 9117 Acked Read 7 Done Sent Done false
33760 9118 Acked Write 8 Done Sent Done false
33761 9119 Acked Read 2 Done Sent Done false
33762 9120 Acked Read 4 Done Sent Done false
33763 9121 Acked Write 2 Done Sent Done false
33764 9122 Acked Read 4 Done Sent Done false
33765 9123 Acked Read 3 Done Sent Done false
33766 9124 Acked Write 5 Done Sent Done false
33767 9125 Acked Write 3 Done Sent Done false
33768 9126 Acked Write 9 Done Sent Done false
33769 9127 Acked Write 3 Done Sent Done false
33770 9128 Acked Read 10 Done Sent Done false
33771 9129 Acked Read 1 Done Sent Done false
33772 9130 Acked Flush 0 Done Sent Done false
33773 9131 Acked Read 2 Done Sent Done false
33774 9132 Acked Write 4 Done Sent Done false
33775 9133 Acked Write 5 Done Sent Done false
33776 9134 Acked Read 10 Done Sent Done false
33777 9135 Acked Write 3 Done Sent Done false
33778 9136 Acked Read 6 Done Sent Done false
33779 9137 Acked Read 9 Done Sent Done false
33780 9138 Acked Flush 0 Done Sent Done false
33781 9139 Acked Read 9 Done Sent Done false
33782 9140 Acked Read 7 Done Sent Done false
33783 9141 Acked Read 5 Done Sent Done false
33784 9142 Acked Read 10 Done Sent Done false
33785 9143 Acked Read 8 Done Sent Done false
33786 9144 Acked Flush 0 Done Sent Done false
33787 9145 Acked Write 3 Done Sent Done false
33788 9146 Acked Read 1 Done Sent Done false
33789 9147 Acked Read 5 Done Sent Done false
33790 9148 Acked Write 6 Done Sent Done false
33791 9149 Acked Write 9 Done Sent Done false
33792 9150 Acked Write 5 Done Sent Done false
33793 STATES DS:0 DS:1 DS:2 TOTAL
33794 Sent 0 301 0 301
33795 Done 301 0 301 602
33796 Skipped 0 0 0 0
33797 Error 0 0 0 0
33798 Last Flush: Some(JobId(9144)) Some(JobId(8849)) Some(JobId(9144))
33799 Downstairs last five completed: 8849 8848 8847 8846 8845
33800 Upstairs last five completed: 9150 9149 9148 9147 9146
33801 CLIENT: Up:0 ds:301 act:2
338022025-03-17T22:26:29.448ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
338032025-03-17T22:26:30.306ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
338042025-03-17T22:26:30.701ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
338052025-03-17T22:26:30.701ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:64213) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
338062025-03-17T22:26:30.702ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
338072025-03-17T22:26:30.703ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
338082025-03-17T22:26:30.703ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
338092025-03-17T22:26:30.703ZINFOcrucible: send last flush ID to this DS: Some(JobId(8849)) = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
338102025-03-17T22:26:30.703ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
338112025-03-17T22:26:30.703ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(8849)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
338122025-03-17T22:26:30.703ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(8849)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
33813 ----------------------------------------------------------------
33814 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
33815 Downstairs last five completed: 9151 9150 9149 9148 9147
33816 Upstairs last five completed: 9151 9150 9149 9148 9147
33817 CLIENT: Up:0 ds:0 act:3
33818 Replay: All jobs finished
33819 Main task has work to do, go find it
33820 got dsc Stop(1)
33821 stop 1
33822 Mar 17 22:26:32.949 INFO [8820] Got stop action so:false kr:false
33823 request completed, latency_us: 392, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 9a0af3e4-d66e-4a01-aa67-32602b8a008d, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
33824 Mar 17 22:26:32.950 INFO request completed, latency_us: 120, response_code: 200, uri: /state/cid/1, method: GET, req_id: 529093a7-5ed7-4e32-9582-83568270e16a, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
338252025-03-17T22:26:32.950ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
338262025-03-17T22:26:32.950ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
338272025-03-17T22:26:32.950ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
338282025-03-17T22:26:32.950ZINFOcrucible: client task is exiting = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
33829 [8820] Exited with: ExitStatus(unix_wait_status(9))
33830 [8820][1] reports Exit
33831 Mar 17 22:26:36.952 INFO request completed, latency_us: 212, response_code: 200, uri: /state/cid/1, method: GET, req_id: 81c871e4-8a26-4cb4-8188-d74d4be80ae8, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
33832 001/300 Read block 3589 len 40960
33833 002/300 Read block 3574 len 32768
33834 003/300 Read block 4559 len 4096
33835 004/300 Read block 3699 len 12288
33836 005/300 Write block 1792 len 12288 data: 5 7 8
33837 006/300 Write block 3291 len 36864 data: 3 3 2 4 4 5 4 5 5
33838 007/300 Read block 829 len 36864
33839 008/300 Write block 1682 len 16384 data: 6 6 7 6
33840 009/300 Read block 2411 len 28672
33841 010/300 Read block 2099 len 40960
33842 011/300 Read block 359 len 4096
33843 012/300 Write block 3997 len 32768 data: 4 4 5 6 6 5 4 3
33844 013/300 Read block 78 len 4096
33845 014/300 Write block 597 len 28672 data: 8 7 6 6 6 5 5
33846 015/300 Read block 3118 len 32768
33847 016/300 Flush
33848 017/300 Read block 2071 len 28672
33849 018/300 Write block 2325 len 36864 data: 4 3 3 3 5 5 5 3 2
33850 019/300 Read block 4606 len 28672
33851 020/300 Write block 202 len 36864 data: 7 6 4 5 5 5 5 5 4
33852 021/300 Read block 1974 len 28672
33853 022/300 Write block 4382 len 36864 data: 8 8 8 7 5 5 3 3 2
33854 023/300 Read block 1471 len 28672
33855 024/300 Read block 3970 len 36864
33856 025/300 Write block 3167 len 20480 data: 5 4 3 3 3
33857 026/300 Read block 3714 len 32768
33858 027/300 Read block 4790 len 16384
33859 028/300 Write block 1460 len 32768 data: 5 4 5 5 4 3 4 4
33860 029/300 Read block 2662 len 20480
33861 030/300 Read block 3051 len 20480
33862 031/300 Read block 1062 len 36864
33863 032/300 Read block 1593 len 36864
33864 033/300 Write block 4584 len 40960 data: 6 4 4 5 5 6 5 7 7 5
33865 034/300 Read block 1713 len 28672
33866 035/300 Read block 3321 len 32768
33867 036/300 Write block 3195 len 12288 data: 4 4 6
33868 037/300 Write block 4236 len 32768 data: 4 3 5 4 4 5 6 5
33869 038/300 Write block 652 len 12288 data: 4 3 3
33870 039/300 Write block 4833 len 24576 data: 5 5 7 7 6 6
33871 040/300 Write block 867 len 12288 data: 4 5 5
33872 041/300 Read block 600 len 20480
33873 042/300 Read block 1410 len 36864
33874 043/300 Write block 2031 len 24576 data: 2 3 4 3 5 5
33875 044/300 Read block 2304 len 40960
33876 045/300 Flush
33877 046/300 Read block 229 len 24576
33878 047/300 Write block 3489 len 16384 data: 5 6 6 5
33879 048/300 Write block 3283 len 8192 data: 5 7
33880 049/300 Read block 3445 len 20480
33881 050/300 Read block 2599 len 36864
33882 051/300 Write block 3700 len 12288 data: 7 7 7
33883 052/300 Read block 113 len 16384
33884 053/300 Read block 405 len 12288
33885 054/300 Write block 2845 len 32768 data: 7 6 5 4 4 5 5 5
33886 055/300 Write block 4584 len 24576 data: 7 5 5 6 6 7
33887 056/300 Write block 2964 len 36864 data: 6 8 7 7 7 10 10 9 6
33888 057/300 Write block 4424 len 16384 data: 4 4 6 6
33889 058/300 Flush
33890 059/300 Read block 1896 len 24576
33891 060/300 Read block 3421 len 20480
33892 061/300 Write block 301 len 36864 data: 6 6 4 4 3 4 3 4 6
33893 062/300 Write block 1213 len 32768 data: 6 6 7 4 3 4 5 6
33894 063/300 Read block 313 len 28672
33895 064/300 Read block 3050 len 20480
33896 065/300 Read block 4535 len 36864
33897 066/300 Write block 3484 len 8192 data: 5 3
33898 067/300 Read block 3049 len 32768
33899 068/300 Write block 820 len 8192 data: 7 6
33900 069/300 Write block 1882 len 20480 data: 4 5 5 5 4
33901 070/300 Read block 4686 len 12288
33902 071/300 Read block 2729 len 4096
33903 072/300 Read block 4121 len 8192
33904 073/300 Write block 1191 len 12288 data: 3 3 4
33905 074/300 Read block 2063 len 36864
33906 075/300 Write block 837 len 12288 data: 6 7 7
33907 076/300 Read block 4233 len 16384
33908 077/300 Write block 1545 len 12288 data: 3 5 4
33909 078/300 Write block 2384 len 20480 data: 10 9 8 6 3
33910 079/300 Read block 2425 len 12288
33911 080/300 Read block 1700 len 8192
33912 081/300 Read block 2337 len 24576
33913 082/300 Flush
33914 083/300 Read block 1936 len 16384
33915 084/300 Read block 1544 len 32768
33916 085/300 Read block 2221 len 24576
33917 086/300 Write block 564 len 8192 data: 5 4
33918 087/300 Read block 4205 len 12288
33919 088/300 Read block 128 len 36864
33920 089/300 Read block 1719 len 8192
33921 090/300 Read block 1306 len 28672
33922 091/300 Read block 4105 len 28672
33923 092/300 Read block 2391 len 20480
33924 093/300 Read block 4639 len 16384
33925 094/300 Read block 965 len 24576
33926 095/300 Write block 2161 len 8192 data: 4 4
33927 096/300 Flush
33928 097/300 Read block 3884 len 16384
33929 098/300 Read block 1680 len 12288
33930 099/300 Read block 4319 len 40960
33931 100/300 Read block 3747 len 4096
33932 101/300 Read block 397 len 12288
33933 102/300 Write block 3897 len 24576 data: 5 4 6 5 4 3
33934 103/300 Flush
33935 104/300 Read block 3610 len 8192
33936 105/300 Read block 1016 len 28672
33937 106/300 Read block 1137 len 40960
33938 107/300 Write block 3564 len 28672 data: 6 6 7 6 4 3 4
33939 108/300 Read block 4776 len 20480
33940 109/300 Write block 306 len 12288 data: 5 4 5
33941 110/300 Write block 3425 len 20480 data: 7 9 8 9 10
33942 111/300 Read block 2872 len 16384
33943 112/300 Read block 3086 len 16384
33944 113/300 Write block 4278 len 16384 data: 4 6 5 6
33945 114/300 Write block 1654 len 36864 data: 5 5 7 5 6 6 6 5 5
33946 115/300 Read block 1062 len 4096
33947 116/300 Read block 598 len 28672
33948 117/300 Read block 3726 len 20480
33949 118/300 Write block 39 len 28672 data: 6 8 8 8 8 11 12
33950 119/300 Flush
33951 120/300 Read block 4106 len 28672
33952 121/300 Write block 4627 len 12288 data: 5 4 3
33953 122/300 Write block 4753 len 16384 data: 5 4 6 7
33954 123/300 Write block 4706 len 40960 data: 5 5 5 6 8 7 8 7 10 9
33955 124/300 Write block 4732 len 20480 data: 5 7 7 5 6
33956 125/300 Write block 1965 len 32768 data: 9 10 9 10 9 7 7 4
33957 126/300 Write block 1292 len 40960 data: 5 5 5 5 4 4 6 5 5 4
33958 127/300 Read block 2691 len 20480
33959 128/300 Write block 2103 len 20480 data: 5 4 4 5 4
33960 129/300 Flush
33961 130/300 Write block 4420 len 4096 data: 3
33962 131/300 Read block 832 len 16384
33963 132/300 Write block 834 len 32768 data: 5 4 5 7 8 8 6 6
33964 133/300 Read block 1243 len 20480
33965 134/300 Write block 2880 len 4096 data: 6
33966 135/300 Write block 2478 len 4096 data: 6
33967 136/300 Read block 2968 len 40960
33968 137/300 Read block 3024 len 36864
33969 138/300 Write block 1921 len 16384 data: 5 7 8 7
33970 139/300 Read block 18 len 32768
33971 140/300 Read block 4435 len 12288
33972 141/300 Read block 47 len 36864
33973 142/300 Write block 3434 len 28672 data: 8 9 8 10 9 8 9
33974 143/300 Write block 2597 len 40960 data: 9 9 8 9 8 11 11 9 9 8
33975 144/300 Read block 3345 len 4096
33976 145/300 Read block 1811 len 4096
33977 146/300 Read block 2928 len 40960
33978 147/300 Read block 1996 len 40960
33979 148/300 Read block 2713 len 40960
33980 149/300 Read block 4977 len 36864
33981 150/300 Read block 3168 len 4096
33982 151/300 Write block 751 len 16384 data: 5 6 8 9
33983 152/300 Write block 2135 len 24576 data: 5 5 6 6 6 7
33984 153/300 Read block 3524 len 32768
33985 154/300 Write block 4513 len 4096 data: 7
33986 155/300 Read block 4504 len 32768
33987 156/300 Flush
33988 157/300 Read block 1843 len 28672
33989 158/300 Write block 3988 len 20480 data: 3 3 4 4 4
33990 159/300 Read block 3338 len 40960
33991 160/300 Read block 3082 len 12288
33992 161/300 Write block 163 len 32768 data: 8 8 4 5 5 5 5 5
33993 162/300 Write block 4281 len 12288 data: 7 5 5
33994 163/300 Read block 2054 len 36864
33995 164/300 Read block 4609 len 28672
33996 165/300 Write block 4780 len 4096 data: 6
33997 166/300 Write block 4986 len 4096 data: 5
33998 167/300 Read block 3323 len 32768
33999 168/300 Read block 1866 len 28672
34000 169/300 Write block 3920 len 16384 data: 5 5 5 5
34001 170/300 Write block 1943 len 24576 data: 4 3 3 3 4 4
34002 171/300 Read block 715 len 8192
34003 172/300 Read block 1786 len 28672
34004 173/300 Flush
34005 174/300 Write block 2028 len 32768 data: 6 4 3 3 4 5 4 6
34006 175/300 Read block 796 len 12288
34007 176/300 Write block 414 len 28672 data: 9 9 10 10 9 10 11
34008 177/300 Read block 3425 len 12288
34009 178/300 Write block 3363 len 20480 data: 3 4 3 3 3
34010 179/300 Write block 4842 len 16384 data: 6 8 7 6
34011 180/300 Write block 3099 len 40960 data: 6 6 5 6 6 5 6 6 6 5
34012 181/300 Read block 4172 len 16384
34013 182/300 Write block 745 len 40960 data: 8 7 5 5 5 3 6 7 9 10
34014 183/300 Read block 4426 len 28672
34015 184/300 Write block 1143 len 28672 data: 3 3 3 2 2 3 3
34016 185/300 Read block 3052 len 4096
34017 186/300 Read block 102 len 12288
34018 187/300 Read block 1203 len 24576
34019 188/300 Flush
34020 189/300 Read block 1353 len 28672
34021 190/300 Write block 4147 len 24576 data: 6 7 8 6 8 8
34022 191/300 Write block 4941 len 20480 data: 7 6 5 5 5
34023 192/300 Read block 3894 len 12288
34024 193/300 Read block 2248 len 8192
34025 194/300 Write block 1754 len 20480 data: 7 7 5 6 7
34026 195/300 Read block 840 len 24576
34027 196/300 Read block 1919 len 32768
34028 197/300 Flush
34029 198/300 Read block 3228 len 28672
34030 199/300 Read block 616 len 12288
34031 200/300 Read block 1706 len 32768
34032 201/300 Write block 4628 len 32768 data: 5 4 3 5 4 4 3 3
34033 202/300 Read block 4220 len 28672
34034 203/300 Write block 1606 len 28672 data: 7 6 8 9 8 7 5
34035 204/300 Flush
34036 205/300 Write block 1972 len 36864 data: 5 4 5 5 6 5 5 7 8
34037 206/300 Read block 1842 len 8192
34038 207/300 Read block 4389 len 4096
34039 208/300 Write block 1072 len 8192 data: 5 4
34040 209/300 Read block 2761 len 8192
34041 210/300 Write block 1148 len 20480 data: 4 4 4 4 5
34042 211/300 Flush
34043 212/300 Read block 642 len 32768
34044 213/300 Flush
34045 214/300 Write block 4039 len 28672 data: 2 2 2 4 3 2 2
34046 215/300 Write block 1797 len 12288 data: 5 5 6
34047 216/300 Read block 3481 len 28672
34048 217/300 Read block 4990 len 12288
34049 218/300 Write block 2727 len 16384 data: 5 5 5 5
34050 219/300 Read block 1776 len 20480
34051 220/300 Write block 4283 len 36864 data: 6 6 5 6 4 5 5 6 7
34052 221/300 Write block 2853 len 16384 data: 4 4 4 4
34053 222/300 Read block 172 len 12288
34054 223/300 Read block 4498 len 24576
34055 224/300 Read block 3948 len 4096
34056 225/300 Write block 943 len 36864 data: 5 7 8 7 8 8 7 5 6
34057 226/300 Write block 1616 len 36864 data: 7 6 5 4 4 5 6 5 4
34058 227/300 Read block 4443 len 16384
34059 228/300 Write block 2574 len 28672 data: 9 9 9 10 8 6 6
34060 229/300 Flush
34061 230/300 Read block 354 len 4096
34062 231/300 Write block 2940 len 28672 data: 5 4 5 3 3 4 6
34063 232/300 Read block 4785 len 40960
34064 233/300 Write block 1660 len 12288 data: 7 6 6
34065 234/300 Write block 2088 len 36864 data: 5 4 4 4 5 5 3 3 4
34066 235/300 Read block 4302 len 28672
34067 236/300 Read block 4165 len 8192
34068 237/300 Read block 2688 len 20480
34069 238/300 Read block 1196 len 20480
34070 239/300 Read block 1824 len 20480
34071 240/300 Write block 1308 len 36864 data: 5 5 6 6 4 4 4 4 5
34072 241/300 Write block 3630 len 16384 data: 6 6 6 6
34073 242/300 Read block 3784 len 28672
34074 243/300 Write block 2591 len 20480 data: 5 5 5 5 5
34075 244/300 Write block 3172 len 4096 data: 4
34076 245/300 Write block 3821 len 36864 data: 7 7 7 6 5 5 6 7 6
34077 246/300 Read block 3391 len 24576
34078 247/300 Write block 3583 len 40960 data: 5 5 4 4 5 5 8 7 7 4
34079 248/300 Read block 7 len 40960
34080 249/300 Write block 2619 len 24576 data: 6 7 7 9 9 7
34081 250/300 Write block 1537 len 4096 data: 4
34082 251/300 Read block 4364 len 40960
34083 252/300 Read block 2695 len 32768
34084 253/300 Read block 768 len 12288
34085 254/300 Write block 2929 len 40960 data: 7 6 6 7 5 5 6 5 5 5
34086 255/300 Read block 4061 len 40960
34087 256/300 Write block 1914 len 4096 data: 3
34088 257/300 Write block 4100 len 36864 data: 5 5 4 3 3 3 5 4 4
34089 258/300 Read block 3050 len 20480
34090 259/300 Read block 3103 len 16384
34091 260/300 Read block 1877 len 16384
34092 261/300 Write block 4663 len 28672 data: 9 9 9 10 9 9 9
34093 262/300 Flush
34094 263/300 Write block 3029 len 36864 data: 6 5 5 2 2 3 5 5 6
34095 264/300 Read block 1271 len 20480
34096 265/300 Write block 934 len 16384 data: 6 6 5 4
34097 266/300 Flush
34098 267/300 Read block 875 len 24576
34099 268/300 Write block 2400 len 40960 data: 9 8 7 7 9 10 8 6 5 4
34100 269/300 Flush
34101 270/300 Read block 4048 len 32768
34102 271/300 Flush
34103 272/300 Read block 668 len 36864
34104 273/300 Write block 1179 len 28672 data: 5 5 5 4 5 5 6
34105 274/300 Write block 1666 len 28672 data: 5 7 6 8 10 9 7
34106 275/300 Write block 2437 len 4096 data: 5
34107 276/300 Write block 2057 len 8192 data: 6 4
34108 277/300 Flush
34109 278/300 Write block 1626 len 20480 data: 6 5 5 5 5
34110 279/300 Write block 3728 len 12288 data: 7 7 11
34111 280/300 Read block 3868 len 32768
34112 281/300 Read block 1846 len 4096
34113 282/300 Write block 271 len 20480 data: 4 4 5 5 7
34114 283/300 Write block 2447 len 40960 data: 10 9 9 9 5 4 4 4 3 4
34115 284/300 Write block 2910 len 40960 data: 4 5 5 5 6 5 5 5 5 5
34116 285/300 Write block 4974 len 12288 data: 6 7 6
34117 286/300 Read block 1127 len 28672
34118 287/300 Write block 1957 len 8192 data: 7 7
34119 288/300 Write block 3864 len 4096 data: 4
34120 289/300 Read block 134 len 12288
34121 290/300 Read block 2905 len 8192
34122 291/300 Read block 4480 len 32768
34123 292/300 Write block 2268 len 36864 data: 3 3 3 3 2 4 5 6 6
34124 293/300 Write block 4038 len 4096 data: 3
34125 294/300 Write block 880 len 12288 data: 2 2 4
34126 295/300 Write block 2544 len 36864 data: 6 7 7 6 7 9 9 7 8
34127 296/300 Flush
34128 297/300 Read block 3515 len 16384
34129 298/300 Read block 4758 len 20480
34130 299/300 Read block 2088 len 32768
34131 300/300 Flush
34132 301/300 Read block 4206 len 8192
34133 Main task has work to do, go find it
34134 got dsc Start(1)
34135 start 1
34136 Mar 17 22:26:37.204 INFO request completed, latency_us: 209, response_code: 204, uri: /start/cid/1, method: POST, req_id: 2e09cdab-031c-4212-a638-c0760f29d064, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
34137 Replay: started 1, returned:Ok(())
34138 ----------------------------------------------------------------
34139 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
34140 States: Active Offline Active
34141 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34142 9152 Acked Read 10 Done Sent Done false
34143 9153 Acked Read 8 Done Sent Done false
34144 9154 Acked Read 1 Done Sent Done false
34145 9155 Acked Read 3 Done Sent Done false
34146 9156 Acked Write 3 Done Sent Done false
34147 9157 Acked Write 9 Done Sent Done false
34148 9158 Acked Read 9 Done Sent Done false
34149 9159 Acked Write 4 Done Sent Done false
34150 9160 Acked Read 7 Done Sent Done false
34151 9161 Acked Read 10 Done Sent Done false
34152 9162 Acked Read 1 Done Sent Done false
34153 9163 Acked Write 8 Done Sent Done false
34154 9164 Acked Read 1 Done Sent Done false
34155 9165 Acked Write 7 Done Sent Done false
34156 9166 Acked Read 8 Done Sent Done false
34157 9167 Acked Flush 0 Done Sent Done false
34158 9168 Acked Read 7 Done Sent Done false
34159 9169 Acked Write 9 Done Sent Done false
34160 9170 Acked Read 7 Done Sent Done false
34161 9171 Acked Write 9 Done Sent Done false
34162 9172 Acked Read 7 Done Sent Done false
34163 9173 Acked Write 9 Done Sent Done false
34164 9174 Acked Read 7 Done Sent Done false
34165 9175 Acked Read 9 Done Sent Done false
34166 9176 Acked Write 5 Done Sent Done false
34167 9177 Acked Read 8 Done Sent Done false
34168 9178 Acked Read 4 Done Sent Done false
34169 9179 Acked Write 8 Done Sent Done false
34170 9180 Acked Read 5 Done Sent Done false
34171 9181 Acked Read 5 Done Sent Done false
34172 9182 Acked Read 9 Done Sent Done false
34173 9183 Acked Read 9 Done Sent Done false
34174 9184 Acked Write 10 Done Sent Done false
34175 9185 Acked Read 7 Done Sent Done false
34176 9186 Acked Read 8 Done Sent Done false
34177 9187 Acked Write 3 Done Sent Done false
34178 9188 Acked Write 8 Done Sent Done false
34179 9189 Acked Write 3 Done Sent Done false
34180 9190 Acked Write 6 Done Sent Done false
34181 9191 Acked Write 3 Done Sent Done false
34182 9192 Acked Read 5 Done Sent Done false
34183 9193 Acked Read 9 Done Sent Done false
34184 9194 Acked Write 6 Done Sent Done false
34185 9195 Acked Read 10 Done Sent Done false
34186 9196 Acked Flush 0 Done Sent Done false
34187 9197 Acked Read 6 Done Sent Done false
34188 9198 Acked Write 4 Done Sent Done false
34189 9199 Acked Write 2 Done Sent Done false
34190 9200 Acked Read 5 Done Sent Done false
34191 9201 Acked Read 9 Done Sent Done false
34192 9202 Acked Write 3 Done Sent Done false
34193 9203 Acked Read 4 Done Sent Done false
34194 9204 Acked Read 3 Done Sent Done false
34195 9205 Acked Write 8 Done Sent Done false
34196 9206 Acked Write 6 Done Sent Done false
34197 9207 Acked Write 9 Done Sent Done false
34198 9208 Acked Write 4 Done Sent Done false
34199 9209 Acked Flush 0 Done Sent Done false
34200 9210 Acked Read 6 Done Sent Done false
34201 9211 Acked Read 5 Done Sent Done false
34202 9212 Acked Write 9 Done Sent Done false
34203 9213 Acked Write 8 Done Sent Done false
34204 9214 Acked Read 7 Done Sent Done false
34205 9215 Acked Read 5 Done Sent Done false
34206 9216 Acked Read 9 Done Sent Done false
34207 9217 Acked Write 2 Done Sent Done false
34208 9218 Acked Read 8 Done Sent Done false
34209 9219 Acked Write 2 Done Sent Done false
34210 9220 Acked Write 5 Done Sent Done false
34211 9221 Acked Read 3 Done Sent Done false
34212 9222 Acked Read 1 Done Sent Done false
34213 9223 Acked Read 2 Done Sent Done false
34214 9224 Acked Write 3 Done Sent Done false
34215 9225 Acked Read 9 Done Sent Done false
34216 9226 Acked Write 3 Done Sent Done false
34217 9227 Acked Read 4 Done Sent Done false
34218 9228 Acked Write 3 Done Sent Done false
34219 9229 Acked Write 5 Done Sent Done false
34220 9230 Acked Read 3 Done Sent Done false
34221 9231 Acked Read 2 Done Sent Done false
34222 9232 Acked Read 6 Done Sent Done false
34223 9233 Acked Flush 0 Done Sent Done false
34224 9234 Acked Read 4 Done Sent Done false
34225 9235 Acked Read 8 Done Sent Done false
34226 9236 Acked Read 6 Done Sent Done false
34227 9237 Acked Write 2 Done Sent Done false
34228 9238 Acked Read 3 Done Sent Done false
34229 9239 Acked Read 9 Done Sent Done false
34230 9240 Acked Read 2 Done Sent Done false
34231 9241 Acked Read 7 Done Sent Done false
34232 9242 Acked Read 7 Done Sent Done false
34233 9243 Acked Read 5 Done Sent Done false
34234 9244 Acked Read 4 Done Sent Done false
34235 9245 Acked Read 6 Done Sent Done false
34236 9246 Acked Write 2 Done Sent Done false
34237 9247 Acked Flush 0 Done Sent Done false
34238 9248 Acked Read 4 Done Sent Done false
34239 9249 Acked Read 3 Done Sent Done false
34240 9250 Acked Read 10 Done Sent Done false
34241 9251 Acked Read 1 Done Sent Done false
34242 9252 Acked Read 3 Done Sent Done false
34243 9253 Acked Write 6 Done Sent Done false
34244 9254 Acked Flush 0 Done Sent Done false
34245 9255 Acked Read 2 Done Sent Done false
34246 9256 Acked Read 7 Done Sent Done false
34247 9257 Acked Read 10 Done Sent Done false
34248 9258 Acked Write 7 Done Sent Done false
34249 9259 Acked Read 5 Done Sent Done false
34250 9260 Acked Write 3 Done Sent Done false
34251 9261 Acked Write 5 Done Sent Done false
34252 9262 Acked Read 4 Done Sent Done false
34253 9263 Acked Read 4 Done Sent Done false
34254 9264 Acked Write 4 Done Sent Done false
34255 9265 Acked Write 9 Done Sent Done false
34256 9266 Acked Read 1 Done Sent Done false
34257 9267 Acked Read 7 Done Sent Done false
34258 9268 Acked Read 5 Done Sent Done false
34259 9269 Acked Write 7 Done Sent Done false
34260 9270 Acked Flush 0 Done Sent Done false
34261 9271 Acked Read 7 Done Sent Done false
34262 9272 Acked Write 3 Done Sent Done false
34263 9273 Acked Write 4 Done Sent Done false
34264 9274 Acked Write 10 Done Sent Done false
34265 9275 Acked Write 5 Done Sent Done false
34266 9276 Acked Write 8 Done Sent Done false
34267 9277 Acked Write 10 Done Sent Done false
34268 9278 Acked Read 5 Done Sent Done false
34269 9279 Acked Write 5 Done Sent Done false
34270 9280 Acked Flush 0 Done Sent Done false
34271 9281 Acked Write 1 Done Sent Done false
34272 9282 Acked Read 4 Done Sent Done false
34273 9283 Acked Write 8 Done Sent Done false
34274 9284 Acked Read 5 Done Sent Done false
34275 9285 Acked Write 1 Done Sent Done false
34276 9286 Acked Write 1 Done Sent Done false
34277 9287 Acked Read 10 Done Sent Done false
34278 9288 Acked Read 9 Done Sent Done false
34279 9289 Acked Write 4 Done Sent Done false
34280 9290 Acked Read 8 Done Sent Done false
34281 9291 Acked Read 3 Done Sent Done false
34282 9292 Acked Read 9 Done Sent Done false
34283 9293 Acked Write 7 Done Sent Done false
34284 9294 Acked Write 10 Done Sent Done false
34285 9295 Acked Read 1 Done Sent Done false
34286 9296 Acked Read 1 Done Sent Done false
34287 9297 Acked Read 10 Done Sent Done false
34288 9298 Acked Read 10 Done Sent Done false
34289 9299 Acked Read 10 Done Sent Done false
34290 9300 Acked Read 9 Done Sent Done false
34291 9301 Acked Read 1 Done Sent Done false
34292 9302 Acked Write 4 Done Sent Done false
34293 9303 Acked Write 6 Done Sent Done false
34294 9304 Acked Read 8 Done Sent Done false
34295 9305 Acked Write 1 Done Sent Done false
34296 9306 Acked Read 8 Done Sent Done false
34297 9307 Acked Flush 0 Done Sent Done false
34298 9308 Acked Read 7 Done Sent Done false
34299 9309 Acked Write 5 Done Sent Done false
34300 9310 Acked Read 10 Done Sent Done false
34301 9311 Acked Read 3 Done Sent Done false
34302 9312 Acked Write 8 Done Sent Done false
34303 9313 Acked Write 3 Done Sent Done false
34304 9314 Acked Read 9 Done Sent Done false
34305 9315 Acked Read 7 Done Sent Done false
34306 9316 Acked Write 1 Done Sent Done false
34307 9317 Acked Write 1 Done Sent Done false
34308 9318 Acked Read 8 Done Sent Done false
34309 9319 Acked Read 7 Done Sent Done false
34310 9320 Acked Write 4 Done Sent Done false
34311 9321 Acked Write 6 Done Sent Done false
34312 9322 Acked Read 2 Done Sent Done false
34313 9323 Acked Read 7 Done Sent Done false
34314 9324 Acked Flush 0 Done Sent Done false
34315 9325 Acked Write 8 Done Sent Done false
34316 9326 Acked Read 3 Done Sent Done false
34317 9327 Acked Write 7 Done Sent Done false
34318 9328 Acked Read 3 Done Sent Done false
34319 9329 Acked Write 5 Done Sent Done false
34320 9330 Acked Write 4 Done Sent Done false
34321 9331 Acked Write 10 Done Sent Done false
34322 9332 Acked Read 4 Done Sent Done false
34323 9333 Acked Write 10 Done Sent Done false
34324 9334 Acked Read 7 Done Sent Done false
34325 9335 Acked Write 7 Done Sent Done false
34326 9336 Acked Read 1 Done Sent Done false
34327 9337 Acked Read 3 Done Sent Done false
34328 9338 Acked Read 6 Done Sent Done false
34329 9339 Acked Flush 0 Done Sent Done false
34330 9340 Acked Read 7 Done Sent Done false
34331 9341 Acked Write 6 Done Sent Done false
34332 9342 Acked Write 5 Done Sent Done false
34333 9343 Acked Read 3 Done Sent Done false
34334 9344 Acked Read 2 Done Sent Done false
34335 9345 Acked Write 5 Done Sent Done false
34336 9346 Acked Read 6 Done Sent Done false
34337 9347 Acked Read 8 Done Sent Done false
34338 9348 Acked Flush 0 Done Sent Done false
34339 9349 Acked Read 7 Done Sent Done false
34340 9350 Acked Read 3 Done Sent Done false
34341 9351 Acked Read 8 Done Sent Done false
34342 9352 Acked Write 8 Done Sent Done false
34343 9353 Acked Read 7 Done Sent Done false
34344 9354 Acked Write 7 Done Sent Done false
34345 9355 Acked Flush 0 Done Sent Done false
34346 9356 Acked Write 9 Done Sent Done false
34347 9357 Acked Read 2 Done Sent Done false
34348 9358 Acked Read 1 Done Sent Done false
34349 9359 Acked Write 2 Done Sent Done false
34350 9360 Acked Read 2 Done Sent Done false
34351 9361 Acked Write 5 Done Sent Done false
34352 9362 Acked Flush 0 Done Sent Done false
34353 9363 Acked Read 8 Done Sent Done false
34354 9364 Acked Flush 0 Done Sent Done false
34355 9365 Acked Write 7 Done Sent Done false
34356 9366 Acked Write 3 Done Sent Done false
34357 9367 Acked Read 7 Done Sent Done false
34358 9368 Acked Read 3 Done Sent Done false
34359 9369 Acked Write 4 Done Sent Done false
34360 9370 Acked Read 5 Done Sent Done false
34361 9371 Acked Write 9 Done Sent Done false
34362 9372 Acked Write 4 Done Sent Done false
34363 9373 Acked Read 3 Done Sent Done false
34364 9374 Acked Read 6 Done Sent Done false
34365 9375 Acked Read 1 Done Sent Done false
34366 9376 Acked Write 9 Done Sent Done false
34367 9377 Acked Write 9 Done Sent Done false
34368 9378 Acked Read 4 Done Sent Done false
34369 9379 Acked Write 7 Done Sent Done false
34370 9380 Acked Flush 0 Done Sent Done false
34371 9381 Acked Read 1 Done Sent Done false
34372 9382 Acked Write 7 Done Sent Done false
34373 9383 Acked Read 10 Done Sent Done false
34374 9384 Acked Write 3 Done Sent Done false
34375 9385 Acked Write 9 Done Sent Done false
34376 9386 Acked Read 7 Done Sent Done false
34377 9387 Acked Read 2 Done Sent Done false
34378 9388 Acked Read 5 Done Sent Done false
34379 9389 Acked Read 5 Done Sent Done false
34380 9390 Acked Read 5 Done Sent Done false
34381 9391 Acked Write 9 Done Sent Done false
34382 9392 Acked Write 4 Done Sent Done false
34383 9393 Acked Read 7 Done Sent Done false
34384 9394 Acked Write 5 Done Sent Done false
34385 9395 Acked Write 1 Done Sent Done false
34386 9396 Acked Write 9 Done Sent Done false
34387 9397 Acked Read 6 Done Sent Done false
34388 9398 Acked Write 10 Done Sent Done false
34389 9399 Acked Read 10 Done Sent Done false
34390 9400 Acked Write 6 Done Sent Done false
34391 9401 Acked Write 1 Done Sent Done false
34392 9402 Acked Read 10 Done Sent Done false
34393 9403 Acked Read 8 Done Sent Done false
34394 9404 Acked Read 3 Done Sent Done false
34395 9405 Acked Write 10 Done Sent Done false
34396 9406 Acked Read 10 Done Sent Done false
34397 9407 Acked Write 1 Done Sent Done false
34398 9408 Acked Write 9 Done Sent Done false
34399 9409 Acked Read 5 Done Sent Done false
34400 9410 Acked Read 4 Done Sent Done false
34401 9411 Acked Read 4 Done Sent Done false
34402 9412 Acked Write 7 Done Sent Done false
34403 9413 Acked Flush 0 Done Sent Done false
34404 9414 Acked Write 9 Done Sent Done false
34405 9415 Acked Read 5 Done Sent Done false
34406 9416 Acked Write 4 Done Sent Done false
34407 9417 Acked Flush 0 Done Sent Done false
34408 9418 Acked Read 6 Done Sent Done false
34409 9419 Acked Write 10 Done Sent Done false
34410 9420 Acked Flush 0 Done Sent Done false
34411 9421 Acked Read 8 Done Sent Done false
34412 9422 Acked Flush 0 Done Sent Done false
34413 9423 Acked Read 9 Done Sent Done false
34414 9424 Acked Write 7 Done Sent Done false
34415 9425 Acked Write 7 Done Sent Done false
34416 9426 Acked Write 1 Done Sent Done false
34417 9427 Acked Write 2 Done Sent Done false
34418 9428 Acked Flush 0 Done Sent Done false
34419 9429 Acked Write 5 Done Sent Done false
34420 9430 Acked Write 3 Done Sent Done false
34421 9431 Acked Read 8 Done Sent Done false
34422 9432 Acked Read 1 Done Sent Done false
34423 9433 Acked Write 5 Done Sent Done false
34424 9434 Acked Write 10 Done Sent Done false
34425 9435 Acked Write 10 Done Sent Done false
34426 9436 Acked Write 3 Done Sent Done false
34427 9437 Acked Read 7 Done Sent Done false
34428 9438 Acked Write 2 Done Sent Done false
34429 9439 Acked Write 1 Done Sent Done false
34430 9440 Acked Read 3 Done Sent Done false
34431 9441 Acked Read 2 Done Sent Done false
34432 9442 Acked Read 8 Done Sent Done false
34433 9443 Acked Write 9 Done Sent Done false
34434 9444 Acked Write 1 Done Sent Done false
34435 9445 Acked Write 3 Done Sent Done false
34436 9446 Acked Write 9 Done Sent Done false
34437 9447 Acked Flush 0 Done Sent Done false
34438 9448 Acked Read 4 Done Sent Done false
34439 9449 Acked Read 5 Done Sent Done false
34440 9450 Acked Read 8 Done Sent Done false
34441 9451 Acked Flush 0 Done Sent Done false
34442 9452 Acked Read 2 Done Sent Done false
34443 STATES DS:0 DS:1 DS:2 TOTAL
34444 Sent 0 301 0 301
34445 Done 301 0 301 602
34446 Skipped 0 0 0 0
34447 Error 0 0 0 0
34448 Last Flush: Some(JobId(9451)) Some(JobId(9151)) Some(JobId(9451))
34449 Downstairs last five completed: 9151 9150 9149 9148 9147
34450 Upstairs last five completed: 9452 9451 9450 9449 9448
34451 CLIENT: Up:0 ds:301 act:2
344522025-03-17T22:26:37.705ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
344532025-03-17T22:26:38.319ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
344542025-03-17T22:26:39.321ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
344552025-03-17T22:26:40.324ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
34456 [8820] got start action
34457 [8820] I am going to restart
34458 Starting downstairs at port 8820
34459 Make output file at "/tmp/dsc/downstairs-8820.txt"
34460 [8820][1] reports Starting
34461 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1239)
34462 [8820][1] reports Running
34463 ----------------------------------------------------------------
34464 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
34465 States: Active Offline Active
34466 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34467 9152 Acked Read 10 Done Sent Done false
34468 9153 Acked Read 8 Done Sent Done false
34469 9154 Acked Read 1 Done Sent Done false
34470 9155 Acked Read 3 Done Sent Done false
34471 9156 Acked Write 3 Done Sent Done false
34472 9157 Acked Write 9 Done Sent Done false
34473 9158 Acked Read 9 Done Sent Done false
34474 9159 Acked Write 4 Done Sent Done false
34475 9160 Acked Read 7 Done Sent Done false
34476 9161 Acked Read 10 Done Sent Done false
34477 9162 Acked Read 1 Done Sent Done false
34478 9163 Acked Write 8 Done Sent Done false
34479 9164 Acked Read 1 Done Sent Done false
34480 9165 Acked Write 7 Done Sent Done false
34481 9166 Acked Read 8 Done Sent Done false
34482 9167 Acked Flush 0 Done Sent Done false
34483 9168 Acked Read 7 Done Sent Done false
34484 9169 Acked Write 9 Done Sent Done false
34485 9170 Acked Read 7 Done Sent Done false
34486 9171 Acked Write 9 Done Sent Done false
34487 9172 Acked Read 7 Done Sent Done false
34488 9173 Acked Write 9 Done Sent Done false
34489 9174 Acked Read 7 Done Sent Done false
34490 9175 Acked Read 9 Done Sent Done false
34491 9176 Acked Write 5 Done Sent Done false
34492 9177 Acked Read 8 Done Sent Done false
34493 9178 Acked Read 4 Done Sent Done false
34494 9179 Acked Write 8 Done Sent Done false
34495 9180 Acked Read 5 Done Sent Done false
34496 9181 Acked Read 5 Done Sent Done false
34497 9182 Acked Read 9 Done Sent Done false
34498 9183 Acked Read 9 Done Sent Done false
34499 9184 Acked Write 10 Done Sent Done false
34500 9185 Acked Read 7 Done Sent Done false
34501 9186 Acked Read 8 Done Sent Done false
34502 9187 Acked Write 3 Done Sent Done false
34503 9188 Acked Write 8 Done Sent Done false
34504 9189 Acked Write 3 Done Sent Done false
34505 9190 Acked Write 6 Done Sent Done false
34506 9191 Acked Write 3 Done Sent Done false
34507 9192 Acked Read 5 Done Sent Done false
34508 9193 Acked Read 9 Done Sent Done false
34509 9194 Acked Write 6 Done Sent Done false
34510 9195 Acked Read 10 Done Sent Done false
34511 9196 Acked Flush 0 Done Sent Done false
34512 9197 Acked Read 6 Done Sent Done false
34513 9198 Acked Write 4 Done Sent Done false
34514 9199 Acked Write 2 Done Sent Done false
34515 9200 Acked Read 5 Done Sent Done false
34516 9201 Acked Read 9 Done Sent Done false
34517 9202 Acked Write 3 Done Sent Done false
34518 9203 Acked Read 4 Done Sent Done false
34519 9204 Acked Read 3 Done Sent Done false
34520 9205 Acked Write 8 Done Sent Done false
34521 9206 Acked Write 6 Done Sent Done false
34522 9207 Acked Write 9 Done Sent Done false
34523 9208 Acked Write 4 Done Sent Done false
34524 9209 Acked Flush 0 Done Sent Done false
34525 9210 Acked Read 6 Done Sent Done false
34526 9211 Acked Read 5 Done Sent Done false
34527 9212 Acked Write 9 Done Sent Done false
34528 9213 Acked Write 8 Done Sent Done false
34529 9214 Acked Read 7 Done Sent Done false
34530 9215 Acked Read 5 Done Sent Done false
34531 9216 Acked Read 9 Done Sent Done false
34532 9217 Acked Write 2 Done Sent Done false
34533 9218 Acked Read 8 Done Sent Done false
34534 9219 Acked Write 2 Done Sent Done false
34535 9220 Acked Write 5 Done Sent Done false
34536 9221 Acked Read 3 Done Sent Done false
34537 9222 Acked Read 1 Done Sent Done false
34538 9223 Acked Read 2 Done Sent Done false
34539 9224 Acked Write 3 Done Sent Done false
34540 9225 Acked Read 9 Done Sent Done false
34541 9226 Acked Write 3 Done Sent Done false
34542 9227 Acked Read 4 Done Sent Done false
34543 9228 Acked Write 3 Done Sent Done false
34544 9229 Acked Write 5 Done Sent Done false
34545 9230 Acked Read 3 Done Sent Done false
34546 9231 Acked Read 2 Done Sent Done false
34547 9232 Acked Read 6 Done Sent Done false
34548 9233 Acked Flush 0 Done Sent Done false
34549 9234 Acked Read 4 Done Sent Done false
34550 9235 Acked Read 8 Done Sent Done false
34551 9236 Acked Read 6 Done Sent Done false
34552 9237 Acked Write 2 Done Sent Done false
34553 9238 Acked Read 3 Done Sent Done false
34554 9239 Acked Read 9 Done Sent Done false
34555 9240 Acked Read 2 Done Sent Done false
34556 9241 Acked Read 7 Done Sent Done false
34557 9242 Acked Read 7 Done Sent Done false
34558 9243 Acked Read 5 Done Sent Done false
34559 9244 Acked Read 4 Done Sent Done false
34560 9245 Acked Read 6 Done Sent Done false
34561 9246 Acked Write 2 Done Sent Done false
34562 9247 Acked Flush 0 Done Sent Done false
34563 9248 Acked Read 4 Done Sent Done false
34564 9249 Acked Read 3 Done Sent Done false
34565 9250 Acked Read 10 Done Sent Done false
34566 9251 Acked Read 1 Done Sent Done false
34567 9252 Acked Read 3 Done Sent Done false
34568 9253 Acked Write 6 Done Sent Done false
34569 9254 Acked Flush 0 Done Sent Done false
34570 9255 Acked Read 2 Done Sent Done false
34571 9256 Acked Read 7 Done Sent Done false
34572 9257 Acked Read 10 Done Sent Done false
34573 9258 Acked Write 7 Done Sent Done false
34574 9259 Acked Read 5 Done Sent Done false
34575 9260 Acked Write 3 Done Sent Done false
34576 9261 Acked Write 5 Done Sent Done false
34577 9262 Acked Read 4 Done Sent Done false
34578 9263 Acked Read 4 Done Sent Done false
34579 9264 Acked Write 4 Done Sent Done false
34580 9265 Acked Write 9 Done Sent Done false
34581 9266 Acked Read 1 Done Sent Done false
34582 9267 Acked Read 7 Done Sent Done false
34583 9268 Acked Read 5 Done Sent Done false
34584 9269 Acked Write 7 Done Sent Done false
34585 9270 Acked Flush 0 Done Sent Done false
34586 9271 Acked Read 7 Done Sent Done false
34587 9272 Acked Write 3 Done Sent Done false
34588 9273 Acked Write 4 Done Sent Done false
34589 9274 Acked Write 10 Done Sent Done false
34590 9275 Acked Write 5 Done Sent Done false
34591 9276 Acked Write 8 Done Sent Done false
34592 9277 Acked Write 10 Done Sent Done false
34593 9278 Acked Read 5 Done Sent Done false
34594 9279 Acked Write 5 Done Sent Done false
34595 9280 Acked Flush 0 Done Sent Done false
34596 9281 Acked Write 1 Done Sent Done false
34597 9282 Acked Read 4 Done Sent Done false
34598 9283 Acked Write 8 Done Sent Done false
34599 9284 Acked Read 5 Done Sent Done false
34600 9285 Acked Write 1 Done Sent Done false
34601 9286 Acked Write 1 Done Sent Done false
34602 9287 Acked Read 10 Done Sent Done false
34603 9288 Acked Read 9 Done Sent Done false
34604 9289 Acked Write 4 Done Sent Done false
34605 9290 Acked Read 8 Done Sent Done false
34606 9291 Acked Read 3 Done Sent Done false
34607 9292 Acked Read 9 Done Sent Done false
34608 9293 Acked Write 7 Done Sent Done false
34609 9294 Acked Write 10 Done Sent Done false
34610 9295 Acked Read 1 Done Sent Done false
34611 9296 Acked Read 1 Done Sent Done false
34612 9297 Acked Read 10 Done Sent Done false
34613 9298 Acked Read 10 Done Sent Done false
34614 9299 Acked Read 10 Done Sent Done false
34615 9300 Acked Read 9 Done Sent Done false
34616 9301 Acked Read 1 Done Sent Done false
34617 9302 Acked Write 4 Done Sent Done false
34618 9303 Acked Write 6 Done Sent Done false
34619 9304 Acked Read 8 Done Sent Done false
34620 9305 Acked Write 1 Done Sent Done false
34621 9306 Acked Read 8 Done Sent Done false
34622 9307 Acked Flush 0 Done Sent Done false
34623 9308 Acked Read 7 Done Sent Done false
34624 9309 Acked Write 5 Done Sent Done false
34625 9310 Acked Read 10 Done Sent Done false
34626 9311 Acked Read 3 Done Sent Done false
34627 9312 Acked Write 8 Done Sent Done false
34628 9313 Acked Write 3 Done Sent Done false
34629 9314 Acked Read 9 Done Sent Done false
34630 9315 Acked Read 7 Done Sent Done false
34631 9316 Acked Write 1 Done Sent Done false
34632 9317 Acked Write 1 Done Sent Done false
34633 9318 Acked Read 8 Done Sent Done false
34634 9319 Acked Read 7 Done Sent Done false
34635 9320 Acked Write 4 Done Sent Done false
34636 9321 Acked Write 6 Done Sent Done false
34637 9322 Acked Read 2 Done Sent Done false
34638 9323 Acked Read 7 Done Sent Done false
34639 9324 Acked Flush 0 Done Sent Done false
34640 9325 Acked Write 8 Done Sent Done false
34641 9326 Acked Read 3 Done Sent Done false
34642 9327 Acked Write 7 Done Sent Done false
34643 9328 Acked Read 3 Done Sent Done false
34644 9329 Acked Write 5 Done Sent Done false
34645 9330 Acked Write 4 Done Sent Done false
34646 9331 Acked Write 10 Done Sent Done false
34647 9332 Acked Read 4 Done Sent Done false
34648 9333 Acked Write 10 Done Sent Done false
34649 9334 Acked Read 7 Done Sent Done false
34650 9335 Acked Write 7 Done Sent Done false
34651 9336 Acked Read 1 Done Sent Done false
34652 9337 Acked Read 3 Done Sent Done false
34653 9338 Acked Read 6 Done Sent Done false
34654 9339 Acked Flush 0 Done Sent Done false
34655 9340 Acked Read 7 Done Sent Done false
34656 9341 Acked Write 6 Done Sent Done false
34657 9342 Acked Write 5 Done Sent Done false
34658 9343 Acked Read 3 Done Sent Done false
34659 9344 Acked Read 2 Done Sent Done false
34660 9345 Acked Write 5 Done Sent Done false
34661 9346 Acked Read 6 Done Sent Done false
34662 9347 Acked Read 8 Done Sent Done false
34663 9348 Acked Flush 0 Done Sent Done false
34664 9349 Acked Read 7 Done Sent Done false
34665 9350 Acked Read 3 Done Sent Done false
34666 9351 Acked Read 8 Done Sent Done false
34667 9352 Acked Write 8 Done Sent Done false
34668 9353 Acked Read 7 Done Sent Done false
34669 9354 Acked Write 7 Done Sent Done false
34670 9355 Acked Flush 0 Done Sent Done false
34671 9356 Acked Write 9 Done Sent Done false
34672 9357 Acked Read 2 Done Sent Done false
34673 9358 Acked Read 1 Done Sent Done false
34674 9359 Acked Write 2 Done Sent Done false
34675 9360 Acked Read 2 Done Sent Done false
34676 9361 Acked Write 5 Done Sent Done false
34677 9362 Acked Flush 0 Done Sent Done false
34678 9363 Acked Read 8 Done Sent Done false
34679 9364 Acked Flush 0 Done Sent Done false
34680 9365 Acked Write 7 Done Sent Done false
34681 9366 Acked Write 3 Done Sent Done false
34682 9367 Acked Read 7 Done Sent Done false
34683 9368 Acked Read 3 Done Sent Done false
34684 9369 Acked Write 4 Done Sent Done false
34685 9370 Acked Read 5 Done Sent Done false
34686 9371 Acked Write 9 Done Sent Done false
34687 9372 Acked Write 4 Done Sent Done false
34688 9373 Acked Read 3 Done Sent Done false
34689 9374 Acked Read 6 Done Sent Done false
34690 9375 Acked Read 1 Done Sent Done false
34691 9376 Acked Write 9 Done Sent Done false
34692 9377 Acked Write 9 Done Sent Done false
34693 9378 Acked Read 4 Done Sent Done false
34694 9379 Acked Write 7 Done Sent Done false
34695 9380 Acked Flush 0 Done Sent Done false
34696 9381 Acked Read 1 Done Sent Done false
34697 9382 Acked Write 7 Done Sent Done false
34698 9383 Acked Read 10 Done Sent Done false
34699 9384 Acked Write 3 Done Sent Done false
34700 9385 Acked Write 9 Done Sent Done false
34701 9386 Acked Read 7 Done Sent Done false
34702 9387 Acked Read 2 Done Sent Done false
34703 9388 Acked Read 5 Done Sent Done false
34704 9389 Acked Read 5 Done Sent Done false
34705 9390 Acked Read 5 Done Sent Done false
34706 9391 Acked Write 9 Done Sent Done false
34707 9392 Acked Write 4 Done Sent Done false
34708 9393 Acked Read 7 Done Sent Done false
34709 9394 Acked Write 5 Done Sent Done false
34710 9395 Acked Write 1 Done Sent Done false
34711 9396 Acked Write 9 Done Sent Done false
34712 9397 Acked Read 6 Done Sent Done false
34713 9398 Acked Write 10 Done Sent Done false
34714 9399 Acked Read 10 Done Sent Done false
34715 9400 Acked Write 6 Done Sent Done false
34716 9401 Acked Write 1 Done Sent Done false
34717 9402 Acked Read 10 Done Sent Done false
34718 9403 Acked Read 8 Done Sent Done false
34719 9404 Acked Read 3 Done Sent Done false
34720 9405 Acked Write 10 Done Sent Done false
34721 9406 Acked Read 10 Done Sent Done false
34722 9407 Acked Write 1 Done Sent Done false
34723 9408 Acked Write 9 Done Sent Done false
34724 9409 Acked Read 5 Done Sent Done false
34725 9410 Acked Read 4 Done Sent Done false
34726 9411 Acked Read 4 Done Sent Done false
34727 9412 Acked Write 7 Done Sent Done false
34728 9413 Acked Flush 0 Done Sent Done false
34729 9414 Acked Write 9 Done Sent Done false
34730 9415 Acked Read 5 Done Sent Done false
34731 9416 Acked Write 4 Done Sent Done false
34732 9417 Acked Flush 0 Done Sent Done false
34733 9418 Acked Read 6 Done Sent Done false
34734 9419 Acked Write 10 Done Sent Done false
34735 9420 Acked Flush 0 Done Sent Done false
34736 9421 Acked Read 8 Done Sent Done false
34737 9422 Acked Flush 0 Done Sent Done false
34738 9423 Acked Read 9 Done Sent Done false
34739 9424 Acked Write 7 Done Sent Done false
34740 9425 Acked Write 7 Done Sent Done false
34741 9426 Acked Write 1 Done Sent Done false
34742 9427 Acked Write 2 Done Sent Done false
34743 9428 Acked Flush 0 Done Sent Done false
34744 9429 Acked Write 5 Done Sent Done false
34745 9430 Acked Write 3 Done Sent Done false
34746 9431 Acked Read 8 Done Sent Done false
34747 9432 Acked Read 1 Done Sent Done false
34748 9433 Acked Write 5 Done Sent Done false
34749 9434 Acked Write 10 Done Sent Done false
34750 9435 Acked Write 10 Done Sent Done false
34751 9436 Acked Write 3 Done Sent Done false
34752 9437 Acked Read 7 Done Sent Done false
34753 9438 Acked Write 2 Done Sent Done false
34754 9439 Acked Write 1 Done Sent Done false
34755 9440 Acked Read 3 Done Sent Done false
34756 9441 Acked Read 2 Done Sent Done false
34757 9442 Acked Read 8 Done Sent Done false
34758 9443 Acked Write 9 Done Sent Done false
34759 9444 Acked Write 1 Done Sent Done false
34760 9445 Acked Write 3 Done Sent Done false
34761 9446 Acked Write 9 Done Sent Done false
34762 9447 Acked Flush 0 Done Sent Done false
34763 9448 Acked Read 4 Done Sent Done false
34764 9449 Acked Read 5 Done Sent Done false
34765 9450 Acked Read 8 Done Sent Done false
34766 9451 Acked Flush 0 Done Sent Done false
34767 9452 Acked Read 2 Done Sent Done false
34768 STATES DS:0 DS:1 DS:2 TOTAL
34769 Sent 0 301 0 301
34770 Done 301 0 301 602
34771 Skipped 0 0 0 0
34772 Error 0 0 0 0
34773 Last Flush: Some(JobId(9451)) Some(JobId(9151)) Some(JobId(9451))
34774 Downstairs last five completed: 9151 9150 9149 9148 9147
34775 Upstairs last five completed: 9452 9451 9450 9449 9448
34776 CLIENT: Up:0 ds:301 act:2
347772025-03-17T22:26:42.455ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
347782025-03-17T22:26:42.951ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
347792025-03-17T22:26:42.951ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:62480) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
347802025-03-17T22:26:42.952ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
347812025-03-17T22:26:42.953ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
347822025-03-17T22:26:42.953ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
347832025-03-17T22:26:42.953ZINFOcrucible: send last flush ID to this DS: Some(JobId(9151)) = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
347842025-03-17T22:26:42.953ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
347852025-03-17T22:26:42.953ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(9151)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
347862025-03-17T22:26:42.953ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(9151)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
34787 ----------------------------------------------------------------
34788 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
34789 Downstairs last five completed: 9453 9452 9451 9450 9449
34790 Upstairs last five completed: 9453 9452 9451 9450 9449
34791 CLIENT: Up:0 ds:0 act:3
34792 Replay: All jobs finished
34793 Main task has work to do, go find it
34794 got dsc Stop(2)
34795 stop 2
34796 Mar 17 22:26:45.210 INFO request completed, latency_us: 326, response_code: 204, uri: /stop/cid/2, method: POST[8830] Got stop action so:false kr:false
34797 , req_id: dbb8c9ca-f129-44f8-af64-7e9b8da3af1a, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
347982025-03-17T22:26:45.210ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
34799 Mar 17 22:26:45.210{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:26:45.210947021Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
34800 INFO request completed, latency_us: 110, response_code:{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:26:45.210984257Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
34801 200, uri: /state/cid/2, method: GET, req_id: 71ba222d-0626-4509-a39a-85727472fdaa, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
348022025-03-17T22:26:45.211ZINFOcrucible: client task is exiting = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
34803 [8830] Exited with: ExitStatus(unix_wait_status(9))
34804 [8830][2] reports Exit
34805 Mar 17 22:26:49.213 INFO request completed, latency_us: 227, response_code: 200, uri: /state/cid/2, method: GET, req_id: f7035ce5-6771-4ed1-8066-7f2a6298a77c, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
34806 001/300 Read block 2731 len 16384
34807 002/300 Flush
34808 003/300 Write block 55 len 12288 data: 5 6 6
34809 004/300 Flush
34810 005/300 Read block 675 len 20480
34811 006/300 Read block 3876 len 20480
34812 007/300 Write block 1407 len 12288 data: 4 3 3
34813 008/300 Read block 3166 len 36864
34814 009/300 Read block 4367 len 28672
34815 010/300 Read block 3104 len 16384
34816 011/300 Read block 768 len 4096
34817 012/300 Write block 1942 len 16384 data: 3 5 4 4
34818 013/300 Read block 502 len 4096
34819 014/300 Read block 1375 len 40960
34820 015/300 Read block 2999 len 20480
34821 016/300 Read block 2200 len 32768
34822 017/300 Write block 2071 len 40960 data: 6 3 3 3 3 2 3 3 6 6
34823 018/300 Read block 423 len 16384
34824 019/300 Read block 1912 len 24576
34825 020/300 Write block 736 len 20480 data: 6 7 8 8 12
34826 021/300 Flush
34827 022/300 Read block 1041 len 12288
34828 023/300 Read block 3525 len 12288
34829 024/300 Write block 775 len 8192 data: 5 4
34830 025/300 Read block 2590 len 28672
34831 026/300 Read block 790 len 40960
34832 027/300 Read block 4570 len 32768
34833 028/300 Read block 1433 len 24576
34834 029/300 Read block 4849 len 28672
34835 030/300 Flush
34836 031/300 Write block 2147 len 16384 data: 7 6 4 3
34837 032/300 Write block 2216 len 12288 data: 2 4 4
34838 033/300 Flush
34839 034/300 Read block 972 len 20480
34840 035/300 Read block 4273 len 24576
34841 036/300 Write block 1863 len 20480 data: 8 8 6 8 7
34842 037/300 Write block 3438 len 12288 data: 10 9 10
34843 038/300 Read block 3876 len 24576
34844 039/300 Flush
34845 040/300 Write block 136 len 36864 data: 3 4 4 4 4 4 4 5 3
34846 041/300 Read block 4395 len 40960
34847 042/300 Write block 4002 len 4096 data: 6
34848 043/300 Write block 1903 len 4096 data: 6
34849 044/300 Read block 2342 len 12288
34850 045/300 Read block 3143 len 28672
34851 046/300 Flush
34852 047/300 Read block 3369 len 32768
34853 048/300 Read block 3018 len 40960
34854 049/300 Read block 4520 len 12288
34855 050/300 Read block 2391 len 12288
34856 051/300 Read block 4571 len 40960
34857 052/300 Write block 797 len 16384 data: 9 5 4 3
34858 053/300 Read block 3568 len 36864
34859 054/300 Write block 2517 len 20480 data: 6 6 8 6 7
34860 055/300 Write block 4905 len 24576 data: 8 8 8 5 6 6
34861 056/300 Flush
34862 057/300 Read block 4505 len 16384
34863 058/300 Read block 2035 len 20480
34864 059/300 Read block 4601 len 20480
34865 060/300 Write block 301 len 12288 data: 7 7 5
34866 061/300 Read block 4171 len 8192
34867 062/300 Read block 3250 len 4096
34868 063/300 Write block 423 len 36864 data: 9 8 7 5 4 3 3 4 4
34869 064/300 Read block 3423 len 20480
34870 065/300 Read block 478 len 36864
34871 066/300 Write block 1452 len 16384 data: 5 4 5 5
34872 067/300 Write block 4612 len 32768 data: 5 5 4 4 3 5 4 3
34873 068/300 Flush
34874 069/300 Write block 2027 len 20480 data: 6 7 5 4 4
34875 070/300 Read block 4474 len 24576
34876 071/300 Write block 3327 len 12288 data: 7 5 4
34877 072/300 Read block 4739 len 24576
34878 073/300 Read block 3237 len 32768
34879 074/300 Read block 658 len 20480
34880 075/300 Read block 1536 len 32768
34881 076/300 Read block 694 len 4096
34882 077/300 Write block 1698 len 8192 data: 6 5
34883 078/300 Read block 2966 len 16384
34884 079/300 Flush
34885 080/300 Read block 1544 len 4096
34886 081/300 Read block 2681 len 40960
34887 082/300 Read block 2670 len 4096
34888 083/300 Write block 1269 len 4096 data: 6
34889 084/300 Flush
34890 085/300 Read block 677 len 20480
34891 086/300 Read block 2588 len 28672
34892 087/300 Write block 4005 len 20480 data: 3 3 2 2 4
34893 088/300 Write block 1188 len 4096 data: 4
34894 089/300 Write block 4461 len 24576 data: 4 4 4 4 4 3
34895 090/300 Read block 4283 len 20480
34896 091/300 Write block 3861 len 40960 data: 6 5 5 5 4 4 3 3 3 3
34897 092/300 Flush
34898 093/300 Write block 3573 len 40960 data: 4 6 8 8 7 9 8 7 6 5
34899 094/300 Write block 3750 len 4096 data: 8
34900 095/300 Read block 2807 len 28672
34901 096/300 Read block 4633 len 12288
34902 097/300 Flush
34903 098/300 Read block 4640 len 4096
34904 099/300 Read block 4897 len 40960
34905 100/300 Read block 1118 len 40960
34906 101/300 Read block 728 len 32768
34907 102/300 Write block 3605 len 20480 data: 7 5 5 4 5
34908 103/300 Read block 2808 len 36864
34909 104/300 Write block 846 len 40960 data: 6 7 6 7 7 6 8 7 7 6
34910 105/300 Read block 599 len 4096
34911 106/300 Read block 309 len 4096
34912 107/300 Write block 2197 len 40960 data: 7 5 3 4 4 6 8 6 6 5
34913 108/300 Read block 3579 len 32768
34914 109/300 Read block 4500 len 8192
34915 110/300 Flush
34916 111/300 Write block 3714 len 24576 data: 5 5 6 7 8 7
34917 112/300 Write block 4951 len 24576 data: 3 3 3 3 5 5
34918 113/300 Read block 2602 len 16384
34919 114/300 Write block 5 len 32768 data: 6 5 4 4 4 5 4 3
34920 115/300 Write block 4487 len 12288 data: 5 5 6
34921 116/300 Write block 3331 len 24576 data: 3 3 2 4 3 4
34922 117/300 Read block 492 len 40960
34923 118/300 Read block 4615 len 24576
34924 119/300 Write block 1998 len 8192 data: 2 2
34925 120/300 Write block 4398 len 20480 data: 3 2 2 2 4
34926 121/300 Read block 2140 len 8192
34927 122/300 Read block 757 len 12288
34928 123/300 Write block 2218 len 32768 data: 5 5 5 7 7 7 8 8
34929 124/300 Read block 2458 len 4096
34930 125/300 Write block 4935 len 12288 data: 8 8 8
34931 126/300 Write block 4047 len 8192 data: 4 4
34932 127/300 Read block 3843 len 24576
34933 128/300 Write block 4297 len 16384 data: 4 3 5 6
34934 129/300 Flush
34935 130/300 Write block 495 len 4096 data: 7
34936 131/300 Read block 526 len 8192
34937 132/300 Write block 3270 len 20480 data: 7 6 7 7 7
34938 133/300 Flush
34939 134/300 Read block 616 len 32768
34940 135/300 Read block 3637 len 20480
34941 136/300 Write block 4111 len 36864 data: 4 3 3 3 4 6 7 6 6
34942 137/300 Read block 4579 len 28672
34943 138/300 Write block 4505 len 4096 data: 5
34944 139/300 Read block 1237 len 8192
34945 140/300 Flush
34946 141/300 Flush
34947 142/300 Write block 4961 len 32768 data: 7 9 10 7 7 7 7 5
34948 143/300 Read block 150 len 4096
34949 144/300 Write block 215 len 16384 data: 4 5 6 4
34950 145/300 Write block 1850 len 12288 data: 3 4 4
34951 146/300 Read block 1697 len 8192
34952 147/300 Flush
34953 148/300 Flush
34954 149/300 Flush
34955 150/300 Flush
34956 151/300 Read block 2178 len 36864
34957 152/300 Read block 1058 len 8192
34958 153/300 Write block 1322 len 28672 data: 5 5 4 5 7 9 7
34959 154/300 Write block 1141 len 32768 data: 5 4 4 4 4 3 3 5
34960 155/300 Read block 1238 len 36864
34961 156/300 Read block 2441 len 32768
34962 157/300 Read block 1500 len 8192
34963 158/300 Flush
34964 159/300 Read block 3150 len 16384
34965 160/300 Write block 815 len 32768 data: 6 6 6 7 8 8 7 6
34966 161/300 Write block 4012 len 20480 data: 5 6 6 6 6
34967 162/300 Read block 2633 len 16384
34968 163/300 Read block 2824 len 36864
34969 164/300 Read block 2940 len 20480
34970 165/300 Flush
34971 166/300 Read block 654 len 32768
34972 167/300 Read block 343 len 8192
34973 168/300 Read block 3802 len 8192
34974 169/300 Write block 1233 len 28672 data: 8 9 6 6 5 5 6
34975 170/300 Read block 3170 len 28672
34976 171/300 Flush
34977 172/300 Read block 4972 len 20480
34978 173/300 Read block 1213 len 20480
34979 174/300 Read block 1587 len 28672
34980 175/300 Read block 1918 len 16384
34981 176/300 Write block 449 len 4096 data: 9
34982 177/300 Flush
34983 178/300 Write block 277 len 12288 data: 5 5 5
34984 179/300 Read block 4956 len 4096
34985 180/300 Write block 3836 len 20480 data: 6 5 4 4 4
34986 181/300 Write block 4203 len 28672 data: 6 7 5 5 5 8 8
34987 182/300 Read block 4577 len 24576
34988 183/300 Write block 2694 len 12288 data: 3 5 6
34989 184/300 Read block 128 len 12288
34990 185/300 Read block 2868 len 4096
34991 186/300 Read block 1537 len 32768
34992 187/300 Write block 638 len 4096 data: 5
34993 188/300 Write block 4122 len 4096 data: 6
34994 189/300 Read block 1490 len 28672
34995 190/300 Flush
34996 191/300 Read block 4806 len 16384
34997 192/300 Write block 1371 len 36864 data: 3 4 4 6 5 4 4 4 5
34998 193/300 Read block 4902 len 40960
34999 194/300 Read block 2830 len 40960
35000 195/300 Read block 3044 len 16384
35001 196/300 Read block 4712 len 36864
35002 197/300 Write block 4188 len 4096 data: 4
35003 198/300 Write block 3508 len 20480 data: 7 6 7 6 4
35004 199/300 Read block 3557 len 28672
35005 200/300 Write block 4287 len 28672 data: 5 6 6 7 8 6 5
35006 201/300 Write block 4350 len 12288 data: 6 6 5
35007 202/300 Write block 2307 len 32768 data: 6 5 5 6 7 7 10 11
35008 203/300 Write block 2906 len 16384 data: 6 6 8 7
35009 204/300 Read block 3836 len 32768
35010 205/300 Flush
35011 206/300 Write block 3568 len 20480 data: 5 4 5 4 4
35012 207/300 Read block 816 len 20480
35013 208/300 Flush
35014 209/300 Write block 859 len 40960 data: 7 7 5 3 3 3 3 4 5 6
35015 210/300 Write block 699 len 36864 data: 5 6 7 7 8 7 7 5 6
35016 211/300 Write block 719 len 24576 data: 4 3 2 2 4 3
35017 212/300 Flush
35018 213/300 Read block 3506 len 20480
35019 214/300 Read block 1783 len 32768
35020 215/300 Write block 187 len 20480 data: 6 5 5 5 4
35021 216/300 Read block 700 len 32768
35022 217/300 Write block 806 len 36864 data: 4 3 4 5 4 5 5 5 6
35023 218/300 Write block 610 len 20480 data: 11 8 9 10 10
35024 219/300 Read block 643 len 16384
35025 220/300 Write block 546 len 32768 data: 3 3 2 3 3 4 3 5
35026 221/300 Write block 612 len 16384 data: 10 11 11 8
35027 222/300 Write block 555 len 8192 data: 5 2
35028 223/300 Read block 1162 len 8192
35029 224/300 Flush
35030 225/300 Write block 3159 len 28672 data: 3 3 3 3 2 2 4
35031 226/300 Read block 1303 len 36864
35032 227/300 Write block 2511 len 20480 data: 4 4 5 5 5
35033 228/300 Read block 3249 len 16384
35034 229/300 Write block 1539 len 16384 data: 4 4 5 4
35035 230/300 Read block 3271 len 40960
35036 231/300 Read block 1767 len 20480
35037 232/300 Write block 2914 len 24576 data: 7 6 6 6 6 6
35038 233/300 Write block 2010 len 12288 data: 3 5 6
35039 234/300 Write block 3255 len 8192 data: 6 8
35040 235/300 Write block 1684 len 24576 data: 8 7 6 6 7 5
35041 236/300 Read block 115 len 40960
35042 237/300 Flush
35043 238/300 Read block 931 len 40960
35044 239/300 Flush
35045 240/300 Read block 2570 len 40960
35046 241/300 Read block 4395 len 32768
35047 242/300 Read block 913 len 12288
35048 243/300 Read block 3915 len 4096
35049 244/300 Write block 3484 len 40960 data: 6 4 3 4 3 6 7 7 6 4
35050 245/300 Read block 266 len 20480
35051 246/300 Flush
35052 247/300 Write block 1604 len 12288 data: 8 8 8
35053 248/300 Write block 2990 len 4096 data: 5
35054 249/300 Write block 3413 len 32768 data: 7 7 5 4 4 6 5 6
35055 250/300 Read block 3922 len 24576
35056 251/300 Write block 715 len 40960 data: 5 5 5 4 5 4 3 3 5 4
35057 252/300 Flush
35058 253/300 Write block 1348 len 24576 data: 7 6 5 4 5 5
35059 254/300 Write block 4604 len 24576 data: 3 3 3 3 4 5
35060 255/300 Write block 282 len 28672 data: 6 5 5 4 4 5 5
35061 256/300 Flush
35062 257/300 Flush
35063 258/300 Read block 384 len 20480
35064 259/300 Read block 4963 len 24576
35065 260/300 Read block 4129 len 8192
35066 261/300 Read block 4902 len 36864
35067 262/300 Write block 933 len 36864 data: 6 7 7 6 5 3 3 3 4
35068 263/300 Read block 2636 len 28672
35069 264/300 Read block 2583 len 4096
35070 265/300 Write block 2799 len 12288 data: 5 5 5
35071 266/300 Write block 1052 len 12288 data: 6 10 10
35072 267/300 Read block 177 len 40960
35073 268/300 Read block 4037 len 16384
35074 269/300 Write block 2533 len 4096 data: 9
35075 270/300 Read block 2900 len 4096
35076 271/300 Read block 223 len 32768
35077 272/300 Write block 1877 len 32768 data: 8 9 11 9 8 5 6 6
35078 273/300 Read block 3724 len 36864
35079 274/300 Write block 1023 len 16384 data: 3 3 2 3
35080 275/300 Write block 4167 len 4096 data: 10
35081 276/300 Read block 2750 len 8192
35082 277/300 Write block 2010 len 24576 data: 4 6 7 6 8 9
35083 278/300 Write block 1370 len 28672 data: 2 4 5 5 7 6 5
35084 279/300 Read block 4753 len 40960
35085 280/300 Read block 3480 len 16384
35086 281/300 Write block 536 len 12288 data: 4 5 5
35087 282/300 Read block 3062 len 8192
35088 283/300 Read block 828 len 4096
35089 284/300 Read block 4245 len 20480
35090 285/300 Flush
35091 286/300 Read block 4207 len 8192
35092 287/300 Write block 3715 len 20480 data: 6 7 8 9 8
35093 288/300 Write block 4565 len 8192 data: 7 5
35094 289/300 Read block 4517 len 8192
35095 290/300 Write block 3006 len 4096 data: 5
35096 291/300 Write block 3637 len 32768 data: 8 8 8 8 6 5 5 5
35097 292/300 Read block 191 len 16384
35098 293/300 Flush
35099 294/300 Read block 4250 len 40960
35100 295/300 Write block 1498 len 24576 data: 3 5 6 5 4 4
35101 296/300 Read block 912 len 4096
35102 297/300 Read block 3333 len 28672
35103 298/300 Write block 1485 len 16384 data: 5 3 2 4
35104 299/300 Write block 3221 len 12288 data: 3 3 3
35105 300/300 Read block 2572 len 28672
35106 301/300 Flush
35107 Main task has work to do, go find it
35108 got dsc Start(2)
35109 start 2
35110 Mar 17 22:26:49.453 INFO request completed, latency_us: 209, response_code: 204, uri: /start/cid/2, method: POST, req_id: 8f462162-061b-4410-9d91-0f08dd1e4a1a, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
35111 Replay: started 2, returned:Ok(())
35112 ----------------------------------------------------------------
35113 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
35114 States: Active Active Offline
35115 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
35116 9454 Acked Read 4 Done Done Sent false
35117 9455 Acked Flush 0 Done Done Sent false
35118 9456 Acked Write 3 Done Done Sent false
35119 9457 Acked Flush 0 Done Done Sent false
35120 9458 Acked Read 5 Done Done Sent false
35121 9459 Acked Read 5 Done Done Sent false
35122 9460 Acked Write 3 Done Done Sent false
35123 9461 Acked Read 9 Done Done Sent false
35124 9462 Acked Read 7 Done Done Sent false
35125 9463 Acked Read 4 Done Done Sent false
35126 9464 Acked Read 1 Done Done Sent false
35127 9465 Acked Write 4 Done Done Sent false
35128 9466 Acked Read 1 Done Done Sent false
35129 9467 Acked Read 10 Done Done Sent false
35130 9468 Acked Read 5 Done Done Sent false
35131 9469 Acked Read 8 Done Done Sent false
35132 9470 Acked Write 10 Done Done Sent false
35133 9471 Acked Read 4 Done Done Sent false
35134 9472 Acked Read 6 Done Done Sent false
35135 9473 Acked Write 5 Done Done Sent false
35136 9474 Acked Flush 0 Done Done Sent false
35137 9475 Acked Read 3 Done Done Sent false
35138 9476 Acked Read 3 Done Done Sent false
35139 9477 Acked Write 2 Done Done Sent false
35140 9478 Acked Read 7 Done Done Sent false
35141 9479 Acked Read 10 Done Done Sent false
35142 9480 Acked Read 8 Done Done Sent false
35143 9481 Acked Read 6 Done Done Sent false
35144 9482 Acked Read 7 Done Done Sent false
35145 9483 Acked Flush 0 Done Done Sent false
35146 9484 Acked Write 4 Done Done Sent false
35147 9485 Acked Write 3 Done Done Sent false
35148 9486 Acked Flush 0 Done Done Sent false
35149 9487 Acked Read 5 Done Done Sent false
35150 9488 Acked Read 6 Done Done Sent false
35151 9489 Acked Write 5 Done Done Sent false
35152 9490 Acked Write 3 Done Done Sent false
35153 9491 Acked Read 6 Done Done Sent false
35154 9492 Acked Flush 0 Done Done Sent false
35155 9493 Acked Write 9 Done Done Sent false
35156 9494 Acked Read 10 Done Done Sent false
35157 9495 Acked Write 1 Done Done Sent false
35158 9496 Acked Write 1 Done Done Sent false
35159 9497 Acked Read 3 Done Done Sent false
35160 9498 Acked Read 7 Done Done Sent false
35161 9499 Acked Flush 0 Done Done Sent false
35162 9500 Acked Read 8 Done Done Sent false
35163 9501 Acked Read 10 Done Done Sent false
35164 9502 Acked Read 3 Done Done Sent false
35165 9503 Acked Read 3 Done Done Sent false
35166 9504 Acked Read 10 Done Done Sent false
35167 9505 Acked Write 4 Done Done Sent false
35168 9506 Acked Read 9 Done Done Sent false
35169 9507 Acked Write 5 Done Done Sent false
35170 9508 Acked Write 6 Done Done Sent false
35171 9509 Acked Flush 0 Done Done Sent false
35172 9510 Acked Read 4 Done Done Sent false
35173 9511 Acked Read 5 Done Done Sent false
35174 9512 Acked Read 5 Done Done Sent false
35175 9513 Acked Write 3 Done Done Sent false
35176 9514 Acked Read 2 Done Done Sent false
35177 9515 Acked Read 1 Done Done Sent false
35178 9516 Acked Write 9 Done Done Sent false
35179 9517 Acked Read 5 Done Done Sent false
35180 9518 Acked Read 9 Done Done Sent false
35181 9519 Acked Write 4 Done Done Sent false
35182 9520 Acked Write 8 Done Done Sent false
35183 9521 Acked Flush 0 Done Done Sent false
35184 9522 Acked Write 5 Done Done Sent false
35185 9523 Acked Read 6 Done Done Sent false
35186 9524 Acked Write 3 Done Done Sent false
35187 9525 Acked Read 6 Done Done Sent false
35188 9526 Acked Read 8 Done Done Sent false
35189 9527 Acked Read 5 Done Done Sent false
35190 9528 Acked Read 8 Done Done Sent false
35191 9529 Acked Read 1 Done Done Sent false
35192 9530 Acked Write 2 Done Done Sent false
35193 9531 Acked Read 4 Done Done Sent false
35194 9532 Acked Flush 0 Done Done Sent false
35195 9533 Acked Read 1 Done Done Sent false
35196 9534 Acked Read 10 Done Done Sent false
35197 9535 Acked Read 1 Done Done Sent false
35198 9536 Acked Write 1 Done Done Sent false
35199 9537 Acked Flush 0 Done Done Sent false
35200 9538 Acked Read 5 Done Done Sent false
35201 9539 Acked Read 7 Done Done Sent false
35202 9540 Acked Write 5 Done Done Sent false
35203 9541 Acked Write 1 Done Done Sent false
35204 9542 Acked Write 6 Done Done Sent false
35205 9543 Acked Read 5 Done Done Sent false
35206 9544 Acked Write 10 Done Done Sent false
35207 9545 Acked Flush 0 Done Done Sent false
35208 9546 Acked Write 10 Done Done Sent false
35209 9547 Acked Write 1 Done Done Sent false
35210 9548 Acked Read 7 Done Done Sent false
35211 9549 Acked Read 3 Done Done Sent false
35212 9550 Acked Flush 0 Done Done Sent false
35213 9551 Acked Read 1 Done Done Sent false
35214 9552 Acked Read 10 Done Done Sent false
35215 9553 Acked Read 10 Done Done Sent false
35216 9554 Acked Read 8 Done Done Sent false
35217 9555 Acked Write 5 Done Done Sent false
35218 9556 Acked Read 9 Done Done Sent false
35219 9557 Acked Write 10 Done Done Sent false
35220 9558 Acked Read 1 Done Done Sent false
35221 9559 Acked Read 1 Done Done Sent false
35222 9560 Acked Write 10 Done Done Sent false
35223 9561 Acked Read 8 Done Done Sent false
35224 9562 Acked Read 2 Done Done Sent false
35225 9563 Acked Flush 0 Done Done Sent false
35226 9564 Acked Write 6 Done Done Sent false
35227 9565 Acked Write 6 Done Done Sent false
35228 9566 Acked Read 4 Done Done Sent false
35229 9567 Acked Write 8 Done Done Sent false
35230 9568 Acked Write 3 Done Done Sent false
35231 9569 Acked Write 6 Done Done Sent false
35232 9570 Acked Read 10 Done Done Sent false
35233 9571 Acked Read 6 Done Done Sent false
35234 9572 Acked Write 2 Done Done Sent false
35235 9573 Acked Write 5 Done Done Sent false
35236 9574 Acked Read 2 Done Done Sent false
35237 9575 Acked Read 3 Done Done Sent false
35238 9576 Acked Write 8 Done Done Sent false
35239 9577 Acked Read 1 Done Done Sent false
35240 9578 Acked Write 3 Done Done Sent false
35241 9579 Acked Write 2 Done Done Sent false
35242 9580 Acked Read 6 Done Done Sent false
35243 9581 Acked Write 4 Done Done Sent false
35244 9582 Acked Flush 0 Done Done Sent false
35245 9583 Acked Write 1 Done Done Sent false
35246 9584 Acked Read 2 Done Done Sent false
35247 9585 Acked Write 5 Done Done Sent false
35248 9586 Acked Flush 0 Done Done Sent false
35249 9587 Acked Read 8 Done Done Sent false
35250 9588 Acked Read 5 Done Done Sent false
35251 9589 Acked Write 9 Done Done Sent false
35252 9590 Acked Read 7 Done Done Sent false
35253 9591 Acked Write 1 Done Done Sent false
35254 9592 Acked Read 2 Done Done Sent false
35255 9593 Acked Flush 0 Done Done Sent false
35256 9594 Acked Flush 0 Done Done Sent false
35257 9595 Acked Write 8 Done Done Sent false
35258 9596 Acked Read 1 Done Done Sent false
35259 9597 Acked Write 4 Done Done Sent false
35260 9598 Acked Write 3 Done Done Sent false
35261 9599 Acked Read 2 Done Done Sent false
35262 9600 Acked Flush 0 Done Done Sent false
35263 9601 Acked Flush 0 Done Done Sent false
35264 9602 Acked Flush 0 Done Done Sent false
35265 9603 Acked Flush 0 Done Done Sent false
35266 9604 Acked Read 9 Done Done Sent false
35267 9605 Acked Read 2 Done Done Sent false
35268 9606 Acked Write 7 Done Done Sent false
35269 9607 Acked Write 8 Done Done Sent false
35270 9608 Acked Read 9 Done Done Sent false
35271 9609 Acked Read 8 Done Done Sent false
35272 9610 Acked Read 2 Done Done Sent false
35273 9611 Acked Flush 0 Done Done Sent false
35274 9612 Acked Read 4 Done Done Sent false
35275 9613 Acked Write 8 Done Done Sent false
35276 9614 Acked Write 5 Done Done Sent false
35277 9615 Acked Read 4 Done Done Sent false
35278 9616 Acked Read 9 Done Done Sent false
35279 9617 Acked Read 5 Done Done Sent false
35280 9618 Acked Flush 0 Done Done Sent false
35281 9619 Acked Read 8 Done Done Sent false
35282 9620 Acked Read 2 Done Done Sent false
35283 9621 Acked Read 2 Done Done Sent false
35284 9622 Acked Write 7 Done Done Sent false
35285 9623 Acked Read 7 Done Done Sent false
35286 9624 Acked Flush 0 Done Done Sent false
35287 9625 Acked Read 5 Done Done Sent false
35288 9626 Acked Read 5 Done Done Sent false
35289 9627 Acked Read 7 Done Done Sent false
35290 9628 Acked Read 4 Done Done Sent false
35291 9629 Acked Write 1 Done Done Sent false
35292 9630 Acked Flush 0 Done Done Sent false
35293 9631 Acked Write 3 Done Done Sent false
35294 9632 Acked Read 1 Done Done Sent false
35295 9633 Acked Write 5 Done Done Sent false
35296 9634 Acked Write 7 Done Done Sent false
35297 9635 Acked Read 6 Done Done Sent false
35298 9636 Acked Write 3 Done Done Sent false
35299 9637 Acked Read 3 Done Done Sent false
35300 9638 Acked Read 1 Done Done Sent false
35301 9639 Acked Read 8 Done Done Sent false
35302 9640 Acked Write 1 Done Done Sent false
35303 9641 Acked Write 1 Done Done Sent false
35304 9642 Acked Read 7 Done Done Sent false
35305 9643 Acked Flush 0 Done Done Sent false
35306 9644 Acked Read 4 Done Done Sent false
35307 9645 Acked Write 9 Done Done Sent false
35308 9646 Acked Read 10 Done Done Sent false
35309 9647 Acked Read 10 Done Done Sent false
35310 9648 Acked Read 4 Done Done Sent false
35311 9649 Acked Read 9 Done Done Sent false
35312 9650 Acked Write 1 Done Done Sent false
35313 9651 Acked Write 5 Done Done Sent false
35314 9652 Acked Read 7 Done Done Sent false
35315 9653 Acked Write 7 Done Done Sent false
35316 9654 Acked Write 3 Done Done Sent false
35317 9655 Acked Write 8 Done Done Sent false
35318 9656 Acked Write 4 Done Done Sent false
35319 9657 Acked Read 8 Done Done Sent false
35320 9658 Acked Flush 0 Done Done Sent false
35321 9659 Acked Write 5 Done Done Sent false
35322 9660 Acked Read 5 Done Done Sent false
35323 9661 Acked Flush 0 Done Done Sent false
35324 9662 Acked Write 10 Done Done Sent false
35325 9663 Acked Write 9 Done Done Sent false
35326 9664 Acked Write 6 Done Done Sent false
35327 9665 Acked Flush 0 Done Done Sent false
35328 9666 Acked Read 5 Done Done Sent false
35329 9667 Acked Read 8 Done Done Sent false
35330 9668 Acked Write 5 Done Done Sent false
35331 9669 Acked Read 8 Done Done Sent false
35332 9670 Acked Write 9 Done Done Sent false
35333 9671 Acked Write 5 Done Done Sent false
35334 9672 Acked Read 4 Done Done Sent false
35335 9673 Acked Write 8 Done Done Sent false
35336 9674 Acked Write 4 Done Done Sent false
35337 9675 Acked Write 2 Done Done Sent false
35338 9676 Acked Read 2 Done Done Sent false
35339 9677 Acked Flush 0 Done Done Sent false
35340 9678 Acked Write 7 Done Done Sent false
35341 9679 Acked Read 9 Done Done Sent false
35342 9680 Acked Write 5 Done Done Sent false
35343 9681 Acked Read 4 Done Done Sent false
35344 9682 Acked Write 4 Done Done Sent false
35345 9683 Acked Read 10 Done Done Sent false
35346 9684 Acked Read 5 Done Done Sent false
35347 9685 Acked Write 6 Done Done Sent false
35348 9686 Acked Write 3 Done Done Sent false
35349 9687 Acked Write 2 Done Done Sent false
35350 9688 Acked Write 6 Done Done Sent false
35351 9689 Acked Read 10 Done Done Sent false
35352 9690 Acked Flush 0 Done Done Sent false
35353 9691 Acked Read 10 Done Done Sent false
35354 9692 Acked Flush 0 Done Done Sent false
35355 9693 Acked Read 10 Done Done Sent false
35356 9694 Acked Read 8 Done Done Sent false
35357 9695 Acked Read 3 Done Done Sent false
35358 9696 Acked Read 1 Done Done Sent false
35359 9697 Acked Write 10 Done Done Sent false
35360 9698 Acked Read 5 Done Done Sent false
35361 9699 Acked Flush 0 Done Done Sent false
35362 9700 Acked Write 3 Done Done Sent false
35363 9701 Acked Write 1 Done Done Sent false
35364 9702 Acked Write 8 Done Done Sent false
35365 9703 Acked Read 6 Done Done Sent false
35366 9704 Acked Write 10 Done Done Sent false
35367 9705 Acked Flush 0 Done Done Sent false
35368 9706 Acked Write 6 Done Done Sent false
35369 9707 Acked Write 6 Done Done Sent false
35370 9708 Acked Write 7 Done Done Sent false
35371 9709 Acked Flush 0 Done Done Sent false
35372 9710 Acked Flush 0 Done Done Sent false
35373 9711 Acked Read 5 Done Done Sent false
35374 9712 Acked Read 6 Done Done Sent false
35375 9713 Acked Read 2 Done Done Sent false
35376 9714 Acked Read 9 Done Done Sent false
35377 9715 Acked Write 9 Done Done Sent false
35378 9716 Acked Read 7 Done Done Sent false
35379 9717 Acked Read 1 Done Done Sent false
35380 9718 Acked Write 3 Done Done Sent false
35381 9719 Acked Write 3 Done Done Sent false
35382 9720 Acked Read 10 Done Done Sent false
35383 9721 Acked Read 4 Done Done Sent false
35384 9722 Acked Write 1 Done Done Sent false
35385 9723 Acked Read 1 Done Done Sent false
35386 9724 Acked Read 8 Done Done Sent false
35387 9725 Acked Write 8 Done Done Sent false
35388 9726 Acked Read 9 Done Done Sent false
35389 9727 Acked Write 4 Done Done Sent false
35390 9728 Acked Write 1 Done Done Sent false
35391 9729 Acked Read 2 Done Done Sent false
35392 9730 Acked Write 6 Done Done Sent false
35393 9731 Acked Write 7 Done Done Sent false
35394 9732 Acked Read 10 Done Done Sent false
35395 9733 Acked Read 4 Done Done Sent false
35396 9734 Acked Write 3 Done Done Sent false
35397 9735 Acked Read 2 Done Done Sent false
35398 9736 Acked Read 1 Done Done Sent false
35399 9737 Acked Read 5 Done Done Sent false
35400 9738 Acked Flush 0 Done Done Sent false
35401 9739 Acked Read 2 Done Done Sent false
35402 9740 Acked Write 5 Done Done Sent false
35403 9741 Acked Write 2 Done Done Sent false
35404 9742 Acked Read 2 Done Done Sent false
35405 9743 Acked Write 1 Done Done Sent false
35406 9744 Acked Write 8 Done Done Sent false
35407 9745 Acked Read 4 Done Done Sent false
35408 9746 Acked Flush 0 Done Done Sent false
35409 9747 Acked Read 10 Done Done Sent false
35410 9748 Acked Write 6 Done Done Sent false
35411 9749 Acked Read 1 Done Done Sent false
35412 9750 Acked Read 7 Done Done Sent false
35413 9751 Acked Write 4 Done Done Sent false
35414 9752 Acked Write 3 Done Done Sent false
35415 9753 Acked Read 7 Done Done Sent false
35416 9754 Acked Flush 0 Done Done Sent false
35417 STATES DS:0 DS:1 DS:2 TOTAL
35418 Sent 0 0 301 301
35419 Done 301 301 0 602
35420 Skipped 0 0 0 0
35421 Error 0 0 0 0
35422 Last Flush: Some(JobId(9754)) Some(JobId(9754)) Some(JobId(9453))
35423 Downstairs last five completed: 9453 9452 9451 9450 9449
35424 Upstairs last five completed: 9754 9753 9752 9751 9750
35425 CLIENT: Up:0 ds:301 act:2
354262025-03-17T22:26:50.342ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
354272025-03-17T22:26:51.343ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
354282025-03-17T22:26:52.346ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
35429 [8830] got start action
35430 [8830] I am going to restart
35431 Starting downstairs at port 8830
35432 Make output file at "/tmp/dsc/downstairs-8830.txt"
35433 [8830][2] reports Starting
35434 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1240)
35435 [8830][2] reports Running
35436 {"msg":"flush check fired despite having jobs; resetting it","v":0,"name":"crucible","level":40----------------------------------------------------------------
35437 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
35438 States: Active Active Offline
35439 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
35440 9454 Acked Read 4 Done Done Sent false
35441 ,"time": 9455 Acked Flush 0" Done Done Sent2025-03-17T22:26:53.456115734Z" false
35442 ," 9456 Acked Write 3hostname" Done: Done" Sent false
35443 w-01JPK2XSVRF7343QRZWA5N22R2" 9457 Acked Flush 0, Done" Donepid Sent" false:
35444 1211 9458 Acked Read 5 Done Done Sent false,
35445 " 9459 Acked Read 5session_id" Done: Done" Sent false
35446 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf" 9460 Acked Write 3} Done
35447 Done Sent false
35448 9461 Acked Read 9 Done Done Sent false
35449 9462 Acked Read 7 Done Done Sent false
35450 9463 Acked Read 4 Done Done Sent false
35451 9464 Acked Read 1 Done Done Sent false
35452 9465 Acked Write 4 Done Done Sent false
35453 9466 Acked Read 1 Done Done Sent false
35454 9467 Acked Read 10 Done Done Sent false
35455 9468 Acked Read 5 Done Done Sent false
35456 9469 Acked Read 8 Done Done Sent false
35457 9470 Acked Write 10 Done Done Sent false
35458 9471 Acked Read 4 Done Done Sent false
35459 9472 Acked Read 6 Done Done Sent false
35460 9473 Acked Write 5 Done Done Sent false
35461 9474 Acked Flush 0 Done Done Sent false
35462 9475 Acked Read 3 Done Done Sent false
35463 9476 Acked Read 3 Done Done Sent false
35464 9477 Acked Write 2 Done Done Sent false
35465 9478 Acked Read 7 Done Done Sent false
35466 9479 Acked Read 10 Done Done Sent false
35467 9480 Acked Read 8 Done Done Sent false
35468 9481 Acked Read 6 Done Done Sent false
35469 9482 Acked Read 7 Done Done Sent false
35470 9483 Acked Flush 0 Done Done Sent false
35471 9484 Acked Write 4 Done Done Sent false
35472 9485 Acked Write 3 Done Done Sent false
35473 9486 Acked Flush 0 Done Done Sent false
35474 9487 Acked Read 5 Done Done Sent false
35475 9488 Acked Read 6 Done Done Sent false
35476 9489 Acked Write 5 Done Done Sent false
35477 9490 Acked Write 3 Done Done Sent false
35478 9491 Acked Read 6 Done Done Sent false
35479 9492 Acked Flush 0 Done Done Sent false
35480 9493 Acked Write 9 Done Done Sent false
35481 9494 Acked Read 10 Done Done Sent false
35482 9495 Acked Write 1 Done Done Sent false
35483 9496 Acked Write 1 Done Done Sent false
35484 9497 Acked Read 3 Done Done Sent false
35485 9498 Acked Read 7 Done Done Sent false
35486 9499 Acked Flush 0 Done Done Sent false
35487 9500 Acked Read 8 Done Done Sent false
35488 9501 Acked Read 10 Done Done Sent false
35489 9502 Acked Read 3 Done Done Sent false
35490 9503 Acked Read 3 Done Done Sent false
35491 9504 Acked Read 10 Done Done Sent false
35492 9505 Acked Write 4 Done Done Sent false
35493 9506 Acked Read 9 Done Done Sent false
35494 9507 Acked Write 5 Done Done Sent false
35495 9508 Acked Write 6 Done Done Sent false
35496 9509 Acked Flush 0 Done Done Sent false
35497 9510 Acked Read 4 Done Done Sent false
35498 9511 Acked Read 5 Done Done Sent false
35499 9512 Acked Read 5 Done Done Sent false
35500 9513 Acked Write 3 Done Done Sent false
35501 9514 Acked Read 2 Done Done Sent false
35502 9515 Acked Read 1 Done Done Sent false
35503 9516 Acked Write 9 Done Done Sent false
35504 9517 Acked Read 5 Done Done Sent false
35505 9518 Acked Read 9 Done Done Sent false
35506 9519 Acked Write 4 Done Done Sent false
35507 9520 Acked Write 8 Done Done Sent false
35508 9521 Acked Flush 0 Done Done Sent false
35509 9522 Acked Write 5 Done Done Sent false
35510 9523 Acked Read 6 Done Done Sent false
35511 9524 Acked Write 3 Done Done Sent false
35512 9525 Acked Read 6 Done Done Sent false
35513 9526 Acked Read 8 Done Done Sent false
35514 9527 Acked Read 5 Done Done Sent false
35515 9528 Acked Read 8 Done Done Sent false
35516 9529 Acked Read 1 Done Done Sent false
35517 9530 Acked Write 2 Done Done Sent false
35518 9531 Acked Read 4 Done Done Sent false
35519 9532 Acked Flush 0 Done Done Sent false
35520 9533 Acked Read 1 Done Done Sent false
35521 9534 Acked Read 10 Done Done Sent false
35522 9535 Acked Read 1 Done Done Sent false
35523 9536 Acked Write 1 Done Done Sent false
35524 9537 Acked Flush 0 Done Done Sent false
35525 9538 Acked Read 5 Done Done Sent false
35526 9539 Acked Read 7 Done Done Sent false
35527 9540 Acked Write 5 Done Done Sent false
35528 9541 Acked Write 1 Done Done Sent false
35529 9542 Acked Write 6 Done Done Sent false
35530 9543 Acked Read 5 Done Done Sent false
35531 9544 Acked Write 10 Done Done Sent false
35532 9545 Acked Flush 0 Done Done Sent false
35533 9546 Acked Write 10 Done Done Sent false
35534 9547 Acked Write 1 Done Done Sent false
35535 9548 Acked Read 7 Done Done Sent false
35536 9549 Acked Read 3 Done Done Sent false
35537 9550 Acked Flush 0 Done Done Sent false
35538 9551 Acked Read 1 Done Done Sent false
35539 9552 Acked Read 10 Done Done Sent false
35540 9553 Acked Read 10 Done Done Sent false
35541 9554 Acked Read 8 Done Done Sent false
35542 9555 Acked Write 5 Done Done Sent false
35543 9556 Acked Read 9 Done Done Sent false
35544 9557 Acked Write 10 Done Done Sent false
35545 9558 Acked Read 1 Done Done Sent false
35546 9559 Acked Read 1 Done Done Sent false
35547 9560 Acked Write 10 Done Done Sent false
35548 9561 Acked Read 8 Done Done Sent false
35549 9562 Acked Read 2 Done Done Sent false
35550 9563 Acked Flush 0 Done Done Sent false
35551 9564 Acked Write 6 Done Done Sent false
35552 9565 Acked Write 6 Done Done Sent false
35553 9566 Acked Read 4 Done Done Sent false
35554 9567 Acked Write 8 Done Done Sent false
35555 9568 Acked Write 3 Done Done Sent false
35556 9569 Acked Write 6 Done Done Sent false
35557 9570 Acked Read 10 Done Done Sent false
35558 9571 Acked Read 6 Done Done Sent false
35559 9572 Acked Write 2 Done Done Sent false
35560 9573 Acked Write 5 Done Done Sent false
35561 9574 Acked Read 2 Done Done Sent false
35562 9575 Acked Read 3 Done Done Sent false
35563 9576 Acked Write 8 Done Done Sent false
35564 9577 Acked Read 1 Done Done Sent false
35565 9578 Acked Write 3 Done Done Sent false
35566 9579 Acked Write 2 Done Done Sent false
35567 9580 Acked Read 6 Done Done Sent false
35568 9581 Acked Write 4 Done Done Sent false
35569 9582 Acked Flush 0 Done Done Sent false
35570 9583 Acked Write 1 Done Done Sent false
35571 9584 Acked Read 2 Done Done Sent false
35572 9585 Acked Write 5 Done Done Sent false
35573 9586 Acked Flush 0 Done Done Sent false
35574 9587 Acked Read 8 Done Done Sent false
35575 9588 Acked Read 5 Done Done Sent false
35576 9589 Acked Write 9 Done Done Sent false
35577 9590 Acked Read 7 Done Done Sent false
35578 9591 Acked Write 1 Done Done Sent false
35579 9592 Acked Read 2 Done Done Sent false
35580 9593 Acked Flush 0 Done Done Sent false
35581 9594 Acked Flush 0 Done Done Sent false
35582 9595 Acked Write 8 Done Done Sent false
35583 9596 Acked Read 1 Done Done Sent false
35584 9597 Acked Write 4 Done Done Sent false
35585 9598 Acked Write 3 Done Done Sent false
35586 9599 Acked Read 2 Done Done Sent false
35587 9600 Acked Flush 0 Done Done Sent false
35588 9601 Acked Flush 0 Done Done Sent false
35589 9602 Acked Flush 0 Done Done Sent false
35590 9603 Acked Flush 0 Done Done Sent false
35591 9604 Acked Read 9 Done Done Sent false
35592 9605 Acked Read 2 Done Done Sent false
35593 9606 Acked Write 7 Done Done Sent false
35594 9607 Acked Write 8 Done Done Sent false
35595 9608 Acked Read 9 Done Done Sent false
35596 9609 Acked Read 8 Done Done Sent false
35597 9610 Acked Read 2 Done Done Sent false
35598 9611 Acked Flush 0 Done Done Sent false
35599 9612 Acked Read 4 Done Done Sent false
35600 9613 Acked Write 8 Done Done Sent false
35601 9614 Acked Write 5 Done Done Sent false
35602 9615 Acked Read 4 Done Done Sent false
35603 9616 Acked Read 9 Done Done Sent false
35604 9617 Acked Read 5 Done Done Sent false
35605 9618 Acked Flush 0 Done Done Sent false
35606 9619 Acked Read 8 Done Done Sent false
35607 9620 Acked Read 2 Done Done Sent false
35608 9621 Acked Read 2 Done Done Sent false
35609 9622 Acked Write 7 Done Done Sent false
35610 9623 Acked Read 7 Done Done Sent false
35611 9624 Acked Flush 0 Done Done Sent false
35612 9625 Acked Read 5 Done Done Sent false
35613 9626 Acked Read 5 Done Done Sent false
35614 9627 Acked Read 7 Done Done Sent false
35615 9628 Acked Read 4 Done Done Sent false
35616 9629 Acked Write 1 Done Done Sent false
35617 9630 Acked Flush 0 Done Done Sent false
35618 9631 Acked Write 3 Done Done Sent false
35619 9632 Acked Read 1 Done Done Sent false
35620 9633 Acked Write 5 Done Done Sent false
35621 9634 Acked Write 7 Done Done Sent false
35622 9635 Acked Read 6 Done Done Sent false
35623 9636 Acked Write 3 Done Done Sent false
35624 9637 Acked Read 3 Done Done Sent false
35625 9638 Acked Read 1 Done Done Sent false
35626 9639 Acked Read 8 Done Done Sent false
35627 9640 Acked Write 1 Done Done Sent false
35628 9641 Acked Write 1 Done Done Sent false
35629 9642 Acked Read 7 Done Done Sent false
35630 9643 Acked Flush 0 Done Done Sent false
35631 9644 Acked Read 4 Done Done Sent false
35632 9645 Acked Write 9 Done Done Sent false
35633 9646 Acked Read 10 Done Done Sent false
35634 9647 Acked Read 10 Done Done Sent false
35635 9648 Acked Read 4 Done Done Sent false
35636 9649 Acked Read 9 Done Done Sent false
35637 9650 Acked Write 1 Done Done Sent false
35638 9651 Acked Write 5 Done Done Sent false
35639 9652 Acked Read 7 Done Done Sent false
35640 9653 Acked Write 7 Done Done Sent false
35641 9654 Acked Write 3 Done Done Sent false
35642 9655 Acked Write 8 Done Done Sent false
35643 9656 Acked Write 4 Done Done Sent false
35644 9657 Acked Read 8 Done Done Sent false
35645 9658 Acked Flush 0 Done Done Sent false
35646 9659 Acked Write 5 Done Done Sent false
35647 9660 Acked Read 5 Done Done Sent false
35648 9661 Acked Flush 0 Done Done Sent false
35649 9662 Acked Write 10 Done Done Sent false
35650 9663 Acked Write 9 Done Done Sent false
35651 9664 Acked Write 6 Done Done Sent false
35652 9665 Acked Flush 0 Done Done Sent false
35653 9666 Acked Read 5 Done Done Sent false
35654 9667 Acked Read 8 Done Done Sent false
35655 9668 Acked Write 5 Done Done Sent false
35656 9669 Acked Read 8 Done Done Sent false
35657 9670 Acked Write 9 Done Done Sent false
35658 9671 Acked Write 5 Done Done Sent false
35659 9672 Acked Read 4 Done Done Sent false
35660 9673 Acked Write 8 Done Done Sent false
35661 9674 Acked Write 4 Done Done Sent false
35662 9675 Acked Write 2 Done Done Sent false
35663 9676 Acked Read 2 Done Done Sent false
35664 9677 Acked Flush 0 Done Done Sent false
35665 9678 Acked Write 7 Done Done Sent false
35666 9679 Acked Read 9 Done Done Sent false
35667 9680 Acked Write 5 Done Done Sent false
35668 9681 Acked Read 4 Done Done Sent false
35669 9682 Acked Write 4 Done Done Sent false
35670 9683 Acked Read 10 Done Done Sent false
35671 9684 Acked Read 5 Done Done Sent false
35672 9685 Acked Write 6 Done Done Sent false
35673 9686 Acked Write 3 Done Done Sent false
35674 9687 Acked Write 2 Done Done Sent false
35675 9688 Acked Write 6 Done Done Sent false
35676 9689 Acked Read 10 Done Done Sent false
35677 9690 Acked Flush 0 Done Done Sent false
35678 9691 Acked Read 10 Done Done Sent false
35679 9692 Acked Flush 0 Done Done Sent false
35680 9693 Acked Read 10 Done Done Sent false
35681 9694 Acked Read 8 Done Done Sent false
35682 9695 Acked Read 3 Done Done Sent false
35683 9696 Acked Read 1 Done Done Sent false
35684 9697 Acked Write 10 Done Done Sent false
35685 9698 Acked Read 5 Done Done Sent false
35686 9699 Acked Flush 0 Done Done Sent false
35687 9700 Acked Write 3 Done Done Sent false
35688 9701 Acked Write 1 Done Done Sent false
35689 9702 Acked Write 8 Done Done Sent false
35690 9703 Acked Read 6 Done Done Sent false
35691 9704 Acked Write 10 Done Done Sent false
35692 9705 Acked Flush 0 Done Done Sent false
35693 9706 Acked Write 6 Done Done Sent false
35694 9707 Acked Write 6 Done Done Sent false
35695 9708 Acked Write 7 Done Done Sent false
35696 9709 Acked Flush 0 Done Done Sent false
35697 9710 Acked Flush 0 Done Done Sent false
35698 9711 Acked Read 5 Done Done Sent false
35699 9712 Acked Read 6 Done Done Sent false
35700 9713 Acked Read 2 Done Done Sent false
35701 9714 Acked Read 9 Done Done Sent false
35702 9715 Acked Write 9 Done Done Sent false
35703 9716 Acked Read 7 Done Done Sent false
35704 9717 Acked Read 1 Done Done Sent false
35705 9718 Acked Write 3 Done Done Sent false
35706 9719 Acked Write 3 Done Done Sent false
35707 9720 Acked Read 10 Done Done Sent false
35708 9721 Acked Read 4 Done Done Sent false
35709 9722 Acked Write 1 Done Done Sent false
35710 9723 Acked Read 1 Done Done Sent false
35711 9724 Acked Read 8 Done Done Sent false
35712 9725 Acked Write 8 Done Done Sent false
35713 9726 Acked Read 9 Done Done Sent false
35714 9727 Acked Write 4 Done Done Sent false
35715 9728 Acked Write 1 Done Done Sent false
35716 9729 Acked Read 2 Done Done Sent false
35717 9730 Acked Write 6 Done Done Sent false
35718 9731 Acked Write 7 Done Done Sent false
35719 9732 Acked Read 10 Done Done Sent false
35720 9733 Acked Read 4 Done Done Sent false
35721 9734 Acked Write 3 Done Done Sent false
35722 9735 Acked Read 2 Done Done Sent false
35723 9736 Acked Read 1 Done Done Sent false
35724 9737 Acked Read 5 Done Done Sent false
35725 9738 Acked Flush 0 Done Done Sent false
35726 9739 Acked Read 2 Done Done Sent false
35727 9740 Acked Write 5 Done Done Sent false
35728 9741 Acked Write 2 Done Done Sent false
35729 9742 Acked Read 2 Done Done Sent false
35730 9743 Acked Write 1 Done Done Sent false
35731 9744 Acked Write 8 Done Done Sent false
35732 9745 Acked Read 4 Done Done Sent false
35733 9746 Acked Flush 0 Done Done Sent false
35734 9747 Acked Read 10 Done Done Sent false
35735 9748 Acked Write 6 Done Done Sent false
35736 9749 Acked Read 1 Done Done Sent false
35737 9750 Acked Read 7 Done Done Sent false
35738 9751 Acked Write 4 Done Done Sent false
35739 9752 Acked Write 3 Done Done Sent false
35740 9753 Acked Read 7 Done Done Sent false
35741 9754 Acked Flush 0 Done Done Sent false
35742 STATES DS:0 DS:1 DS:2 TOTAL
35743 Sent 0 0 301 301
35744 Done 301 301 0 602
35745 Skipped 0 0 0 0
35746 Error 0 0 0 0
35747 Last Flush: Some(JobId(9754)) Some(JobId(9754)) Some(JobId(9453))
35748 Downstairs last five completed: 9453 9452 9451 9450 9449
35749 Upstairs last five completed: 9754 9753 9752 9751 9750
35750 CLIENT: Up:0 ds:301 act:2
357512025-03-17T22:26:54.347ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
357522025-03-17T22:26:55.212ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
357532025-03-17T22:26:55.212ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58861) = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
357542025-03-17T22:26:55.214ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
357552025-03-17T22:26:55.214ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
357562025-03-17T22:26:55.214ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
357572025-03-17T22:26:55.214ZINFOcrucible: send last flush ID to this DS: Some(JobId(9453)) = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
357582025-03-17T22:26:55.214ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
357592025-03-17T22:26:55.214ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(9453)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
357602025-03-17T22:26:55.215ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(9453)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
35761 ----------------------------------------------------------------
35762 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
35763 Downstairs last five completed: 9754 9753 9752 9751 9750
35764 Upstairs last five completed: 9754 9753 9752 9751 9750
35765 CLIENT: Up:0 ds:0 act:3
35766 Replay: All jobs finished
35767 Main task has work to do, go find it
35768 got dsc Stop(2)
35769 stop 2
35770 Mar 17 22:26:57.460[8830] Got stop action so:false kr:false
35771 INFO request completed, latency_us: 348, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 04e71394-c853-49ba-aa67-3f1eef39e0cd, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
35772 Mar 17 22:26:57.461 INFO request completed, {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:26:57.461144916Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
35773 latency_us: 137, response_code: 200, uri: /state/cid/2, method: GET, req_id: 38b1ef00-f2b2-4368-86bc-d8580dca3bb7, remote_addr{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:26:57.461215808Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"":"io task","client":"2","":"downstairs","session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
35774 : 127.0.0.1:58317, local_addr:{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:26:57.461269902Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
35775 127.0.0.1:9998
357762025-03-17T22:26:57.461ZINFOcrucible: client task is exiting = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
35777 [8830] Exited with: ExitStatus(unix_wait_status(9))
35778 [8830][2] reports Exit
35779 Mar 17 22:27:01.463 INFO request completed, latency_us: 212, response_code: 200, uri: /state/cid/2, method: GET, req_id: aa225e2d-f6d9-4dd0-b2b1-1123f246c422, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
35780 001/300 Read block 2547 len 32768
35781 002/300 Read block 116 len 24576
35782 003/300 Write block 2060 len 24576 data: 3 3 3 5 7 8
35783 004/300 Write block 3858 len 8192 data: 5 5
35784 005/300 Write block 2116 len 40960 data: 5 4 5 7 5 5 5 5 5 5
35785 006/300 Flush
35786 007/300 Read block 1996 len 36864
35787 008/300 Write block 473 len 40960 data: 5 5 7 7 7 6 6 5 5 6
35788 009/300 Read block 1610 len 24576
35789 010/300 Read block 4627 len 24576
35790 011/300 Write block 4946 len 36864 data: 6 5 4 4 4 4 4 4 4
35791 012/300 Read block 2890 len 32768
35792 013/300 Write block 1948 len 20480 data: 5 5 5 5 5
35793 014/300 Write block 3649 len 28672 data: 5 5 6 9 11 11 10
35794 015/300 Write block 2625 len 36864 data: 6 6 6 5 4 4 4 3 3
35795 016/300 Flush
35796 017/300 Read block 3867 len 24576
35797 018/300 Flush
35798 019/300 Write block 2399 len 16384 data: 8 10 9 8
35799 020/300 Read block 2500 len 12288
35800 021/300 Read block 1362 len 8192
35801 022/300 Read block 2836 len 16384
35802 023/300 Write block 1818 len 28672 data: 2 3 3 3 4 4 4
35803 024/300 Write block 2851 len 20480 data: 6 6 5 5 5
35804 025/300 Write block 3204 len 28672 data: 7 6 6 5 6 5 5
35805 026/300 Read block 1912 len 32768
35806 027/300 Read block 4067 len 4096
35807 028/300 Read block 2089 len 20480
35808 029/300 Write block 1111 len 8192 data: 9 9
35809 030/300 Flush
35810 031/300 Write block 2644 len 8192 data: 4 6
35811 032/300 Write block 1173 len 20480 data: 4 4 4 4 3
35812 033/300 Read block 943 len 4096
35813 034/300 Read block 3336 len 4096
35814 035/300 Write block 337 len 32768 data: 6 5 5 7 8 7 7 7
35815 036/300 Flush
35816 037/300 Write block 1673 len 28672 data: 8 6 6 6 5 5 4
35817 038/300 Read block 3508 len 4096
35818 039/300 Read block 206 len 32768
35819 040/300 Read block 1547 len 36864
35820 041/300 Read block 1518 len 28672
35821 042/300 Read block 1063 len 12288
35822 043/300 Write block 2992 len 40960 data: 4 4 4 3 3 4 5 4 5 6
35823 044/300 Read block 162 len 24576
35824 045/300 Write block 4637 len 36864 data: 6 6 6 7 9 6 6 6 7
35825 046/300 Write block 3132 len 8192 data: 7 6
35826 047/300 Write block 4606 len 32768 data: 4 4 5 6 5 5 6 6
35827 048/300 Write block 3795 len 36864 data: 5 3 3 3 4 3 3 5 5
35828 049/300 Read block 2659 len 8192
35829 050/300 Write block 2408 len 8192 data: 6 5
35830 051/300 Read block 500 len 8192
35831 052/300 Write block 2842 len 36864 data: 5 6 7 8 7 6 5 5 6
35832 053/300 Write block 2117 len 8192 data: 5 6
35833 054/300 Write block 505 len 28672 data: 8 5 4 6 7 7 8
35834 055/300 Write block 2113 len 40960 data: 6 7 4 6 6 7 8 6 6 6
35835 056/300 Read block 3711 len 28672
35836 057/300 Read block 4301 len 4096
35837 058/300 Write block 1986 len 8192 data: 6 7
35838 059/300 Read block 157 len 4096
35839 060/300 Write block 471 len 32768 data: 6 7 6 6 8 8 8 7
35840 061/300 Read block 2721 len 12288
35841 062/300 Write block 4884 len 4096 data: 3
35842 063/300 Write block 3661 len 4096 data: 5
35843 064/300 Read block 741 len 16384
35844 065/300 Write block 1959 len 20480 data: 7 5 5 8 9
35845 066/300 Read block 820 len 12288
35846 067/300 Read block 2679 len 40960
35847 068/300 Read block 1755 len 8192
35848 069/300 Read block 4831 len 28672
35849 070/300 Write block 3532 len 32768 data: 7 7 7 7 8 9 9 7
35850 071/300 Read block 4540 len 8192
35851 072/300 Read block 1915 len 20480
35852 073/300 Flush
35853 074/300 Write block 474 len 4096 data: 7
35854 075/300 Read block 1955 len 16384
35855 076/300 Write block 3631 len 20480 data: 7 7 7 5 6
35856 077/300 Read block 2481 len 12288
35857 078/300 Read block 2024 len 20480
35858 079/300 Write block 961 len 4096 data: 7
35859 080/300 Read block 3333 len 20480
35860 081/300 Write block 155 len 24576 data: 9 8 8 8 8 7
35861 082/300 Write block 3724 len 20480 data: 10 7 7 7 8
35862 083/300 Write block 583 len 12288 data: 6 5 5
35863 084/300 Read block 3351 len 20480
35864 085/300 Read block 4688 len 28672
35865 086/300 Write block 3700 len 32768 data: 8 8 8 5 4 4 3 3
35866 087/300 Read block 3341 len 36864
35867 088/300 Read block 1952 len 28672
35868 089/300 Read block 3472 len 36864
35869 090/300 Read block 1414 len 28672
35870 091/300 Write block 1449 len 32768 data: 4 4 3 6 5 6 6 5
35871 092/300 Read block 2256 len 8192
35872 093/300 Write block 3333 len 24576 data: 3 5 4 5 4 4
35873 094/300 Read block 633 len 8192
35874 095/300 Write block 3651 len 20480 data: 7 10 12 12 11
35875 096/300 Read block 497 len 8192
35876 097/300 Write block 949 len 40960 data: 8 6 7 5 5 5 5 7 9 9
35877 098/300 Write block 891 len 20480 data: 6 6 6 6 7
35878 099/300 Write block 3511 len 36864 data: 7 5 5 4 5 6 6 7 9
35879 100/300 Write block 4637 len 8192 data: 7 7
35880 101/300 Read block 3441 len 28672
35881 102/300 Read block 1024 len 36864
35882 103/300 Read block 1688 len 8192
35883 104/300 Write block 3829 len 40960 data: 7 6 4 4 5 5 7 7 6 5
35884 105/300 Read block 2465 len 24576
35885 106/300 Write block 4709 len 20480 data: 7 9 8 9 8
35886 107/300 Read block 140 len 36864
35887 108/300 Read block 3663 len 36864
35888 109/300 Write block 1300 len 20480 data: 6 5 4 3 3
35889 110/300 Write block 4590 len 12288 data: 6 8 8
35890 111/300 Write block 2336 len 12288 data: 3 3 7
35891 112/300 Read block 2467 len 32768
35892 113/300 Read block 4470 len 36864
35893 114/300 Flush
35894 115/300 Write block 4538 len 8192 data: 9 9
35895 116/300 Read block 3252 len 12288
35896 117/300 Read block 3614 len 12288
35897 118/300 Read block 4685 len 36864
35898 119/300 Write block 2330 len 40960 data: 6 6 4 3 2 2 4 4 8 6
35899 120/300 Flush
35900 121/300 Write block 212 len 24576 data: 3 3 7 5 6 7
35901 122/300 Write block 4571 len 4096 data: 6
35902 123/300 Read block 4463 len 20480
35903 124/300 Read block 2573 len 24576
35904 125/300 Write block 4463 len 12288 data: 5 5 5
35905 126/300 Write block 1950 len 32768 data: 6 6 6 4 2 4 5 8
35906 127/300 Read block 2784 len 4096
35907 128/300 Write block 1050 len 20480 data: 9 7 7 11 11
35908 129/300 Read block 1005 len 8192
35909 130/300 Read block 3322 len 20480
35910 131/300 Read block 139 len 40960
35911 132/300 Write block 2634 len 4096 data: 3
35912 133/300 Read block 4300 len 20480
35913 134/300 Write block 1639 len 20480 data: 8 7 6 4 5
35914 135/300 Write block 4013 len 24576 data: 7 7 7 7 7 6
35915 136/300 Write block 4162 len 16384 data: 7 8 9 8
35916 137/300 Write block 1583 len 36864 data: 9 9 10 8 11 12 12 10 8
35917 138/300 Read block 3350 len 16384
35918 139/300 Read block 1900 len 4096
35919 140/300 Read block 510 len 4096
35920 141/300 Write block 3055 len 40960 data: 7 8 8 6 4 4 4 5 4 4
35921 142/300 Read block 1587 len 28672
35922 143/300 Read block 1447 len 12288
35923 144/300 Write block 1848 len 16384 data: 3 3 4 5
35924 145/300 Read block 1648 len 16384
35925 146/300 Flush
35926 147/300 Write block 3371 len 32768 data: 6 7 6 5 5 6 5 5
35927 148/300 Read block 4603 len 40960
35928 149/300 Read block 3407 len 32768
35929 150/300 Read block 1351 len 8192
35930 151/300 Read block 2576 len 12288
35931 152/300 Read block 257 len 28672
35932 153/300 Write block 2589 len 20480 data: 7 5 6 6 6
35933 154/300 Flush
35934 155/300 Write block 4993 len 24576 data: 6 3 3 4 4 4
35935 156/300 Read block 2325 len 40960
35936 157/300 Write block 1511 len 24576 data: 6 6 7 5 5 5
35937 158/300 Flush
35938 159/300 Read block 4922 len 8192
35939 160/300 Read block 2996 len 4096
35940 161/300 Write block 3946 len 20480 data: 8 7 6 7 8
35941 162/300 Read block 3770 len 32768
35942 163/300 Write block 4926 len 4096 data: 8
35943 164/300 Read block 1972 len 28672
35944 165/300 Write block 1332 len 4096 data: 9
35945 166/300 Read block 4784 len 28672
35946 167/300 Read block 3422 len 24576
35947 168/300 Read block 2042 len 16384
35948 169/300 Read block 1603 len 16384
35949 170/300 Read block 1614 len 20480
35950 171/300 Read block 4030 len 28672
35951 172/300 Write block 939 len 24576 data: 4 4 5 4 6 8
35952 173/300 Read block 203 len 4096
35953 174/300 Flush
35954 175/300 Read block 3112 len 12288
35955 176/300 Read block 2069 len 8192
35956 177/300 Write block 4943 len 36864 data: 6 6 6 7 6 5 5 5 5
35957 178/300 Write block 2094 len 32768 data: 4 4 5 4 5 6 6 6
35958 179/300 Read block 1624 len 16384
35959 180/300 Read block 4665 len 36864
35960 181/300 Flush
35961 182/300 Flush
35962 183/300 Read block 643 len 20480
35963 184/300 Read block 2934 len 8192
35964 185/300 Write block 875 len 12288 data: 5 4 5
35965 186/300 Write block 3901 len 28672 data: 5 4 3 2 2 2 2
35966 187/300 Write block 2706 len 16384 data: 5 6 5 5
35967 188/300 Read block 584 len 40960
35968 189/300 Write block 1787 len 40960 data: 4 3 5 5 6 6 8 9 8 8
35969 190/300 Read block 2420 len 16384
35970 191/300 Write block 1426 len 8192 data: 6 7
35971 192/300 Read block 4956 len 8192
35972 193/300 Read block 1713 len 28672
35973 194/300 Write block 4805 len 40960 data: 7 5 4 5 6 6 7 6 5 8
35974 195/300 Flush
35975 196/300 Write block 2405 len 12288 data: 11 9 7
35976 197/300 Read block 2432 len 8192
35977 198/300 Write block 3337 len 24576 data: 5 5 4 4 4 5
35978 199/300 Write block 1766 len 32768 data: 5 5 5 7 6 5 5 5
35979 200/300 Flush
35980 201/300 Read block 2244 len 28672
35981 202/300 Write block 935 len 32768 data: 8 7 6 4 5 5 6 5
35982 203/300 Write block 4650 len 4096 data: 6
35983 204/300 Read block 2015 len 40960
35984 205/300 Read block 1616 len 24576
35985 206/300 Read block 1520 len 16384
35986 207/300 Read block 4848 len 4096
35987 208/300 Read block 1031 len 4096
35988 209/300 Write block 2930 len 12288 data: 7 7 8
35989 210/300 Read block 1289 len 4096
35990 211/300 Read block 2176 len 24576
35991 212/300 Write block 1974 len 24576 data: 6 6 7 6 6 8
35992 213/300 Read block 2010 len 20480
35993 214/300 Write block 792 len 20480 data: 6 5 4 5 6
35994 215/300 Write block 3722 len 24576 data: 10 11 11 8 8 8
35995 216/300 Read block 1256 len 12288
35996 217/300 Write block 3011 len 40960 data: 5 5 5 5 6 7 10 9 9 9
35997 218/300 Read block 2835 len 32768
35998 219/300 Read block 3710 len 24576
35999 220/300 Read block 3998 len 20480
36000 221/300 Write block 3096 len 8192 data: 3 6
36001 222/300 Read block 578 len 4096
36002 223/300 Write block 2847 len 24576 data: 7 6 6 7 7 7
36003 224/300 Read block 1842 len 36864
36004 225/300 Read block 3049 len 24576
36005 226/300 Read block 2081 len 16384
36006 227/300 Read block 524 len 16384
36007 228/300 Read block 2174 len 40960
36008 229/300 Write block 3297 len 28672 data: 5 6 6 5 5 5 4
36009 230/300 Write block 2755 len 40960 data: 2 2 3 3 4 4 4 3 4 6
36010 231/300 Flush
36011 232/300 Write block 1283 len 40960 data: 8 7 7 9 9 9 7 5 5 6
36012 233/300 Write block 1184 len 32768 data: 6 7 6 5 5 4 3 4
36013 234/300 Write block 3940 len 8192 data: 6 9
36014 235/300 Read block 1386 len 40960
36015 236/300 Read block 1223 len 28672
36016 237/300 Write block 3015 len 36864 data: 7 8 11 10 10 10 9 9 9
36017 238/300 Read block 4935 len 16384
36018 239/300 Read block 4267 len 16384
36019 240/300 Write block 1729 len 16384 data: 8 8 9 7
36020 241/300 Read block 2863 len 4096
36021 242/300 Write block 4547 len 28672 data: 6 7 7 8 10 8 7
36022 243/300 Read block 4336 len 40960
36023 244/300 Read block 3616 len 24576
36024 245/300 Write block 582 len 20480 data: 6 7 6 6 5
36025 246/300 Write block 4966 len 8192 data: 8 8
36026 247/300 Write block 3564 len 20480 data: 7 7 8 7 6
36027 248/300 Read block 1752 len 36864
36028 249/300 Write block 1507 len 40960 data: 5 5 6 5 7 7 8 6 6 6
36029 250/300 Read block 1517 len 28672
36030 251/300 Read block 3521 len 28672
36031 252/300 Read block 1350 len 32768
36032 253/300 Read block 1516 len 4096
36033 254/300 Flush
36034 255/300 Flush
36035 256/300 Read block 1996 len 28672
36036 257/300 Flush
36037 258/300 Write block 414 len 16384 data: 10 10 11 11
36038 259/300 Read block 4919 len 4096
36039 260/300 Flush
36040 261/300 Write block 4506 len 12288 data: 4 4 5
36041 262/300 Read block 1825 len 40960
36042 263/300 Read block 4259 len 8192
36043 264/300 Write block 4424 len 4096 data: 5
36044 265/300 Read block 98 len 28672
36045 266/300 Read block 1821 len 4096
36046 267/300 Write block 101 len 32768 data: 6 6 6 4 5 6 4 7
36047 268/300 Read block 3439 len 40960
36048 269/300 Read block 855 len 28672
36049 270/300 Write block 390 len 32768 data: 8 7 6 6 6 6 4 4
36050 271/300 Read block 4101 len 16384
36051 272/300 Write block 3047 len 24576 data: 7 7 7 6 5 4
36052 273/300 Read block 4164 len 32768
36053 274/300 Write block 989 len 28672 data: 5 5 4 4 4 4 3
36054 275/300 Read block 2080 len 36864
36055 276/300 Read block 4399 len 16384
36056 277/300 Write block 2412 len 24576 data: 8 11 10 8 8 11
36057 278/300 Flush
36058 279/300 Read block 385 len 12288
36059 280/300 Flush
36060 281/300 Read block 3351 len 28672
36061 282/300 Read block 2904 len 8192
36062 283/300 Read block 1843 len 16384
36063 284/300 Write block 1770 len 40960 data: 7 6 6 6 5 6 7 7 7 5
36064 285/300 Read block 2261 len 36864
36065 286/300 Write block 2529 len 40960 data: 8 10 10 9 10 8 9 8 7 7
36066 287/300 Read block 1611 len 8192
36067 288/300 Read block 1591 len 4096
36068 289/300 Read block 1998 len 12288
36069 290/300 Write block 578 len 16384 data: 5 5 7 7
36070 291/300 Write block 3662 len 24576 data: 6 6 6 5 3 4
36071 292/300 Read block 4318 len 32768
36072 293/300 Read block 4296 len 16384
36073 294/300 Read block 675 len 40960
36074 295/300 Flush
36075 296/300 Write block 3238 len 40960 data: 5 4 4 5 5 4 5 5 5 7
36076 297/300 Flush
36077 298/300 Write block 1265 len 16384 data: 4 5 4 4
36078 299/300 Read block 2146 len 32768
36079 300/300 Read block 690 len 36864
36080 301/300 Write block 4475 len 16384 data: 4 4 4 5
36081 Main task has work to do, go find it
36082 got dsc Start(2)
36083 start 2
36084 Mar 17 22:27:01.709 INFO request completed, latency_us: 220, response_code: 204, uri: /start/cid/2, method: POST, req_id: 1d5d4bc5-97af-4524-9505-964fcc46886b, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
36085 Replay: started 2, returned:Ok(())
36086 ----------------------------------------------------------------
36087 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
36088 States: Active Active Offline
36089 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
36090 9755 Acked Read 8 Done Done Sent false
36091 9756 Acked Read 6 Done Done Sent false
36092 9757 Acked Write 6 Done Done Sent false
36093 9758 Acked Write 2 Done Done Sent false
36094 9759 Acked Write 10 Done Done Sent false
36095 9760 Acked Flush 0 Done Done Sent false
36096 9761 Acked Read 9 Done Done Sent false
36097 9762 Acked Write 10 Done Done Sent false
36098 9763 Acked Read 6 Done Done Sent false
36099 9764 Acked Read 6 Done Done Sent false
36100 9765 Acked Write 9 Done Done Sent false
36101 9766 Acked Read 8 Done Done Sent false
36102 9767 Acked Write 5 Done Done Sent false
36103 9768 Acked Write 7 Done Done Sent false
36104 9769 Acked Write 9 Done Done Sent false
36105 9770 Acked Flush 0 Done Done Sent false
36106 9771 Acked Read 6 Done Done Sent false
36107 9772 Acked Flush 0 Done Done Sent false
36108 9773 Acked Write 4 Done Done Sent false
36109 9774 Acked Read 3 Done Done Sent false
36110 9775 Acked Read 2 Done Done Sent false
36111 9776 Acked Read 4 Done Done Sent false
36112 9777 Acked Write 7 Done Done Sent false
36113 9778 Acked Write 5 Done Done Sent false
36114 9779 Acked Write 7 Done Done Sent false
36115 9780 Acked Read 8 Done Done Sent false
36116 9781 Acked Read 1 Done Done Sent false
36117 9782 Acked Read 5 Done Done Sent false
36118 9783 Acked Write 2 Done Done Sent false
36119 9784 Acked Flush 0 Done Done Sent false
36120 9785 Acked Write 2 Done Done Sent false
36121 9786 Acked Write 5 Done Done Sent false
36122 9787 Acked Read 1 Done Done Sent false
36123 9788 Acked Read 1 Done Done Sent false
36124 9789 Acked Write 8 Done Done Sent false
36125 9790 Acked Flush 0 Done Done Sent false
36126 9791 Acked Write 7 Done Done Sent false
36127 9792 Acked Read 1 Done Done Sent false
36128 9793 Acked Read 8 Done Done Sent false
36129 9794 Acked Read 9 Done Done Sent false
36130 9795 Acked Read 7 Done Done Sent false
36131 9796 Acked Read 3 Done Done Sent false
36132 9797 Acked Write 10 Done Done Sent false
36133 9798 Acked Read 6 Done Done Sent false
36134 9799 Acked Write 9 Done Done Sent false
36135 9800 Acked Write 2 Done Done Sent false
36136 9801 Acked Write 8 Done Done Sent false
36137 9802 Acked Write 9 Done Done Sent false
36138 9803 Acked Read 2 Done Done Sent false
36139 9804 Acked Write 2 Done Done Sent false
36140 9805 Acked Read 2 Done Done Sent false
36141 9806 Acked Write 9 Done Done Sent false
36142 9807 Acked Write 2 Done Done Sent false
36143 9808 Acked Write 7 Done Done Sent false
36144 9809 Acked Write 10 Done Done Sent false
36145 9810 Acked Read 7 Done Done Sent false
36146 9811 Acked Read 1 Done Done Sent false
36147 9812 Acked Write 2 Done Done Sent false
36148 9813 Acked Read 1 Done Done Sent false
36149 9814 Acked Write 8 Done Done Sent false
36150 9815 Acked Read 3 Done Done Sent false
36151 9816 Acked Write 1 Done Done Sent false
36152 9817 Acked Write 1 Done Done Sent false
36153 9818 Acked Read 4 Done Done Sent false
36154 9819 Acked Write 5 Done Done Sent false
36155 9820 Acked Read 3 Done Done Sent false
36156 9821 Acked Read 10 Done Done Sent false
36157 9822 Acked Read 2 Done Done Sent false
36158 9823 Acked Read 7 Done Done Sent false
36159 9824 Acked Write 8 Done Done Sent false
36160 9825 Acked Read 2 Done Done Sent false
36161 9826 Acked Read 5 Done Done Sent false
36162 9827 Acked Flush 0 Done Done Sent false
36163 9828 Acked Write 1 Done Done Sent false
36164 9829 Acked Read 4 Done Done Sent false
36165 9830 Acked Write 5 Done Done Sent false
36166 9831 Acked Read 3 Done Done Sent false
36167 9832 Acked Read 5 Done Done Sent false
36168 9833 Acked Write 1 Done Done Sent false
36169 9834 Acked Read 5 Done Done Sent false
36170 9835 Acked Write 6 Done Done Sent false
36171 9836 Acked Write 5 Done Done Sent false
36172 9837 Acked Write 3 Done Done Sent false
36173 9838 Acked Read 5 Done Done Sent false
36174 9839 Acked Read 7 Done Done Sent false
36175 9840 Acked Write 8 Done Done Sent false
36176 9841 Acked Read 9 Done Done Sent false
36177 9842 Acked Read 7 Done Done Sent false
36178 9843 Acked Read 9 Done Done Sent false
36179 9844 Acked Read 7 Done Done Sent false
36180 9845 Acked Write 8 Done Done Sent false
36181 9846 Acked Read 2 Done Done Sent false
36182 9847 Acked Write 6 Done Done Sent false
36183 9848 Acked Read 2 Done Done Sent false
36184 9849 Acked Write 5 Done Done Sent false
36185 9850 Acked Read 2 Done Done Sent false
36186 9851 Acked Write 10 Done Done Sent false
36187 9852 Acked Write 5 Done Done Sent false
36188 9853 Acked Write 9 Done Done Sent false
36189 9854 Acked Write 2 Done Done Sent false
36190 9855 Acked Read 7 Done Done Sent false
36191 9856 Acked Read 9 Done Done Sent false
36192 9857 Acked Read 2 Done Done Sent false
36193 9858 Acked Write 10 Done Done Sent false
36194 9859 Acked Read 6 Done Done Sent false
36195 9860 Acked Write 5 Done Done Sent false
36196 9861 Acked Read 9 Done Done Sent false
36197 9862 Acked Read 9 Done Done Sent false
36198 9863 Acked Write 5 Done Done Sent false
36199 9864 Acked Write 3 Done Done Sent false
36200 9865 Acked Write 3 Done Done Sent false
36201 9866 Acked Read 8 Done Done Sent false
36202 9867 Acked Read 9 Done Done Sent false
36203 9868 Acked Flush 0 Done Done Sent false
36204 9869 Acked Write 2 Done Done Sent false
36205 9870 Acked Read 3 Done Done Sent false
36206 9871 Acked Read 3 Done Done Sent false
36207 9872 Acked Read 9 Done Done Sent false
36208 9873 Acked Write 10 Done Done Sent false
36209 9874 Acked Flush 0 Done Done Sent false
36210 9875 Acked Write 6 Done Done Sent false
36211 9876 Acked Write 1 Done Done Sent false
36212 9877 Acked Read 5 Done Done Sent false
36213 9878 Acked Read 6 Done Done Sent false
36214 9879 Acked Write 3 Done Done Sent false
36215 9880 Acked Write 8 Done Done Sent false
36216 9881 Acked Read 1 Done Done Sent false
36217 9882 Acked Write 5 Done Done Sent false
36218 9883 Acked Read 2 Done Done Sent false
36219 9884 Acked Read 5 Done Done Sent false
36220 9885 Acked Read 10 Done Done Sent false
36221 9886 Acked Write 1 Done Done Sent false
36222 9887 Acked Read 5 Done Done Sent false
36223 9888 Acked Write 5 Done Done Sent false
36224 9889 Acked Write 6 Done Done Sent false
36225 9890 Acked Write 4 Done Done Sent false
36226 9891 Acked Write 9 Done Done Sent false
36227 9892 Acked Read 4 Done Done Sent false
36228 9893 Acked Read 1 Done Done Sent false
36229 9894 Acked Read 1 Done Done Sent false
36230 9895 Acked Write 10 Done Done Sent false
36231 9896 Acked Read 7 Done Done Sent false
36232 9897 Acked Read 3 Done Done Sent false
36233 9898 Acked Write 4 Done Done Sent false
36234 9899 Acked Read 4 Done Done Sent false
36235 9900 Acked Flush 0 Done Done Sent false
36236 9901 Acked Write 8 Done Done Sent false
36237 9902 Acked Read 10 Done Done Sent false
36238 9903 Acked Read 8 Done Done Sent false
36239 9904 Acked Read 2 Done Done Sent false
36240 9905 Acked Read 3 Done Done Sent false
36241 9906 Acked Read 7 Done Done Sent false
36242 9907 Acked Write 5 Done Done Sent false
36243 9908 Acked Flush 0 Done Done Sent false
36244 9909 Acked Write 6 Done Done Sent false
36245 9910 Acked Read 10 Done Done Sent false
36246 9911 Acked Write 6 Done Done Sent false
36247 9912 Acked Flush 0 Done Done Sent false
36248 9913 Acked Read 2 Done Done Sent false
36249 9914 Acked Read 1 Done Done Sent false
36250 9915 Acked Write 5 Done Done Sent false
36251 9916 Acked Read 8 Done Done Sent false
36252 9917 Acked Write 1 Done Done Sent false
36253 9918 Acked Read 7 Done Done Sent false
36254 9919 Acked Write 1 Done Done Sent false
36255 9920 Acked Read 7 Done Done Sent false
36256 9921 Acked Read 6 Done Done Sent false
36257 9922 Acked Read 4 Done Done Sent false
36258 9923 Acked Read 4 Done Done Sent false
36259 9924 Acked Read 5 Done Done Sent false
36260 9925 Acked Read 7 Done Done Sent false
36261 9926 Acked Write 6 Done Done Sent false
36262 9927 Acked Read 1 Done Done Sent false
36263 9928 Acked Flush 0 Done Done Sent false
36264 9929 Acked Read 3 Done Done Sent false
36265 9930 Acked Read 2 Done Done Sent false
36266 9931 Acked Write 9 Done Done Sent false
36267 9932 Acked Write 8 Done Done Sent false
36268 9933 Acked Read 4 Done Done Sent false
36269 9934 Acked Read 9 Done Done Sent false
36270 9935 Acked Flush 0 Done Done Sent false
36271 9936 Acked Flush 0 Done Done Sent false
36272 9937 Acked Read 5 Done Done Sent false
36273 9938 Acked Read 2 Done Done Sent false
36274 9939 Acked Write 3 Done Done Sent false
36275 9940 Acked Write 7 Done Done Sent false
36276 9941 Acked Write 4 Done Done Sent false
36277 9942 Acked Read 10 Done Done Sent false
36278 9943 Acked Write 10 Done Done Sent false
36279 9944 Acked Read 4 Done Done Sent false
36280 9945 Acked Write 2 Done Done Sent false
36281 9946 Acked Read 2 Done Done Sent false
36282 9947 Acked Read 7 Done Done Sent false
36283 9948 Acked Write 10 Done Done Sent false
36284 9949 Acked Flush 0 Done Done Sent false
36285 9950 Acked Write 3 Done Done Sent false
36286 9951 Acked Read 2 Done Done Sent false
36287 9952 Acked Write 6 Done Done Sent false
36288 9953 Acked Write 8 Done Done Sent false
36289 9954 Acked Flush 0 Done Done Sent false
36290 9955 Acked Read 7 Done Done Sent false
36291 9956 Acked Write 8 Done Done Sent false
36292 9957 Acked Write 1 Done Done Sent false
36293 9958 Acked Read 10 Done Done Sent false
36294 9959 Acked Read 6 Done Done Sent false
36295 9960 Acked Read 4 Done Done Sent false
36296 9961 Acked Read 1 Done Done Sent false
36297 9962 Acked Read 1 Done Done Sent false
36298 9963 Acked Write 3 Done Done Sent false
36299 9964 Acked Read 1 Done Done Sent false
36300 9965 Acked Read 6 Done Done Sent false
36301 9966 Acked Write 6 Done Done Sent false
36302 9967 Acked Read 5 Done Done Sent false
36303 9968 Acked Write 5 Done Done Sent false
36304 9969 Acked Write 6 Done Done Sent false
36305 9970 Acked Read 3 Done Done Sent false
36306 9971 Acked Write 10 Done Done Sent false
36307 9972 Acked Read 8 Done Done Sent false
36308 9973 Acked Read 6 Done Done Sent false
36309 9974 Acked Read 5 Done Done Sent false
36310 9975 Acked Write 2 Done Done Sent false
36311 9976 Acked Read 1 Done Done Sent false
36312 9977 Acked Write 6 Done Done Sent false
36313 9978 Acked Read 9 Done Done Sent false
36314 9979 Acked Read 6 Done Done Sent false
36315 9980 Acked Read 4 Done Done Sent false
36316 9981 Acked Read 4 Done Done Sent false
36317 9982 Acked Read 10 Done Done Sent false
36318 9983 Acked Write 7 Done Done Sent false
36319 9984 Acked Write 10 Done Done Sent false
36320 9985 Acked Flush 0 Done Done Sent false
36321 9986 Acked Write 10 Done Done Sent false
36322 9987 Acked Write 8 Done Done Sent false
36323 9988 Acked Write 2 Done Done Sent false
36324 9989 Acked Read 10 Done Done Sent false
36325 9990 Acked Read 7 Done Done Sent false
36326 9991 Acked Write 9 Done Done Sent false
36327 9992 Acked Read 4 Done Done Sent false
36328 9993 Acked Read 4 Done Done Sent false
36329 9994 Acked Write 4 Done Done Sent false
36330 9995 Acked Read 1 Done Done Sent false
36331 9996 Acked Write 7 Done Done Sent false
36332 9997 Acked Read 10 Done Done Sent false
36333 9998 Acked Read 6 Done Done Sent false
36334 9999 Acked Write 5 Done Done Sent false
36335 10000 Acked Write 2 Done Done Sent false
36336 10001 Acked Write 5 Done Done Sent false
36337 10002 Acked Read 9 Done Done Sent false
36338 10003 Acked Write 10 Done Done Sent false
36339 10004 Acked Read 7 Done Done Sent false
36340 10005 Acked Read 7 Done Done Sent false
36341 10006 Acked Read 8 Done Done Sent false
36342 10007 Acked Read 1 Done Done Sent false
36343 10008 Acked Flush 0 Done Done Sent false
36344 10009 Acked Flush 0 Done Done Sent false
36345 10010 Acked Read 7 Done Done Sent false
36346 10011 Acked Flush 0 Done Done Sent false
36347 10012 Acked Write 4 Done Done Sent false
36348 10013 Acked Read 1 Done Done Sent false
36349 10014 Acked Flush 0 Done Done Sent false
36350 10015 Acked Write 3 Done Done Sent false
36351 10016 Acked Read 10 Done Done Sent false
36352 10017 Acked Read 2 Done Done Sent false
36353 10018 Acked Write 1 Done Done Sent false
36354 10019 Acked Read 7 Done Done Sent false
36355 10020 Acked Read 1 Done Done Sent false
36356 10021 Acked Write 8 Done Done Sent false
36357 10022 Acked Read 10 Done Done Sent false
36358 10023 Acked Read 7 Done Done Sent false
36359 10024 Acked Write 8 Done Done Sent false
36360 10025 Acked Read 4 Done Done Sent false
36361 10026 Acked Write 6 Done Done Sent false
36362 10027 Acked Read 8 Done Done Sent false
36363 10028 Acked Write 7 Done Done Sent false
36364 10029 Acked Read 9 Done Done Sent false
36365 10030 Acked Read 4 Done Done Sent false
36366 10031 Acked Write 6 Done Done Sent false
36367 10032 Acked Flush 0 Done Done Sent false
36368 10033 Acked Read 3 Done Done Sent false
36369 10034 Acked Flush 0 Done Done Sent false
36370 10035 Acked Read 7 Done Done Sent false
36371 10036 Acked Read 2 Done Done Sent false
36372 10037 Acked Read 4 Done Done Sent false
36373 10038 Acked Write 10 Done Done Sent false
36374 10039 Acked Read 9 Done Done Sent false
36375 10040 Acked Write 10 Done Done Sent false
36376 10041 Acked Read 2 Done Done Sent false
36377 10042 Acked Read 1 Done Done Sent false
36378 10043 Acked Read 3 Done Done Sent false
36379 10044 Acked Write 4 Done Done Sent false
36380 10045 Acked Write 6 Done Done Sent false
36381 10046 Acked Read 8 Done Done Sent false
36382 10047 Acked Read 4 Done Done Sent false
36383 10048 Acked Read 10 Done Done Sent false
36384 10049 Acked Flush 0 Done Done Sent false
36385 10050 Acked Write 10 Done Done Sent false
36386 10051 Acked Flush 0 Done Done Sent false
36387 10052 Acked Write 4 Done Done Sent false
36388 10053 Acked Read 8 Done Done Sent false
36389 10054 Acked Read 9 Done Done Sent false
36390 10055 Acked Write 4 Done Done Sent false
36391 STATES DS:0 DS:1 DS:2 TOTAL
36392 Sent 0 0 301 301
36393 Done 301 301 0 602
36394 Skipped 0 0 0 0
36395 Error 0 0 0 0
36396 Last Flush: Some(JobId(10051)) Some(JobId(10051)) Some(JobId(9754))
36397 Downstairs last five completed: 9754 9753 9752 9751 9750
36398 Upstairs last five completed: 10055 10054 10053 10052 10051
36399 CLIENT: Up:0 ds:301 act:2
364002025-03-17T22:27:02.362ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
364012025-03-17T22:27:03.457ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
36402 [8830] got start action
36403 [8830] I am going to restart
36404 Starting downstairs at port 8830
36405 Make output file at "/tmp/dsc/downstairs-8830.txt"
36406 [8830][2] reports Starting
36407 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1241)
36408 [8830][2] reports Running
364092025-03-17T22:27:04.365ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
364102025-03-17T22:27:05.367ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
36411 ----------------------------------------------------------------
36412 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
36413 States: Active Active Offline
36414 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
36415 9755 Acked Read 8 Done Done Sent false
36416 9756 Acked Read 6 Done Done Sent false
36417 9757 Acked Write 6 Done Done Sent false
36418 9758 Acked Write 2 Done Done Sent false
36419 9759 Acked Write 10 Done Done Sent false
36420 9760 Acked Flush 0 Done Done Sent false
36421 9761 Acked Read 9 Done Done Sent false
36422 9762 Acked Write 10 Done Done Sent false
36423 9763 Acked Read 6 Done Done Sent false
36424 9764 Acked Read 6 Done Done Sent false
36425 9765 Acked Write 9 Done Done Sent false
36426 9766 Acked Read 8 Done Done Sent false
36427 9767 Acked Write 5 Done Done Sent false
36428 9768 Acked Write 7 Done Done Sent false
36429 9769 Acked Write 9 Done Done Sent false
36430 9770 Acked Flush 0 Done Done Sent false
36431 9771 Acked Read 6 Done Done Sent false
36432 9772 Acked Flush 0 Done Done Sent false
36433 9773 Acked Write 4 Done Done Sent false
36434 9774 Acked Read 3 Done Done Sent false
36435 9775 Acked Read 2 Done Done Sent false
36436 9776 Acked Read 4 Done Done Sent false
36437 9777 Acked Write 7 Done Done Sent false
36438 9778 Acked Write 5 Done Done Sent false
36439 9779 Acked Write 7 Done Done Sent false
36440 9780 Acked Read 8 Done Done Sent false
36441 9781 Acked Read 1 Done Done Sent false
36442 9782 Acked Read 5 Done Done Sent false
36443 9783 Acked Write 2 Done Done Sent false
36444 9784 Acked Flush 0 Done Done Sent false
36445 9785 Acked Write 2 Done Done Sent false
36446 9786 Acked Write 5 Done Done Sent false
36447 9787 Acked Read 1 Done Done Sent false
36448 9788 Acked Read 1 Done Done Sent false
36449 9789 Acked Write 8 Done Done Sent false
36450 9790 Acked Flush 0 Done Done Sent false
36451 9791 Acked Write 7 Done Done Sent false
36452 9792 Acked Read 1 Done Done Sent false
36453 9793 Acked Read 8 Done Done Sent false
36454 9794 Acked Read 9 Done Done Sent false
36455 9795 Acked Read 7 Done Done Sent false
36456 9796 Acked Read 3 Done Done Sent false
36457 9797 Acked Write 10 Done Done Sent false
36458 9798 Acked Read 6 Done Done Sent false
36459 9799 Acked Write 9 Done Done Sent false
36460 9800 Acked Write 2 Done Done Sent false
36461 9801 Acked Write 8 Done Done Sent false
36462 9802 Acked Write 9 Done Done Sent false
36463 9803 Acked Read 2 Done Done Sent false
36464 9804 Acked Write 2 Done Done Sent false
36465 9805 Acked Read 2 Done Done Sent false
36466 9806 Acked Write 9 Done Done Sent false
36467 9807 Acked Write 2 Done Done Sent false
36468 9808 Acked Write 7 Done Done Sent false
36469 9809 Acked Write 10 Done Done Sent false
36470 9810 Acked Read 7 Done Done Sent false
36471 9811 Acked Read 1 Done Done Sent false
36472 9812 Acked Write 2 Done Done Sent false
36473 9813 Acked Read 1 Done Done Sent false
36474 9814 Acked Write 8 Done Done Sent false
36475 9815 Acked Read 3 Done Done Sent false
36476 9816 Acked Write 1 Done Done Sent false
36477 9817 Acked Write 1 Done Done Sent false
36478 9818 Acked Read 4 Done Done Sent false
36479 9819 Acked Write 5 Done Done Sent false
36480 9820 Acked Read 3 Done Done Sent false
36481 9821 Acked Read 10 Done Done Sent false
36482 9822 Acked Read 2 Done Done Sent false
36483 9823 Acked Read 7 Done Done Sent false
36484 9824 Acked Write 8 Done Done Sent false
36485 9825 Acked Read 2 Done Done Sent false
36486 9826 Acked Read 5 Done Done Sent false
36487 9827 Acked Flush 0 Done Done Sent false
36488 9828 Acked Write 1 Done Done Sent false
36489 9829 Acked Read 4 Done Done Sent false
36490 9830 Acked Write 5 Done Done Sent false
36491 9831 Acked Read 3 Done Done Sent false
36492 9832 Acked Read 5 Done Done Sent false
36493 9833 Acked Write 1 Done Done Sent false
36494 9834 Acked Read 5 Done Done Sent false
36495 9835 Acked Write 6 Done Done Sent false
36496 9836 Acked Write 5 Done Done Sent false
36497 9837 Acked Write 3 Done Done Sent false
36498 9838 Acked Read 5 Done Done Sent false
36499 9839 Acked Read 7 Done Done Sent false
36500 9840 Acked Write 8 Done Done Sent false
36501 9841 Acked Read 9 Done Done Sent false
36502 9842 Acked Read 7 Done Done Sent false
36503 9843 Acked Read 9 Done Done Sent false
36504 9844 Acked Read 7 Done Done Sent false
36505 9845 Acked Write 8 Done Done Sent false
36506 9846 Acked Read 2 Done Done Sent false
36507 9847 Acked Write 6 Done Done Sent false
36508 9848 Acked Read 2 Done Done Sent false
36509 9849 Acked Write 5 Done Done Sent false
36510 9850 Acked Read 2 Done Done Sent false
36511 9851 Acked Write 10 Done Done Sent false
36512 9852 Acked Write 5 Done Done Sent false
36513 9853 Acked Write 9 Done Done Sent false
36514 9854 Acked Write 2 Done Done Sent false
36515 9855 Acked Read 7 Done Done Sent false
36516 9856 Acked Read 9 Done Done Sent false
36517 9857 Acked Read 2 Done Done Sent false
36518 9858 Acked Write 10 Done Done Sent false
36519 9859 Acked Read 6 Done Done Sent false
36520 9860 Acked Write 5 Done Done Sent false
36521 9861 Acked Read 9 Done Done Sent false
36522 9862 Acked Read 9 Done Done Sent false
36523 9863 Acked Write 5 Done Done Sent false
36524 9864 Acked Write 3 Done Done Sent false
36525 9865 Acked Write 3 Done Done Sent false
36526 9866 Acked Read 8 Done Done Sent false
36527 9867 Acked Read 9 Done Done Sent false
36528 9868 Acked Flush 0 Done Done Sent false
36529 9869 Acked Write 2 Done Done Sent false
36530 9870 Acked Read 3 Done Done Sent false
36531 9871 Acked Read 3 Done Done Sent false
36532 9872 Acked Read 9 Done Done Sent false
36533 9873 Acked Write 10 Done Done Sent false
36534 9874 Acked Flush 0 Done Done Sent false
36535 9875 Acked Write 6 Done Done Sent false
36536 9876 Acked Write 1 Done Done Sent false
36537 9877 Acked Read 5 Done Done Sent false
36538 9878 Acked Read 6 Done Done Sent false
36539 9879 Acked Write 3 Done Done Sent false
36540 9880 Acked Write 8 Done Done Sent false
36541 9881 Acked Read 1 Done Done Sent false
36542 9882 Acked Write 5 Done Done Sent false
36543 9883 Acked Read 2 Done Done Sent false
36544 9884 Acked Read 5 Done Done Sent false
36545 9885 Acked Read 10 Done Done Sent false
36546 9886 Acked Write 1 Done Done Sent false
36547 9887 Acked Read 5 Done Done Sent false
36548 9888 Acked Write 5 Done Done Sent false
36549 9889 Acked Write 6 Done Done Sent false
36550 9890 Acked Write 4 Done Done Sent false
36551 9891 Acked Write 9 Done Done Sent false
36552 9892 Acked Read 4 Done Done Sent false
36553 9893 Acked Read 1 Done Done Sent false
36554 9894 Acked Read 1 Done Done Sent false
36555 9895 Acked Write 10 Done Done Sent false
36556 9896 Acked Read 7 Done Done Sent false
36557 9897 Acked Read 3 Done Done Sent false
36558 9898 Acked Write 4 Done Done Sent false
36559 9899 Acked Read 4 Done Done Sent false
36560 9900 Acked Flush 0 Done Done Sent false
36561 9901 Acked Write 8 Done Done Sent false
36562 9902 Acked Read 10 Done Done Sent false
36563 9903 Acked Read 8 Done Done Sent false
36564 9904 Acked Read 2 Done Done Sent false
36565 9905 Acked Read 3 Done Done Sent false
36566 9906 Acked Read 7 Done Done Sent false
36567 9907 Acked Write 5 Done Done Sent false
36568 9908 Acked Flush 0 Done Done Sent false
36569 9909 Acked Write 6 Done Done Sent false
36570 9910 Acked Read 10 Done Done Sent false
36571 9911 Acked Write 6 Done Done Sent false
36572 9912 Acked Flush 0 Done Done Sent false
36573 9913 Acked Read 2 Done Done Sent false
36574 9914 Acked Read 1 Done Done Sent false
36575 9915 Acked Write 5 Done Done Sent false
36576 9916 Acked Read 8 Done Done Sent false
36577 9917 Acked Write 1 Done Done Sent false
36578 9918 Acked Read 7 Done Done Sent false
36579 9919 Acked Write 1 Done Done Sent false
36580 9920 Acked Read 7 Done Done Sent false
36581 9921 Acked Read 6 Done Done Sent false
36582 9922 Acked Read 4 Done Done Sent false
36583 9923 Acked Read 4 Done Done Sent false
36584 9924 Acked Read 5 Done Done Sent false
36585 9925 Acked Read 7 Done Done Sent false
36586 9926 Acked Write 6 Done Done Sent false
36587 9927 Acked Read 1 Done Done Sent false
36588 9928 Acked Flush 0 Done Done Sent false
36589 9929 Acked Read 3 Done Done Sent false
36590 9930 Acked Read 2 Done Done Sent false
36591 9931 Acked Write 9 Done Done Sent false
36592 9932 Acked Write 8 Done Done Sent false
36593 9933 Acked Read 4 Done Done Sent false
36594 9934 Acked Read 9 Done Done Sent false
36595 9935 Acked Flush 0 Done Done Sent false
36596 9936 Acked Flush 0 Done Done Sent false
36597 9937 Acked Read 5 Done Done Sent false
36598 9938 Acked Read 2 Done Done Sent false
36599 9939 Acked Write 3 Done Done Sent false
36600 9940 Acked Write 7 Done Done Sent false
36601 9941 Acked Write 4 Done Done Sent false
36602 9942 Acked Read 10 Done Done Sent false
36603 9943 Acked Write 10 Done Done Sent false
36604 9944 Acked Read 4 Done Done Sent false
36605 9945 Acked Write 2 Done Done Sent false
36606 9946 Acked Read 2 Done Done Sent false
36607 9947 Acked Read 7 Done Done Sent false
36608 9948 Acked Write 10 Done Done Sent false
36609 9949 Acked Flush 0 Done Done Sent false
36610 9950 Acked Write 3 Done Done Sent false
36611 9951 Acked Read 2 Done Done Sent false
36612 9952 Acked Write 6 Done Done Sent false
36613 9953 Acked Write 8 Done Done Sent false
36614 9954 Acked Flush 0 Done Done Sent false
36615 9955 Acked Read 7 Done Done Sent false
36616 9956 Acked Write 8 Done Done Sent false
36617 9957 Acked Write 1 Done Done Sent false
36618 9958 Acked Read 10 Done Done Sent false
36619 9959 Acked Read 6 Done Done Sent false
36620 9960 Acked Read 4 Done Done Sent false
36621 9961 Acked Read 1 Done Done Sent false
36622 9962 Acked Read 1 Done Done Sent false
36623 9963 Acked Write 3 Done Done Sent false
36624 9964 Acked Read 1 Done Done Sent false
36625 9965 Acked Read 6 Done Done Sent false
36626 9966 Acked Write 6 Done Done Sent false
36627 9967 Acked Read 5 Done Done Sent false
36628 9968 Acked Write 5 Done Done Sent false
36629 9969 Acked Write 6 Done Done Sent false
36630 9970 Acked Read 3 Done Done Sent false
36631 9971 Acked Write 10 Done Done Sent false
36632 9972 Acked Read 8 Done Done Sent false
36633 9973 Acked Read 6 Done Done Sent false
36634 9974 Acked Read 5 Done Done Sent false
36635 9975 Acked Write 2 Done Done Sent false
36636 9976 Acked Read 1 Done Done Sent false
36637 9977 Acked Write 6 Done Done Sent false
36638 9978 Acked Read 9 Done Done Sent false
36639 9979 Acked Read 6 Done Done Sent false
36640 9980 Acked Read 4 Done Done Sent false
36641 9981 Acked Read 4 Done Done Sent false
36642 9982 Acked Read 10 Done Done Sent false
36643 9983 Acked Write 7 Done Done Sent false
36644 9984 Acked Write 10 Done Done Sent false
36645 9985 Acked Flush 0 Done Done Sent false
36646 9986 Acked Write 10 Done Done Sent false
36647 9987 Acked Write 8 Done Done Sent false
36648 9988 Acked Write 2 Done Done Sent false
36649 9989 Acked Read 10 Done Done Sent false
36650 9990 Acked Read 7 Done Done Sent false
36651 9991 Acked Write 9 Done Done Sent false
36652 9992 Acked Read 4 Done Done Sent false
36653 9993 Acked Read 4 Done Done Sent false
36654 9994 Acked Write 4 Done Done Sent false
36655 9995 Acked Read 1 Done Done Sent false
36656 9996 Acked Write 7 Done Done Sent false
36657 9997 Acked Read 10 Done Done Sent false
36658 9998 Acked Read 6 Done Done Sent false
36659 9999 Acked Write 5 Done Done Sent false
36660 10000 Acked Write 2 Done Done Sent false
36661 10001 Acked Write 5 Done Done Sent false
36662 10002 Acked Read 9 Done Done Sent false
36663 10003 Acked Write 10 Done Done Sent false
36664 10004 Acked Read 7 Done Done Sent false
36665 10005 Acked Read 7 Done Done Sent false
36666 10006 Acked Read 8 Done Done Sent false
36667 10007 Acked Read 1 Done Done Sent false
36668 10008 Acked Flush 0 Done Done Sent false
36669 10009 Acked Flush 0 Done Done Sent false
36670 10010 Acked Read 7 Done Done Sent false
36671 10011 Acked Flush 0 Done Done Sent false
36672 10012 Acked Write 4 Done Done Sent false
36673 10013 Acked Read 1 Done Done Sent false
36674 10014 Acked Flush 0 Done Done Sent false
36675 10015 Acked Write 3 Done Done Sent false
36676 10016 Acked Read 10 Done Done Sent false
36677 10017 Acked Read 2 Done Done Sent false
36678 10018 Acked Write 1 Done Done Sent false
36679 10019 Acked Read 7 Done Done Sent false
36680 10020 Acked Read 1 Done Done Sent false
36681 10021 Acked Write 8 Done Done Sent false
36682 10022 Acked Read 10 Done Done Sent false
36683 10023 Acked Read 7 Done Done Sent false
36684 10024 Acked Write 8 Done Done Sent false
36685 10025 Acked Read 4 Done Done Sent false
36686 10026 Acked Write 6 Done Done Sent false
36687 10027 Acked Read 8 Done Done Sent false
36688 10028 Acked Write 7 Done Done Sent false
36689 10029 Acked Read 9 Done Done Sent false
36690 10030 Acked Read 4 Done Done Sent false
36691 10031 Acked Write 6 Done Done Sent false
36692 10032 Acked Flush 0 Done Done Sent false
36693 10033 Acked Read 3 Done Done Sent false
36694 10034 Acked Flush 0 Done Done Sent false
36695 10035 Acked Read 7 Done Done Sent false
36696 10036 Acked Read 2 Done Done Sent false
36697 10037 Acked Read 4 Done Done Sent false
36698 10038 Acked Write 10 Done Done Sent false
36699 10039 Acked Read 9 Done Done Sent false
36700 10040 Acked Write 10 Done Done Sent false
36701 10041 Acked Read 2 Done Done Sent false
36702 10042 Acked Read 1 Done Done Sent false
36703 10043 Acked Read 3 Done Done Sent false
36704 10044 Acked Write 4 Done Done Sent false
36705 10045 Acked Write 6 Done Done Sent false
36706 10046 Acked Read 8 Done Done Sent false
36707 10047 Acked Read 4 Done Done Sent false
36708 10048 Acked Read 10 Done Done Sent false
36709 10049 Acked Flush 0 Done Done Sent false
36710 10050 Acked Write 10 Done Done Sent false
36711 10051 Acked Flush 0 Done Done Sent false
36712 10052 Acked Write 4 Done Done Sent false
36713 10053 Acked Read 8 Done Done Sent false
36714 10054 Acked Read 9 Done Done Sent false
36715 10055 Acked Write 4 Done Done Sent false
36716 STATES DS:0 DS:1 DS:2 TOTAL
36717 Sent 0 0 301 301
36718 Done 301 301 0 602
36719 Skipped 0 0 0 0
36720 Error 0 0 0 0
36721 Last Flush: Some(JobId(10051)) Some(JobId(10051)) Some(JobId(9754))
36722 Downstairs last five completed: 9754 9753 9752 9751 9750
36723 Upstairs last five completed: 10055 10054 10053 10052 10051
36724 CLIENT: Up:0 ds:301 act:2
367252025-03-17T22:27:06.368ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
367262025-03-17T22:27:07.370ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
367272025-03-17T22:27:07.463ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
367282025-03-17T22:27:07.463ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58159) = io task client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
367292025-03-17T22:27:07.464ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
367302025-03-17T22:27:07.464ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
367312025-03-17T22:27:07.464ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
367322025-03-17T22:27:07.464ZINFOcrucible: send last flush ID to this DS: Some(JobId(9754)) = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
367332025-03-17T22:27:07.464ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
367342025-03-17T22:27:07.464ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(9754)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
367352025-03-17T22:27:07.465ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(9754)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
36736 ----------------------------------------------------------------
36737 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
36738 Downstairs last five completed: 10056 10055 10054 10053 10052
36739 Upstairs last five completed: 10056 10055 10054 10053 10052
36740 CLIENT: Up:0 ds:0 act:3
36741 Replay: All jobs finished
36742 Main task has work to do, go find it
36743 got dsc Stop(1)
36744 stop 1
36745 Mar 17 22:27:09.715 INFO request completed, latency_us: 356, response_code: [8820] Got stop action so:false kr:false
36746 204, uri: /stop/cid/1, method: POST, req_id: 462e9241-8eaf-4bb8-a5ca-2a8657006928, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
367472025-03-17T22:27:09.715ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
367482025-03-17T22:27:09.715ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
367492025-03-17T22:27:09.715ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
36750 Mar 17 22:27:09.716 INFO request completed, latency_us: 139, response_code: 200, uri: /state/cid/1, method: GET, req_id: 7da17b0c-7ab7-414e-afd4-f6a444edb0e4, remote_addr: 127.0.0.1:58317{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:27:09.716084413Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1211,"":"io task","client":"1","":"downstairs","session_id":"93f3af30-b26d-4de2-88d0-24ab0f3b1fcf"}
36751 , local_addr: 127.0.0.1:9998
36752 [8820] Exited with: ExitStatus(unix_wait_status(9))
36753 [8820][1] reports Exit
36754 Mar 17 22:27:13.718 INFO request completed, latency_us: 218, response_code: 200, uri: /state/cid/1, method: GET, req_id: 3c7b3730-0762-476e-8b1f-c73715e680e2, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
36755 001/300 Write block 1652 len 40960 data: 3 4 6 6 8 6 7 7 8 7
36756 002/300 Write block 1203 len 28672 data: 3 2 2 2 3 3 4
36757 003/300 Write block 4934 len 12288 data: 7 9 9
36758 004/300 Write block 1662 len 28672 data: 7 5 5 6 6 8 7
36759 005/300 Read block 2951 len 32768
36760 006/300 Write block 4204 len 24576 data: 8 6 6 6 9 9
36761 007/300 Read block 3311 len 32768
36762 008/300 Write block 493 len 4096 data: 7
36763 009/300 Read block 218 len 40960
36764 010/300 Write block 4327 len 28672 data: 9 9 8 7 7 7 6
36765 011/300 Read block 2300 len 36864
36766 012/300 Write block 1747 len 16384 data: 7 7 7 7
36767 013/300 Read block 649 len 8192
36768 014/300 Read block 1329 len 8192
36769 015/300 Read block 4967 len 32768
36770 016/300 Write block 4795 len 40960 data: 5 5 5 5 5 4 6 7 8 7
36771 017/300 Write block 919 len 28672 data: 7 7 7 7 9 7 6
36772 018/300 Read block 1231 len 4096
36773 019/300 Write block 2407 len 4096 data: 8
36774 020/300 Flush
36775 021/300 Read block 558 len 32768
36776 022/300 Read block 4290 len 4096
36777 023/300 Write block 2288 len 8192 data: 7 5
36778 024/300 Read block 4553 len 8192
36779 025/300 Write block 1562 len 16384 data: 7 8 6 4
36780 026/300 Flush
36781 027/300 Read block 2882 len 32768
36782 028/300 Read block 3150 len 20480
36783 029/300 Read block 821 len 20480
36784 030/300 Read block 2437 len 40960
36785 031/300 Flush
36786 032/300 Write block 3651 len 4096 data: 8
36787 033/300 Read block 3571 len 8192
36788 034/300 Read block 2294 len 24576
36789 035/300 Read block 2919 len 28672
36790 036/300 Read block 1082 len 8192
36791 037/300 Flush
36792 038/300 Read block 4685 len 40960
36793 039/300 Flush
36794 040/300 Write block 4066 len 16384 data: 4 4 4 4
36795 041/300 Read block 1935 len 20480
36796 042/300 Read block 2353 len 32768
36797 043/300 Read block 286 len 28672
36798 044/300 Write block 4278 len 12288 data: 5 7 6
36799 045/300 Read block 3609 len 20480
36800 046/300 Flush
36801 047/300 Write block 4064 len 16384 data: 4 4 5 5
36802 048/300 Flush
36803 049/300 Read block 3315 len 24576
36804 050/300 Read block 2620 len 20480
36805 051/300 Read block 3587 len 32768
36806 052/300 Read block 4032 len 36864
36807 053/300 Read block 4919 len 8192
36808 054/300 Write block 3964 len 16384 data: 2 2 2 2
36809 055/300 Read block 3829 len 4096
36810 056/300 Read block 364 len 12288
36811 057/300 Read block 99 len 12288
36812 058/300 Write block 1795 len 40960 data: 9 9 6 6 7 6 5 6 6 7
36813 059/300 Write block 644 len 36864 data: 4 4 6 5 2 3 4 6 5
36814 060/300 Write block 1219 len 16384 data: 6 7 5 5
36815 061/300 Read block 4312 len 24576
36816 062/300 Write block 2313 len 16384 data: 11 12 9 7
36817 063/300 Read block 2795 len 20480
36818 064/300 Read block 3015 len 32768
36819 065/300 Read block 4368 len 24576
36820 066/300 Write block 364 len 4096 data: 6
36821 067/300 Read block 1906 len 32768
36822 068/300 Read block 3081 len 12288
36823 069/300 Write block 4869 len 4096 data: 9
36824 070/300 Write block 4626 len 8192 data: 5 6
36825 071/300 Read block 4424 len 8192
36826 072/300 Flush
36827 073/300 Flush
36828 074/300 Flush
36829 075/300 Read block 3254 len 20480
36830 076/300 Read block 2904 len 8192
36831 077/300 Write block 4954 len 12288 data: 5 6 6
36832 078/300 Read block 2315 len 40960
36833 079/300 Read block 1238 len 28672
36834 080/300 Write block 1322 len 28672 data: 6 6 5 6 8 10 8
36835 081/300 Read block 3130 len 24576
36836 082/300 Write block 374 len 24576 data: 5 6 6 5 5 3
36837 083/300 Write block 2222 len 12288 data: 8 8 9
36838 084/300 Write block 2181 len 24576 data: 5 4 3 3 3 3
36839 085/300 Flush
36840 086/300 Write block 4775 len 40960 data: 5 6 6 6 6 7 7 8 8 7
36841 087/300 Write block 1819 len 12288 data: 4 4 4
36842 088/300 Read block 348 len 16384
36843 089/300 Read block 672 len 4096
36844 090/300 Read block 2421 len 36864
36845 091/300 Write block 156 len 4096 data: 9
36846 092/300 Read block 2804 len 16384
36847 093/300 Write block 4078 len 36864 data: 7 6 4 5 5 6 6 7 6
36848 094/300 Write block 2890 len 20480 data: 4 4 3 3 6
36849 095/300 Read block 1758 len 16384
36850 096/300 Read block 1043 len 32768
36851 097/300 Flush
36852 098/300 Flush
36853 099/300 Read block 2091 len 36864
36854 100/300 Read block 3932 len 20480
36855 101/300 Write block 1846 len 8192 data: 3 3
36856 102/300 Write block 4754 len 36864 data: 5 7 8 8 6 4 6 7 8
36857 103/300 Write block 4511 len 32768 data: 8 7 8 7 7 7 5 5
36858 104/300 Write block 4083 len 8192 data: 7 7
36859 105/300 Write block 2847 len 24576 data: 8 7 7 8 8 8
36860 106/300 Write block 2959 len 24576 data: 9 10 7 8 6 7
36861 107/300 Read block 2978 len 32768
36862 108/300 Write block 4677 len 36864 data: 7 6 5 5 5 5 5 4 2
36863 109/300 Read block 2355 len 32768
36864 110/300 Flush
36865 111/300 Read block 3094 len 28672
36866 112/300 Write block 2484 len 16384 data: 5 4 6 5
36867 113/300 Write block 3968 len 32768 data: 3 3 3 4 5 5 4 5
36868 114/300 Read block 149 len 36864
36869 115/300 Read block 488 len 8192
36870 116/300 Read block 4939 len 12288
36871 117/300 Write block 3209 len 12288 data: 6 6 5
36872 118/300 Read block 211 len 4096
36873 119/300 Flush
36874 120/300 Read block 1131 len 40960
36875 121/300 Read block 1613 len 20480
36876 122/300 Write block 3889 len 32768 data: 4 4 5 5 4 5 6 6
36877 123/300 Write block 4619 len 28672 data: 4 4 4 5 4 5 6
36878 124/300 Read block 4321 len 32768
36879 125/300 Write block 3128 len 4096 data: 8
36880 126/300 Write block 297 len 28672 data: 6 5 3 5 8 8 6
36881 127/300 Read block 646 len 20480
36882 128/300 Write block 979 len 32768 data: 6 6 7 7 6 6 6 5
36883 129/300 Flush
36884 130/300 Read block 2331 len 32768
36885 131/300 Flush
36886 132/300 Write block 1838 len 20480 data: 6 8 7 6 3
36887 133/300 Write block 461 len 16384 data: 5 5 6 7
36888 134/300 Read block 4380 len 36864
36889 135/300 Read block 876 len 40960
36890 136/300 Read block 2208 len 20480
36891 137/300 Read block 2780 len 40960
36892 138/300 Read block 1950 len 24576
36893 139/300 Flush
36894 140/300 Write block 3520 len 20480 data: 9 6 8 9 7
36895 141/300 Read block 4537 len 12288
36896 142/300 Write block 3692 len 32768 data: 8 9 10 10 10 7 7 7
36897 143/300 Read block 41 len 32768
36898 144/300 Read block 2942 len 40960
36899 145/300 Flush
36900 146/300 Read block 3138 len 8192
36901 147/300 Read block 2318 len 12288
36902 148/300 Write block 4136 len 20480 data: 5 6 5 6 7
36903 149/300 Flush
36904 150/300 Read block 2395 len 20480
36905 151/300 Write block 2088 len 12288 data: 6 5 5
36906 152/300 Write block 2203 len 40960 data: 9 7 7 6 5 5 4 4 5 5
36907 153/300 Write block 1697 len 20480 data: 6 7 6 5 4
36908 154/300 Read block 3044 len 28672
36909 155/300 Read block 3318 len 20480
36910 156/300 Write block 2819 len 12288 data: 4 6 6
36911 157/300 Write block 3172 len 8192 data: 5 6
36912 158/300 Read block 145 len 12288
36913 159/300 Read block 476 len 24576
36914 160/300 Write block 2895 len 36864 data: 6 7 8 10 10 12 12 11 7
36915 161/300 Write block 2605 len 32768 data: 10 9 9 9 7 8 7 5
36916 162/300 Read block 797 len 20480
36917 163/300 Read block 130 len 36864
36918 164/300 Flush
36919 165/300 Read block 602 len 16384
36920 166/300 Write block 3578 len 8192 data: 10 9
36921 167/300 Read block 961 len 12288
36922 168/300 Read block 1856 len 8192
36923 169/300 Write block 92 len 32768 data: 4 4 4 5 5 6 7 7
36924 170/300 Flush
36925 171/300 Write block 1671 len 8192 data: 10 8
36926 172/300 Write block 2063 len 24576 data: 6 8 9 10 9 8
36927 173/300 Write block 631 len 24576 data: 3 3 3 4 4 4
36928 174/300 Write block 2734 len 28672 data: 6 8 8 8 7 8 7
36929 175/300 Read block 2531 len 12288
36930 176/300 Write block 2657 len 20480 data: 5 6 6 5 6
36931 177/300 Write block 2530 len 32768 data: 11 11 10 11 9 10 9 8
36932 178/300 Read block 3957 len 36864
36933 179/300 Write block 81 len 8192 data: 7 5
36934 180/300 Write block 2536 len 24576 data: 10 9 8 7 8 8
36935 181/300 Write block 2262 len 16384 data: 10 8 6 5
36936 182/300 Write block 2004 len 12288 data: 2 2 2
36937 183/300 Write block 2366 len 28672 data: 5 5 4 3 4 3 4
36938 184/300 Write block 2688 len 32768 data: 10 8 6 4 4 3 4 6
36939 185/300 Read block 2557 len 12288
36940 186/300 Write block 323 len 16384 data: 7 5 6 5
36941 187/300 Read block 2577 len 32768
36942 188/300 Write block 953 len 8192 data: 6 6
36943 189/300 Write block 2417 len 24576 data: 12 11 10 12 10 10
36944 190/300 Read block 1858 len 32768
36945 191/300 Read block 2547 len 4096
36946 192/300 Read block 1440 len 12288
36947 193/300 Flush
36948 194/300 Read block 4578 len 28672
36949 195/300 Write block 475 len 28672 data: 9 9 9 8 7 6 6
36950 196/300 Write block 225 len 20480 data: 6 5 5 4 3
36951 197/300 Write block 4399 len 12288 data: 3 3 3
36952 198/300 Read block 2870 len 32768
36953 199/300 Write block 4128 len 8192 data: 3 3
36954 200/300 Write block 4614 len 40960 data: 5 5 4 6 5 5 5 5 6 5
36955 201/300 Write block 3207 len 4096 data: 6
36956 202/300 Read block 3751 len 8192
36957 203/300 Read block 1250 len 24576
36958 204/300 Write block 933 len 36864 data: 7 8 9 8 7 5 6 6 7
36959 205/300 Write block 150 len 4096 data: 6
36960 206/300 Read block 1103 len 28672
36961 207/300 Flush
36962 208/300 Read block 4910 len 24576
36963 209/300 Read block 3490 len 12288
36964 210/300 Read block 3217 len 4096
36965 211/300 Read block 1323 len 28672
36966 212/300 Write block 3869 len 36864 data: 4 4 3 4 5 7 8 9 9
36967 213/300 Read block 2333 len 4096
36968 214/300 Read block 3798 len 16384
36969 215/300 Write block 1988 len 16384 data: 7 8 8 8
36970 216/300 Read block 3355 len 28672
36971 217/300 Read block 1963 len 28672
36972 218/300 Read block 1378 len 32768
36973 219/300 Write block 282 len 32768 data: 7 6 6 5 5 6 6 6
36974 220/300 Flush
36975 221/300 Read block 417 len 24576
36976 222/300 Flush
36977 223/300 Write block 4913 len 40960 data: 3 2 2 3 4 4 4 4 6 8
36978 224/300 Read block 3516 len 40960
36979 225/300 Read block 2565 len 40960
36980 226/300 Read block 1569 len 40960
36981 227/300 Write block 2092 len 40960 data: 6 6 5 5 6 5 6 7 7 7
36982 228/300 Read block 3508 len 12288
36983 229/300 Write block 320 len 28672 data: 8 9 7 8 6 7 6
36984 230/300 Write block 2044 len 32768 data: 7 9 10 8 8 8 5 6
36985 231/300 Flush
36986 232/300 Write block 2001 len 28672 data: 2 2 2 3 3 3 2
36987 233/300 Write block 1305 len 40960 data: 3 4 5 6 6 7 7 5 5 5
36988 234/300 Write block 3277 len 32768 data: 5 6 5 7 6 4 6 8
36989 235/300 Write block 2593 len 36864 data: 7 6 6 6 10 10 9 10 9
36990 236/300 Write block 3034 len 20480 data: 4 6 6 7 6
36991 237/300 Read block 475 len 4096
36992 238/300 Write block 4455 len 36864 data: 7 5 6 5 5 4 5 5 6
36993 239/300 Write block 2314 len 4096 data: 13
36994 240/300 Read block 3574 len 24576
36995 241/300 Write block 3471 len 8192 data: 4 6
36996 242/300 Read block 4463 len 24576
36997 243/300 Write block 3881 len 8192 data: 6 5
36998 244/300 Read block 114 len 4096
36999 245/300 Write block 1475 len 36864 data: 10 8 7 8 7 8 7 5 5
37000 246/300 Read block 2504 len 8192
37001 247/300 Write block 4307 len 36864 data: 5 5 5 6 7 7 6 6 6
37002 248/300 Write block 3490 len 12288 data: 8 8 7
37003 249/300 Write block 3167 len 4096 data: 6
37004 250/300 Read block 679 len 20480
37005 251/300 Write block 4440 len 36864 data: 2 2 2 2 5 6 6 7 8
37006 252/300 Write block 4176 len 8192 data: 10 8
37007 253/300 Read block 4525 len 8192
37008 254/300 Read block 1698 len 16384
37009 255/300 Flush
37010 256/300 Read block 701 len 32768
37011 257/300 Write block 3646 len 8192 data: 5 4
37012 258/300 Read block 758 len 20480
37013 259/300 Read block 3994 len 28672
37014 260/300 Write block 1332 len 20480 data: 10 9 9 7 7
37015 261/300 Write block 1989 len 36864 data: 9 9 9 7 5 5 5 4 5
37016 262/300 Read block 2176 len 24576
37017 263/300 Read block 3688 len 40960
37018 264/300 Read block 213 len 8192
37019 265/300 Read block 1239 len 36864
37020 266/300 Read block 4633 len 24576
37021 267/300 Read block 3951 len 20480
37022 268/300 Flush
37023 269/300 Write block 1271 len 20480 data: 7 6 5 4 6
37024 270/300 Write block 946 len 32768 data: 8 9 9 9 7 8 6 7
37025 271/300 Flush
37026 272/300 Read block 3300 len 12288
37027 273/300 Write block 4570 len 4096 data: 7
37028 274/300 Read block 3078 len 36864
37029 275/300 Read block 4592 len 28672
37030 276/300 Write block 3857 len 28672 data: 5 6 6 4 7 6 6
37031 277/300 Read block 2441 len 4096
37032 278/300 Write block 1502 len 12288 data: 5 5 5
37033 279/300 Flush
37034 280/300 Write block 35 len 36864 data: 5 7 8 7 7 9 9 9 9
37035 281/300 Write block 384 len 24576 data: 5 5 6 6 8 9
37036 282/300 Flush
37037 283/300 Read block 3043 len 40960
37038 284/300 Write block 1298 len 40960 data: 7 6 7 6 5 4 4 4 5 6
37039 285/300 Read block 2358 len 12288
37040 286/300 Read block 4389 len 12288
37041 287/300 Write block 1039 len 16384 data: 5 3 4 4
37042 288/300 Write block 1145 len 24576 data: 5 4 4 6 5 5
37043 289/300 Read block 386 len 24576
37044 290/300 Write block 215 len 24576 data: 6 7 8 5 4 7
37045 291/300 Write block 3583 len 32768 data: 6 6 5 5 6 6 9 8
37046 292/300 Read block 1782 len 12288
37047 293/300 Write block 2676 len 8192 data: 4 4
37048 294/300 Write block 555 len 16384 data: 6 3 2 3
37049 295/300 Read block 4887 len 28672
37050 296/300 Write block 4778 len 32768 data: 7 7 8 8 9 9 8 7
37051 297/300 Write block 2908 len 36864 data: 9 8 5 6 6 6 8 7 7
37052 298/300 Write block 3377 len 40960 data: 6 6 4 4 6 7 6 7 6 5
37053 299/300 Write block 922 len 20480 data: 8 10 8 7 5
37054 300/300 Read block 4400 len 32768
37055 301/300 Write block 3594 len 24576 data: 4 4 3 6 6 6
37056 Main task has work to do, go find it
37057 got dsc Start(1)
37058 start 1
37059 Mar 17 22:27:13.953 INFO request completed, latency_us: 243, response_code: 204, uri: /start/cid/1, method: POST, req_id: 2aa0af80-a0d2-4619-837b-fda1baf36a4d, remote_addr: 127.0.0.1:58317, local_addr: 127.0.0.1:9998
37060 Replay: started 1, returned:Ok(())
37061 ----------------------------------------------------------------
37062 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
37063 States: Active Offline Active
37064 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
37065 10057 Acked Write 10 Done Sent Done false
37066 10058 Acked Write 7 Done Sent Done false
37067 10059 Acked Write 3 Done Sent Done false
37068 10060 Acked Write 7 Done Sent Done false
37069 10061 Acked Read 8 Done Sent Done false
37070 10062 Acked Write 6 Done Sent Done false
37071 10063 Acked Read 8 Done Sent Done false
37072 10064 Acked Write 1 Done Sent Done false
37073 10065 Acked Read 10 Done Sent Done false
37074 10066 Acked Write 7 Done Sent Done false
37075 10067 Acked Read 9 Done Sent Done false
37076 10068 Acked Write 4 Done Sent Done false
37077 10069 Acked Read 2 Done Sent Done false
37078 10070 Acked Read 2 Done Sent Done false
37079 10071 Acked Read 8 Done Sent Done false
37080 10072 Acked Write 10 Done Sent Done false
37081 10073 Acked Write 7 Done Sent Done false
37082 10074 Acked Read 1 Done Sent Done false
37083 10075 Acked Write 1 Done Sent Done false
37084 10076 Acked Flush 0 Done Sent Done false
37085 10077 Acked Read 8 Done Sent Done false
37086 10078 Acked Read 1 Done Sent Done false
37087 10079 Acked Write 2 Done Sent Done false
37088 10080 Acked Read 2 Done Sent Done false
37089 10081 Acked Write 4 Done Sent Done false
37090 10082 Acked Flush 0 Done Sent Done false
37091 10083 Acked Read 8 Done Sent Done false
37092 10084 Acked Read 5 Done Sent Done false
37093 10085 Acked Read 5 Done Sent Done false
37094 10086 Acked Read 10 Done Sent Done false
37095 10087 Acked Flush 0 Done Sent Done false
37096 10088 Acked Write 1 Done Sent Done false
37097 10089 Acked Read 2 Done Sent Done false
37098 10090 Acked Read 6 Done Sent Done false
37099 10091 Acked Read 7 Done Sent Done false
37100 10092 Acked Read 2 Done Sent Done false
37101 10093 Acked Flush 0 Done Sent Done false
37102 10094 Acked Read 10 Done Sent Done false
37103 10095 Acked Flush 0 Done Sent Done false
37104 10096 Acked Write 4 Done Sent Done false
37105 10097 Acked Read 5 Done Sent Done false
37106 10098 Acked Read 8 Done Sent Done false
37107 10099 Acked Read 7 Done Sent Done false
37108 10100 Acked Write 3 Done Sent Done false
37109 10101 Acked Read 5 Done Sent Done false
37110 10102 Acked Flush 0 Done Sent Done false
37111 10103 Acked Write 4 Done Sent Done false
37112 10104 Acked Flush 0 Done Sent Done false
37113 10105 Acked Read 6 Done Sent Done false
37114 10106 Acked Read 5 Done Sent Done false
37115 10107 Acked Read 8 Done Sent Done false
37116 10108 Acked Read 9 Done Sent Done false
37117 10109 Acked Read 2 Done Sent Done false
37118 10110 Acked Write 4 Done Sent Done false
37119 10111 Acked Read 1 Done Sent Done false
37120 10112 Acked Read 3 Done Sent Done false
37121 10113 Acked Read 3 Done Sent Done false
37122 10114 Acked Write 10 Done Sent Done false
37123 10115 Acked Write 9 Done Sent Done false
37124 10116 Acked Write 4 Done Sent Done false
37125 10117 Acked Read 6 Done Sent Done false
37126 10118 Acked Write 4 Done Sent Done false
37127 10119 Acked Read 5 Done Sent Done false
37128 10120 Acked Read 8 Done Sent Done false
37129 10121 Acked Read 6 Done Sent Done false
37130 10122 Acked Write 1 Done Sent Done false
37131 10123 Acked Read 8 Done Sent Done false
37132 10124 Acked Read 3 Done Sent Done false
37133 10125 Acked Write 1 Done Sent Done false
37134 10126 Acked Write 2 Done Sent Done false
37135 10127 Acked Read 2 Done Sent Done false
37136 10128 Acked Flush 0 Done Sent Done false
37137 10129 Acked Flush 0 Done Sent Done false
37138 10130 Acked Flush 0 Done Sent Done false
37139 10131 Acked Read 5 Done Sent Done false
37140 10132 Acked Read 2 Done Sent Done false
37141 10133 Acked Write 3 Done Sent Done false
37142 10134 Acked Read 10 Done Sent Done false
37143 10135 Acked Read 7 Done Sent Done false
37144 10136 Acked Write 7 Done Sent Done false
37145 10137 Acked Read 6 Done Sent Done false
37146 10138 Acked Write 6 Done Sent Done false
37147 10139 Acked Write 3 Done Sent Done false
37148 10140 Acked Write 6 Done Sent Done false
37149 10141 Acked Flush 0 Done Sent Done false
37150 10142 Acked Write 10 Done Sent Done false
37151 10143 Acked Write 3 Done Sent Done false
37152 10144 Acked Read 4 Done Sent Done false
37153 10145 Acked Read 1 Done Sent Done false
37154 10146 Acked Read 9 Done Sent Done false
37155 10147 Acked Write 1 Done Sent Done false
37156 10148 Acked Read 4 Done Sent Done false
37157 10149 Acked Write 9 Done Sent Done false
37158 10150 Acked Write 5 Done Sent Done false
37159 10151 Acked Read 4 Done Sent Done false
37160 10152 Acked Read 8 Done Sent Done false
37161 10153 Acked Flush 0 Done Sent Done false
37162 10154 Acked Flush 0 Done Sent Done false
37163 10155 Acked Read 9 Done Sent Done false
37164 10156 Acked Read 5 Done Sent Done false
37165 10157 Acked Write 2 Done Sent Done false
37166 10158 Acked Write 9 Done Sent Done false
37167 10159 Acked Write 8 Done Sent Done false
37168 10160 Acked Write 2 Done Sent Done false
37169 10161 Acked Write 6 Done Sent Done false
37170 10162 Acked Write 6 Done Sent Done false
37171 10163 Acked Read 8 Done Sent Done false
37172 10164 Acked Write 9 Done Sent Done false
37173 10165 Acked Read 8 Done Sent Done false
37174 10166 Acked Flush 0 Done Sent Done false
37175 10167 Acked Read 7 Done Sent Done false
37176 10168 Acked Write 4 Done Sent Done false
37177 10169 Acked Write 8 Done Sent Done false
37178 10170 Acked Read 9 Done Sent Done false
37179 10171 Acked Read 2 Done Sent Done false
37180 10172 Acked Read 3 Done Sent Done false
37181 10173 Acked Write 3 Done Sent Done false
37182 10174 Acked Read 1 Done Sent Done false
37183 10175 Acked Flush 0 Done Sent Done false
37184 10176 Acked Read 10 Done Sent Done false
37185 10177 Acked Read 5 Done Sent Done false
37186 10178 Acked Write 8 Done Sent Done false
37187 10179 Acked Write 7 Done Sent Done false
37188 10180 Acked Read 8 Done Sent Done false
37189 10181 Acked Write 1 Done Sent Done false
37190 10182 Acked Write 7 Done Sent Done false
37191 10183 Acked Read 5 Done Sent Done false
37192 10184 Acked Write 8 Done Sent Done false
37193 10185 Acked Flush 0 Done Sent Done false
37194 10186 Acked Read 8 Done Sent Done false
37195 10187 Acked Flush 0 Done Sent Done false
37196 10188 Acked Write 5 Done Sent Done false
37197 10189 Acked Write 4 Done Sent Done false
37198 10190 Acked Read 9 Done Sent Done false
37199 10191 Acked Read 10 Done Sent Done false
37200 10192 Acked Read 5 Done Sent Done false
37201 10193 Acked Read 10 Done Sent Done false
37202 10194 Acked Read 6 Done Sent Done false
37203 10195 Acked Flush 0 Done Sent Done false
37204 10196 Acked Write 5 Done Sent Done false
37205 10197 Acked Read 3 Done Sent Done false
37206 10198 Acked Write 8 Done Sent Done false
37207 10199 Acked Read 8 Done Sent Done false
37208 10200 Acked Read 10 Done Sent Done false
37209 10201 Acked Flush 0 Done Sent Done false
37210 10202 Acked Read 2 Done Sent Done false
37211 10203 Acked Read 3 Done Sent Done false
37212 10204 Acked Write 5 Done Sent Done false
37213 10205 Acked Flush 0 Done Sent Done false
37214 10206 Acked Read 5 Done Sent Done false
37215 10207 Acked Write 3 Done Sent Done false
37216 10208 Acked Write 10 Done Sent Done false
37217 10209 Acked Write 5 Done Sent Done false
37218 10210 Acked Read 7 Done Sent Done false
37219 10211 Acked Read 5 Done Sent Done false
37220 10212 Acked Write 3 Done Sent Done false
37221 10213 Acked Write 2 Done Sent Done false
37222 10214 Acked Read 3 Done Sent Done false
37223 10215 Acked Read 6 Done Sent Done false
37224 10216 Acked Write 9 Done Sent Done false
37225 10217 Acked Write 8 Done Sent Done false
37226 10218 Acked Read 5 Done Sent Done false
37227 10219 Acked Read 9 Done Sent Done false
37228 10220 Acked Flush 0 Done Sent Done false
37229 10221 Acked Read 4 Done Sent Done false
37230 10222 Acked Write 2 Done Sent Done false
37231 10223 Acked Read 3 Done Sent Done false
37232 10224 Acked Read 2 Done Sent Done false
37233 10225 Acked Write 8 Done Sent Done false
37234 10226 Acked Flush 0 Done Sent Done false
37235 10227 Acked Write 2 Done Sent Done false
37236 10228 Acked Write 6 Done Sent Done false
37237 10229 Acked Write 6 Done Sent Done false
37238 10230 Acked Write 7 Done Sent Done false
37239 10231 Acked Read 3 Done Sent Done false
37240 10232 Acked Write 5 Done Sent Done false
37241 10233 Acked Write 8 Done Sent Done false
37242 10234 Acked Read 9 Done Sent Done false
37243 10235 Acked Write 2 Done Sent Done false
37244 10236 Acked Write 6 Done Sent Done false
37245 10237 Acked Write 4 Done Sent Done false
37246 10238 Acked Write 3 Done Sent Done false
37247 10239 Acked Write 7 Done Sent Done false
37248 10240 Acked Write 8 Done Sent Done false
37249 10241 Acked Read 3 Done Sent Done false
37250 10242 Acked Write 4 Done Sent Done false
37251 10243 Acked Read 8 Done Sent Done false
37252 10244 Acked Write 2 Done Sent Done false
37253 10245 Acked Write 6 Done Sent Done false
37254 10246 Acked Read 8 Done Sent Done false
37255 10247 Acked Read 1 Done Sent Done false
37256 10248 Acked Read 3 Done Sent Done false
37257 10249 Acked Flush 0 Done Sent Done false
37258 10250 Acked Read 7 Done Sent Done false
37259 10251 Acked Write 7 Done Sent Done false
37260 10252 Acked Write 5 Done Sent Done false
37261 10253 Acked Write 3 Done Sent Done false
37262 10254 Acked Read 8 Done Sent Done false
37263 10255 Acked Write 2 Done Sent Done false
37264 10256 Acked Write 10 Done Sent Done false
37265 10257 Acked Write 1 Done Sent Done false
37266 10258 Acked Read 2 Done Sent Done false
37267 10259 Acked Read 6 Done Sent Done false
37268 10260 Acked Write 9 Done Sent Done false
37269 10261 Acked Write 1 Done Sent Done false
37270 10262 Acked Read 7 Done Sent Done false
37271 10263 Acked Flush 0 Done Sent Done false
37272 10264 Acked Read 6 Done Sent Done false
37273 10265 Acked Read 3 Done Sent Done false
37274 10266 Acked Read 1 Done Sent Done false
37275 10267 Acked Read 7 Done Sent Done false
37276 10268 Acked Write 9 Done Sent Done false
37277 10269 Acked Read 1 Done Sent Done false
37278 10270 Acked Read 4 Done Sent Done false
37279 10271 Acked Write 4 Done Sent Done false
37280 10272 Acked Read 7 Done Sent Done false
37281 10273 Acked Read 7 Done Sent Done false
37282 10274 Acked Read 8 Done Sent Done false
37283 10275 Acked Write 8 Done Sent Done false
37284 10276 Acked Flush 0 Done Sent Done false
37285 10277 Acked Read 6 Done Sent Done false
37286 10278 Acked Flush 0 Done Sent Done false
37287 10279 Acked Write 10 Done Sent Done false
37288 10280 Acked Read 10 Done Sent Done false
37289 10281 Acked Read 10 Done Sent Done false
37290 10282 Acked Read 10 Done Sent Done false
37291 10283 Acked Write 10 Done Sent Done false
37292 10284 Acked Read 3 Done Sent Done false
37293 10285 Acked Write 7 Done Sent Done false
37294 10286 Acked Write 8 Done Sent Done false
37295 10287 Acked Flush 0 Done Sent Done false
37296 10288 Acked Write 7 Done Sent Done false
37297 10289 Acked Write 10 Done Sent Done false
37298 10290 Acked Write 8 Done Sent Done false
37299 10291 Acked Write 9 Done Sent Done false
37300 10292 Acked Write 5 Done Sent Done false
37301 10293 Acked Read 1 Done Sent Done false
37302 10294 Acked Write 9 Done Sent Done false
37303 10295 Acked Write 1 Done Sent Done false
37304 10296 Acked Read 6 Done Sent Done false
37305 10297 Acked Write 2 Done Sent Done false
37306 10298 Acked Read 6 Done Sent Done false
37307 10299 Acked Write 2 Done Sent Done false
37308 10300 Acked Read 1 Done Sent Done false
37309 10301 Acked Write 9 Done Sent Done false
37310 10302 Acked Read 2 Done Sent Done false
37311 10303 Acked Write 9 Done Sent Done false
37312 10304 Acked Write 3 Done Sent Done false
37313 10305 Acked Write 1 Done Sent Done false
37314 10306 Acked Read 5 Done Sent Done false
37315 10307 Acked Write 9 Done Sent Done false
37316 10308 Acked Write 2 Done Sent Done false
37317 10309 Acked Read 2 Done Sent Done false
37318 10310 Acked Read 4 Done Sent Done false
37319 10311 Acked Flush 0 Done Sent Done false
37320 10312 Acked Read 8 Done Sent Done false
37321 10313 Acked Write 2 Done Sent Done false
37322 10314 Acked Read 5 Done Sent Done false
37323 10315 Acked Read 7 Done Sent Done false
37324 10316 Acked Write 5 Done Sent Done false
37325 10317 Acked Write 9 Done Sent Done false
37326 10318 Acked Read 6 Done Sent Done false
37327 10319 Acked Read 10 Done Sent Done false
37328 10320 Acked Read 2 Done Sent Done false
37329 10321 Acked Read 9 Done Sent Done false
37330 10322 Acked Read 6 Done Sent Done false
37331 10323 Acked Read 5 Done Sent Done false
37332 10324 Acked Flush 0 Done Sent Done false
37333 10325 Acked Write 5 Done Sent Done false
37334 10326 Acked Write 8 Done Sent Done false
37335 10327 Acked Flush 0 Done Sent Done false
37336 10328 Acked Read 3 Done Sent Done false
37337 10329 Acked Write 1 Done Sent Done false
37338 10330 Acked Read 9 Done Sent Done false
37339 10331 Acked Read 7 Done Sent Done false
37340 10332 Acked Write 7 Done Sent Done false
37341 10333 Acked Read 1 Done Sent Done false
37342 10334 Acked Write 3 Done Sent Done false
37343 10335 Acked Flush 0 Done Sent Done false
37344 10336 Acked Write 9 Done Sent Done false
37345 10337 Acked Write 6 Done Sent Done false
37346 10338 Acked Flush 0 Done Sent Done false
37347 10339 Acked Read 10 Done Sent Done false
37348 10340 Acked Write 10 Done Sent Done false
37349 10341 Acked Read 3 Done Sent Done false
37350 10342 Acked Read 3 Done Sent Done false
37351 10343 Acked Write 4 Done Sent Done false
37352 10344 Acked Write 6 Done Sent Done false
37353 10345 Acked Read 6 Done Sent Done false
37354 10346 Acked Write 6 Done Sent Done false
37355 10347 Acked Write 8 Done Sent Done false
37356 10348 Acked Read 3 Done Sent Done false
37357 10349 Acked Write 2 Done Sent Done false
37358 10350 Acked Write 4 Done Sent Done false
37359 10351 Acked Read 7 Done Sent Done false
37360 10352 Acked Write 8 Done Sent Done false
37361 10353 Acked Write 9 Done Sent Done false
37362 10354 Acked Write 10 Done Sent Done false
37363 10355 Acked Write 5 Done Sent Done false
37364 10356 Acked Read 8 Done Sent Done false
37365 10357 Acked Write 6 Done Sent Done false
37366 STATES DS:0 DS:1 DS:2 TOTAL
37367 Sent 0 301 0 301
37368 Done 301 0 301 602
37369 Skipped 0 0 0 0
37370 Error 0 0 0 0
37371 Last Flush: Some(JobId(10338)) Some(JobId(10056)) Some(JobId(10338))
37372 Downstairs last five completed: 10056 10055 10054 10053 10052
37373 Upstairs last five completed: 10357 10356 10355 10354 10353
37374 CLIENT: Up:0 ds:301 act:2
373752025-03-17T22:27:14.456ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
373762025-03-17T22:27:15.384ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
37377 [8820] got start action
37378 [8820] I am going to restart
37379 Starting downstairs at port 8820
37380 Make output file at "/tmp/dsc/downstairs-8820.txt"
37381 [8820][1] reports Starting
37382 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1242)
37383 [8820][1] reports Running
373842025-03-17T22:27:16.385ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
373852025-03-17T22:27:17.387ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
37386 ----------------------------------------------------------------
37387 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
37388 States: Active Offline Active
37389 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
37390 10057 Acked Write 10 Done Sent Done false
37391 10058 Acked Write 7 Done Sent Done false
37392 10059 Acked Write 3 Done Sent Done false
37393 10060 Acked Write 7 Done Sent Done false
37394 10061 Acked Read 8 Done Sent Done false
37395 10062 Acked Write 6 Done Sent Done false
37396 10063 Acked Read 8 Done Sent Done false
37397 10064 Acked Write 1 Done Sent Done false
37398 10065 Acked Read 10 Done Sent Done false
37399 10066 Acked Write 7 Done Sent Done false
37400 10067 Acked Read 9 Done Sent Done false
37401 10068 Acked Write 4 Done Sent Done false
37402 10069 Acked Read 2 Done Sent Done false
37403 10070 Acked Read 2 Done Sent Done false
37404 10071 Acked Read 8 Done Sent Done false
37405 10072 Acked Write 10 Done Sent Done false
37406 10073 Acked Write 7 Done Sent Done false
37407 10074 Acked Read 1 Done Sent Done false
37408 10075 Acked Write 1 Done Sent Done false
37409 10076 Acked Flush 0 Done Sent Done false
37410 10077 Acked Read 8 Done Sent Done false
37411 10078 Acked Read 1 Done Sent Done false
37412 10079 Acked Write 2 Done Sent Done false
37413 10080 Acked Read 2 Done Sent Done false
37414 10081 Acked Write 4 Done Sent Done false
37415 10082 Acked Flush 0 Done Sent Done false
37416 10083 Acked Read 8 Done Sent Done false
37417 10084 Acked Read 5 Done Sent Done false
37418 10085 Acked Read 5 Done Sent Done false
37419 10086 Acked Read 10 Done Sent Done false
37420 10087 Acked Flush 0 Done Sent Done false
37421 10088 Acked Write 1 Done Sent Done false
37422 10089 Acked Read 2 Done Sent Done false
37423 10090 Acked Read 6 Done Sent Done false
37424 10091 Acked Read 7 Done Sent Done false
37425 10092 Acked Read 2 Done Sent Done false
37426 10093 Acked Flush 0 Done Sent Done false
37427 10094 Acked Read 10 Done Sent Done false
37428 10095 Acked Flush 0 Done Sent Done false
37429 10096 Acked Write 4 Done Sent Done false
37430 10097 Acked Read 5 Done Sent Done false
37431 10098 Acked Read 8 Done Sent Done false
37432 10099 Acked Read 7 Done Sent Done false
37433 10100 Acked Write 3 Done Sent Done false
37434 10101 Acked Read 5 Done Sent Done false
37435 10102 Acked Flush 0 Done Sent Done false
37436 10103 Acked Write 4 Done Sent Done false
37437 10104 Acked Flush 0 Done Sent Done false
37438 10105 Acked Read 6 Done Sent Done false
37439 10106 Acked Read 5 Done Sent Done false
37440 10107 Acked Read 8 Done Sent Done false
37441 10108 Acked Read 9 Done Sent Done false
37442 10109 Acked Read 2 Done Sent Done false
37443 10110 Acked Write 4 Done Sent Done false
37444 10111 Acked Read 1 Done Sent Done false
37445 10112 Acked Read 3 Done Sent Done false
37446 10113 Acked Read 3 Done Sent Done false
37447 10114 Acked Write 10 Done Sent Done false
37448 10115 Acked Write 9 Done Sent Done false
37449 10116 Acked Write 4 Done Sent Done false
37450 10117 Acked Read 6 Done Sent Done false
37451 10118 Acked Write 4 Done Sent Done false
37452 10119 Acked Read 5 Done Sent Done false
37453 10120 Acked Read 8 Done Sent Done false
37454 10121 Acked Read 6 Done Sent Done false
37455 10122 Acked Write 1 Done Sent Done false
37456 10123 Acked Read 8 Done Sent Done false
37457 10124 Acked Read 3 Done Sent Done false
37458 10125 Acked Write 1 Done Sent Done false
37459 10126 Acked Write 2 Done Sent Done false
37460 10127 Acked Read 2 Done Sent Done false
37461 10128 Acked Flush 0 Done Sent Done false
37462 10129 Acked Flush 0 Done Sent Done false
37463 10130 Acked Flush 0 Done Sent Done false
37464 10131 Acked Read 5 Done Sent Done false
37465 10132 Acked Read 2 Done Sent Done false
37466 10133 Acked Write 3 Done Sent Done false
37467 10134 Acked Read 10 Done Sent Done false
37468 10135 Acked Read 7 Done Sent Done false
37469 10136 Acked Write 7 Done Sent Done false
37470 10137 Acked Read 6 Done Sent Done false
37471 10138 Acked Write 6 Done Sent Done false
37472 10139 Acked Write 3 Done Sent Done false
37473 10140 Acked Write 6 Done Sent Done false
37474 10141 Acked Flush 0 Done Sent Done false
37475 10142 Acked Write 10 Done Sent Done false
37476 10143 Acked Write 3 Done Sent Done false
37477 10144 Acked Read 4 Done Sent Done false
37478 10145 Acked Read 1 Done Sent Done false
37479 10146 Acked Read 9 Done Sent Done false
37480 10147 Acked Write 1 Done Sent Done false
37481 10148 Acked Read 4 Done Sent Done false
37482 10149 Acked Write 9 Done Sent Done false
37483 10150 Acked Write 5 Done Sent Done false
37484 10151 Acked Read 4 Done Sent Done false
37485 10152 Acked Read 8 Done Sent Done false
37486 10153 Acked Flush 0 Done Sent Done false
37487 10154 Acked Flush 0 Done Sent Done false
37488 10155 Acked Read 9 Done Sent Done false
37489 10156 Acked Read 5 Done Sent Done false
37490 10157 Acked Write 2 Done Sent Done false
37491 10158 Acked Write 9 Done Sent Done false
37492 10159 Acked Write 8 Done Sent Done false
37493 10160 Acked Write 2 Done Sent Done false
37494 10161 Acked Write 6 Done Sent Done false
37495 10162 Acked Write 6 Done Sent Done false
37496 10163 Acked Read 8 Done Sent Done false
37497 10164 Acked Write 9 Done Sent Done false
37498 10165 Acked Read 8 Done Sent Done false
37499 10166 Acked Flush 0 Done Sent Done false
37500 10167 Acked Read 7 Done Sent Done false
37501 10168 Acked Write 4 Done Sent Done false
37502 10169 Acked Write 8 Done Sent Done false
37503 10170 Acked Read 9 Done Sent Done false
37504 10171 Acked Read 2 Done Sent Done false
37505 10172 Acked Read 3 Done Sent Done false
37506 10173 Acked Write 3 Done Sent Done false
37507 10174 Acked Read 1 Done Sent Done false
37508 10175 Acked Flush 0 Done Sent Done false
37509 10176 Acked Read 10 Done Sent Done false
37510 10177 Acked Read 5 Done Sent Done false
37511 10178 Acked Write 8 Done Sent Done false
37512 10179 Acked Write 7 Done Sent Done false
37513 10180 Acked Read 8 Done Sent Done false
37514 10181 Acked Write 1 Done Sent Done false
37515 10182 Acked Write 7 Done Sent Done false
37516 10183 Acked Read 5 Done Sent Done false
37517 10184 Acked Write 8 Done Sent Done false
37518 10185 Acked Flush 0 Done Sent Done false
37519 10186 Acked Read 8 Done Sent Done false
37520 10187 Acked Flush 0 Done Sent Done false
37521 10188 Acked Write 5 Done Sent Done false
37522 10189 Acked Write 4 Done Sent Done false
37523 10190 Acked Read 9 Done Sent Done false
37524 10191 Acked Read 10 Done Sent Done false
37525 10192 Acked Read 5 Done Sent Done false
37526 10193 Acked Read 10 Done Sent Done false
37527 10194 Acked Read 6 Done Sent Done false
37528 10195 Acked Flush 0 Done Sent Done false
37529 10196 Acked Write 5 Done Sent Done false
37530 10197 Acked Read 3 Done Sent Done false
37531 10198 Acked Write 8 Done Sent Done false
37532 10199 Acked Read 8 Done Sent Done false
37533 10200 Acked Read 10 Done Sent Done false
37534 10201 Acked Flush 0 Done Sent Done false
37535 10202 Acked Read 2 Done Sent Done false
37536 10203 Acked Read 3 Done Sent Done false
37537 10204 Acked Write 5 Done Sent Done false
37538 10205 Acked Flush 0 Done Sent Done false
37539 10206 Acked Read 5 Done Sent Done false
37540 10207 Acked Write 3 Done Sent Done false
37541 10208 Acked Write 10 Done Sent Done false
37542 10209 Acked Write 5 Done Sent Done false
37543 10210 Acked Read 7 Done Sent Done false
37544 10211 Acked Read 5 Done Sent Done false
37545 10212 Acked Write 3 Done Sent Done false
37546 10213 Acked Write 2 Done Sent Done false
37547 10214 Acked Read 3 Done Sent Done false
37548 10215 Acked Read 6 Done Sent Done false
37549 10216 Acked Write 9 Done Sent Done false
37550 10217 Acked Write 8 Done Sent Done false
37551 10218 Acked Read 5 Done Sent Done false
37552 10219 Acked Read 9 Done Sent Done false
37553 10220 Acked Flush 0 Done Sent Done false
37554 10221 Acked Read 4 Done Sent Done false
37555 10222 Acked Write 2 Done Sent Done false
37556 10223 Acked Read 3 Done Sent Done false
37557 10224 Acked Read 2 Done Sent Done false
37558 10225 Acked Write 8 Done Sent Done false
37559 10226 Acked Flush 0 Done Sent Done false
37560 10227 Acked Write 2 Done Sent Done false
37561 10228 Acked Write 6 Done Sent Done false
37562 10229 Acked Write 6 Done Sent Done false
37563 10230 Acked Write 7 Done Sent Done false
37564 10231 Acked Read 3 Done Sent Done false
37565 10232 Acked Write 5 Done Sent Done false
37566 10233 Acked Write 8 Done Sent Done false
37567 10234 Acked Read 9 Done Sent Done false
37568 10235 Acked Write 2 Done Sent Done false
37569 10236 Acked Write 6 Done Sent Done false
37570 10237 Acked Write 4 Done Sent Done false
37571 10238 Acked Write 3 Done Sent Done false
37572 10239 Acked Write 7 Done Sent Done false
37573 10240 Acked Write 8 Done Sent Done false
37574 10241 Acked Read 3 Done Sent Done false
37575 10242 Acked Write 4 Done Sent Done false
37576 10243 Acked Read 8 Done Sent Done false
37577 10244 Acked Write 2 Done Sent Done false
37578 10245 Acked Write 6 Done Sent Done false
37579 10246 Acked Read 8 Done Sent Done false
37580 10247 Acked Read 1 Done Sent Done false
37581 10248 Acked Read 3 Done Sent Done false
37582 10249 Acked Flush 0 Done Sent Done false
37583 10250 Acked Read 7 Done Sent Done false
37584 10251 Acked Write 7 Done Sent Done false
37585 10252 Acked Write 5 Done Sent Done false
37586 10253 Acked Write 3 Done Sent Done false
37587 10254 Acked Read 8 Done Sent Done false
37588 10255 Acked Write 2 Done Sent Done false
37589 10256 Acked Write 10 Done Sent Done false
37590 10257 Acked Write 1 Done Sent Done false
37591 10258 Acked Read 2 Done Sent Done false
37592 10259 Acked Read 6 Done Sent Done false
37593 10260 Acked Write 9 Done Sent Done false
37594 10261 Acked Write 1 Done Sent Done false
37595 10262 Acked Read 7 Done Sent Done false
37596 10263 Acked Flush 0 Done Sent Done false
37597 10264 Acked Read 6 Done Sent Done false
37598 10265 Acked Read 3 Done Sent Done false
37599 10266 Acked Read 1 Done Sent Done false
37600 10267 Acked Read 7 Done Sent Done false
37601 10268 Acked Write 9 Done Sent Done false
37602 10269 Acked Read 1 Done Sent Done false
37603 10270 Acked Read 4 Done Sent Done false
37604 10271 Acked Write 4 Done Sent Done false
37605 10272 Acked Read 7 Done Sent Done false
37606 10273 Acked Read 7 Done Sent Done false
37607 10274 Acked Read 8 Done Sent Done false
37608 10275 Acked Write 8 Done Sent Done false
37609 10276 Acked Flush 0 Done Sent Done false
37610 10277 Acked Read 6 Done Sent Done false
37611 10278 Acked Flush 0 Done Sent Done false
37612 10279 Acked Write 10 Done Sent Done false
37613 10280 Acked Read 10 Done Sent Done false
37614 10281 Acked Read 10 Done Sent Done false
37615 10282 Acked Read 10 Done Sent Done false
37616 10283 Acked Write 10 Done Sent Done false
37617 10284 Acked Read 3 Done Sent Done false
37618 10285 Acked Write 7 Done Sent Done false
37619 10286 Acked Write 8 Done Sent Done false
37620 10287 Acked Flush 0 Done Sent Done false
37621 10288 Acked Write 7 Done Sent Done false
37622 10289 Acked Write 10 Done Sent Done false
37623 10290 Acked Write 8 Done Sent Done false
37624 10291 Acked Write 9 Done Sent Done false
37625 10292 Acked Write 5 Done Sent Done false
37626 10293 Acked Read 1 Done Sent Done false
37627 10294 Acked Write 9 Done Sent Done false
37628 10295 Acked Write 1 Done Sent Done false
37629 10296 Acked Read 6 Done Sent Done false
37630 10297 Acked Write 2 Done Sent Done false
37631 10298 Acked Read 6 Done Sent Done false
37632 10299 Acked Write 2 Done Sent Done false
37633 10300 Acked Read 1 Done Sent Done false
37634 10301 Acked Write 9 Done Sent Done false
37635 10302 Acked Read 2 Done Sent Done false
37636 10303 Acked Write 9 Done Sent Done false
37637 10304 Acked Write 3 Done Sent Done false
37638 10305 Acked Write 1 Done Sent Done false
37639 10306 Acked Read 5 Done Sent Done false
37640 10307 Acked Write 9 Done Sent Done false
37641 10308 Acked Write 2 Done Sent Done false
37642 10309 Acked Read 2 Done Sent Done false
37643 10310 Acked Read 4 Done Sent Done false
37644 10311 Acked Flush 0 Done Sent Done false
37645 10312 Acked Read 8 Done Sent Done false
37646 10313 Acked Write 2 Done Sent Done false
37647 10314 Acked Read 5 Done Sent Done false
37648 10315 Acked Read 7 Done Sent Done false
37649 10316 Acked Write 5 Done Sent Done false
37650 10317 Acked Write 9 Done Sent Done false
37651 10318 Acked Read 6 Done Sent Done false
37652 10319 Acked Read 10 Done Sent Done false
37653 10320 Acked Read 2 Done Sent Done false
37654 10321 Acked Read 9 Done Sent Done false
37655 10322 Acked Read 6 Done Sent Done false
37656 10323 Acked Read 5 Done Sent Done false
37657 10324 Acked Flush 0 Done Sent Done false
37658 10325 Acked Write 5 Done Sent Done false
37659 10326 Acked Write 8 Done Sent Done false
37660 10327 Acked Flush 0 Done Sent Done false
37661 10328 Acked Read 3 Done Sent Done false
37662 10329 Acked Write 1 Done Sent Done false
37663 10330 Acked Read 9 Done Sent Done false
37664 10331 Acked Read 7 Done Sent Done false
37665 10332 Acked Write 7 Done Sent Done false
37666 10333 Acked Read 1 Done Sent Done false
37667 10334 Acked Write 3 Done Sent Done false
37668 10335 Acked Flush 0 Done Sent Done false
37669 10336 Acked Write 9 Done Sent Done false
37670 10337 Acked Write 6 Done Sent Done false
37671 10338 Acked Flush 0 Done Sent Done false
37672 10339 Acked Read 10 Done Sent Done false
37673 10340 Acked Write 10 Done Sent Done false
37674 10341 Acked Read 3 Done Sent Done false
37675 10342 Acked Read 3 Done Sent Done false
37676 10343 Acked Write 4 Done Sent Done false
37677 10344 Acked Write 6 Done Sent Done false
37678 10345 Acked Read 6 Done Sent Done false
37679 10346 Acked Write 6 Done Sent Done false
37680 10347 Acked Write 8 Done Sent Done false
37681 10348 Acked Read 3 Done Sent Done false
37682 10349 Acked Write 2 Done Sent Done false
37683 10350 Acked Write 4 Done Sent Done false
37684 10351 Acked Read 7 Done Sent Done false
37685 10352 Acked Write 8 Done Sent Done false
37686 10353 Acked Write 9 Done Sent Done false
37687 10354 Acked Write 10 Done Sent Done false
37688 10355 Acked Write 5 Done Sent Done false
37689 10356 Acked Read 8 Done Sent Done false
37690 10357 Acked Write 6 Done Sent Done false
37691 STATES DS:0 DS:1 DS:2 TOTAL
37692 Sent 0 301 0 301
37693 Done 301 0 301 602
37694 Skipped 0 0 0 0
37695 Error 0 0 0 0
37696 Last Flush: Some(JobId(10338)) Some(JobId(10056)) Some(JobId(10338))
37697 Downstairs last five completed: 10056 10055 10054 10053 10052
37698 Upstairs last five completed: 10357 10356 10355 10354 10353
37699 CLIENT: Up:0 ds:301 act:2
377002025-03-17T22:27:18.458ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
377012025-03-17T22:27:19.391ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
377022025-03-17T22:27:19.717ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
377032025-03-17T22:27:19.717ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:45573) = io task client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
377042025-03-17T22:27:19.718ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
377052025-03-17T22:27:19.719ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
377062025-03-17T22:27:19.719ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
377072025-03-17T22:27:19.719ZINFOcrucible: send last flush ID to this DS: Some(JobId(10056)) = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
377082025-03-17T22:27:19.719ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
377092025-03-17T22:27:19.719ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(10056)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
377102025-03-17T22:27:19.719ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(10056)) = downstairs session_id = 93f3af30-b26d-4de2-88d0-24ab0f3b1fcf
37711 ----------------------------------------------------------------
37712 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
37713 Downstairs last five completed: 10358 10357 10356 10355 10354
37714 Upstairs last five completed: 10358 10357 10356 10355 10354
37715 CLIENT: Up:0 ds:0 act:3
37716 Replay: All jobs finished
37717 Test replay has completed
37718 Wrote out file "/tmp/test_replay_verify.log"
37719 CLIENT: Tests done. All submitted work has been ACK'd
37720 ----------------------------------------------------------------
37721 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
37722 Downstairs last five completed: 10358 10357 10356 10355 10354
37723 Upstairs last five completed: 10358 10357 10356 10355 10354
37724 CLIENT: Up:0 ds:0 act:3
37725 CLIENT: All jobs finished, all DS active.
37726 Replays:30 time: 8:24
37727 Do final verify
37728 Mar 17 22:27:22.033 INFO accepted connection, remote_addr: 127.0.0.1:50764, local_addr: 127.0.0.1:9998
37729 Mar 17 22:27:22.034 INFO request completed, latency_us: 215, response_code: 200, uri: /regioninfo, method: GET, req_id: 67928012-25d9-4458-b168-0ee5b174db3f, remote_addr: 127.0.0.1:50764, local_addr: 127.0.0.1:9998
377302025-03-17T22:27:22.034ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
37731 Mar 17 22:27:22.035 INFO request completed, latency_us: 91, response_code: 200, uri: /regioncount, method: GET, req_id: 42ac2bc9-d1d5-403f-8a89-f096575b73e5, remote_addr: 127.0.0.1:50764, local_addr: 127.0.0.1:9998
377322025-03-17T22:27:22.035ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
37733 Mar 17 22:27:22.035 INFO request completed, latency_us: 110, response_code: 200, uri: /port/cid/0, method: GET, req_id: 6e7b1627-adfb-4897-92b7-d16d447957a2, remote_addr: 127.0.0.1:50764, local_addr: 127.0.0.1:9998
37734 Mar 17 22:27:22.036 INFO request completed, latency_us: 91, response_code: 200, uri: /port/cid/1, method: GET, req_id: 3b1c956f-6a56-4a05-ab3b-310068345876, remote_addr: 127.0.0.1:50764, local_addr: 127.0.0.1:9998
37735 Mar 17 22:27:22.036 INFO request completed, latency_us: 89, response_code: 200, uri: /port/cid/2, method: GET, req_id: 19bea49b-0bd9-4980-b129-8afebd7d28a0, remote_addr: 127.0.0.1:50764, local_addr: 127.0.0.1:9998
377362025-03-17T22:27:22.036ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
377372025-03-17T22:27:22.037ZINFOcrucible: Upstairs starts
377382025-03-17T22:27:22.037ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "acc6b11fa639390710fd9a0449e05c901b048ed2", git_commit_timestamp: "2025-03-17T22:02:02.000000000Z", git_branch: "renovate/itertools-0.x", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
377392025-03-17T22:27:22.037ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
377402025-03-17T22:27:22.037ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
377412025-03-17T22:27:22.037ZINFOcrucible: Crucible c3f88421-3db7-49d9-bf7d-b274a2b5ece3 has session id: 376ae193-af42-46eb-981f-19dac30f53e5 session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377422025-03-17T22:27:22.037ZINFOcrucible: Upstairs opts: Upstairs UUID: c3f88421-3db7-49d9-bf7d-b274a2b5ece3, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377432025-03-17T22:27:22.037ZINFOcrucible: Crucible stats registered with UUID: c3f88421-3db7-49d9-bf7d-b274a2b5ece3 session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377442025-03-17T22:27:22.037ZWARNcrucible: could not find Downstairs address for Nexus session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377452025-03-17T22:27:22.038ZINFOcrucible: Downstairs in volume = 3 task = crutest
377462025-03-17T22:27:22.038ZINFOcrucible: The guest has requested activation with gen:3
377472025-03-17T22:27:22.038ZINFOcrucible: c3f88421-3db7-49d9-bf7d-b274a2b5ece3 active request set session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377482025-03-17T22:27:22.038ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377492025-03-17T22:27:22.038ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377502025-03-17T22:27:22.038ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377512025-03-17T22:27:22.038ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377522025-03-17T22:27:22.038ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377532025-03-17T22:27:22.038ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377542025-03-17T22:27:22.038ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377552025-03-17T22:27:22.038ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377562025-03-17T22:27:22.038ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377572025-03-17T22:27:22.038ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:61279) = io task client = 2 session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377582025-03-17T22:27:22.038ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:41248) = io task client = 0 session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377592025-03-17T22:27:22.039ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:42094) = io task client = 1 session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377602025-03-17T22:27:22.039ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377612025-03-17T22:27:22.039ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377622025-03-17T22:27:22.039ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377632025-03-17T22:27:22.040ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377642025-03-17T22:27:22.040ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377652025-03-17T22:27:22.040ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377662025-03-17T22:27:22.041ZINFOcrucible: Waiting for more clients to be ready session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377672025-03-17T22:27:22.041ZINFOcrucible: Waiting for more clients to be ready session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377682025-03-17T22:27:22.041ZINFOcrucible: [0]R flush_numbers[0..12]: [973, 965, 974, 973, 965, 974, 964, 932, 930, 974, 974, 974] = downstairs session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377692025-03-17T22:27:22.041ZINFOcrucible: [0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377702025-03-17T22:27:22.041ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377712025-03-17T22:27:22.041ZINFOcrucible: [1]R flush_numbers[0..12]: [973, 965, 974, 973, 965, 974, 964, 932, 930, 974, 974, 974] = downstairs session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377722025-03-17T22:27:22.041ZINFOcrucible: [1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377732025-03-17T22:27:22.041ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377742025-03-17T22:27:22.041ZINFOcrucible: [2]R flush_numbers[0..12]: [973, 965, 974, 973, 965, 974, 964, 932, 930, 974, 974, 974] = downstairs session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377752025-03-17T22:27:22.041ZINFOcrucible: [2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377762025-03-17T22:27:22.041ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377772025-03-17T22:27:22.041ZINFOcrucible: Max found gen is 3 = downstairs session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377782025-03-17T22:27:22.041ZINFOcrucible: Generation requested: 3 >= found:3 = downstairs session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377792025-03-17T22:27:22.041ZINFOcrucible: Next flush: 975 = downstairs session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377802025-03-17T22:27:22.041ZINFOcrucible: All extents match = downstairs session_id = 376ae193-af42-46eb-981f-19dac30f53e5
37781 {"msg":"No downstairs reconciliation required","Wait for a query_work_queue command to finish before sending IO
37782 v":0,"name":"crucible","level":30,"time":"2025-03-17T22:27:22.041567727Z","hostname":"w-01JPK2XSVRF7343QRZWA5N22R2","pid":1247,"session_id":"376ae193-af42-46eb-981f-19dac30f53e5"}
377832025-03-17T22:27:22.041ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377842025-03-17T22:27:22.041ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377852025-03-17T22:27:22.041ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377862025-03-17T22:27:22.041ZINFOcrucible: All required reconciliation work is completed session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377872025-03-17T22:27:22.041ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377882025-03-17T22:27:22.041ZINFOcrucible: c3f88421-3db7-49d9-bf7d-b274a2b5ece3 is now active with session: 376ae193-af42-46eb-981f-19dac30f53e5 session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377892025-03-17T22:27:22.041ZINFOcrucible: Set Active after no reconciliation session_id = 376ae193-af42-46eb-981f-19dac30f53e5
377902025-03-17T22:27:22.041ZINFOcrucible: The guest has finished waiting for activation with:3
377912025-03-17T22:27:22.041ZINFOcrucible: Activated sub_volume c3f88421-3db7-49d9-bf7d-b274a2b5ece3
37792 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
37793 Loading write count information from file "/tmp/test_replay_verify.log"
37794 Read and Verify all blocks (0..5000 range:false)
37795 Wrote out file "/tmp/test_replay_verify.log"
37796 Verify test completed
37797 Wrote out file "/tmp/test_replay_verify.log"
37798 CLIENT: Tests done. All submitted work has been ACK'd
37799 ----------------------------------------------------------------
37800 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:20
37801 States: Active Active Active
37802 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
37803 1000 Acked Read 256 Done Done Done false
37804 1001 Acked Read 256 Done Done Done false
37805 1002 Acked Read 256 Done Done Done false
37806 1003 Acked Read 256 Done Done Done false
37807 1004 Acked Read 256 Done Done Done false
37808 1005 Acked Read 256 Done Done Done false
37809 1006 Acked Read 256 Done Done Done false
37810 1007 Acked Read 256 Done Done Done false
37811 1008 Acked Read 256 Done Done Done false
37812 1009 Acked Read 256 Done Done Done false
37813 1010 Acked Read 256 Done Done Done false
37814 1011 Acked Read 256 Done Done Done false
37815 1012 Acked Read 256 Done Done Done false
37816 1013 Acked Read 256 Done Done Done false
37817 1014 Acked Read 256 Done Done Done false
37818 1015 Acked Read 256 Done Done Done false
37819 1016 Acked Read 256 Done Done Done false
37820 1017 Acked Read 256 Done Done Done false
37821 1018 Acked Read 256 Done Done Done false
37822 1019 Acked Read 136 Done Done Done false
37823 STATES DS:0 DS:1 DS:2 TOTAL
37824 Sent 0 0 0 0
37825 Done 20 20 20 60
37826 Skipped 0 0 0 0
37827 Error 0 0 0 0
37828 Last Flush: None None None
37829 Downstairs last five completed:
37830 Upstairs last five completed: 1019 1018 1017 1016 1015
37831 CLIENT: Up:0 ds:20 act:3
37832 ----------------------------------------------------------------
37833 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0
37834 Downstairs last five completed: 1020 1019 1018 1017 1016
37835 Upstairs last five completed: 1020 1019 1018 1017 1016
37836 CLIENT: Up:0 ds:0 act:3
37837 CLIENT: All crucible jobs finished, exiting program
378382025-03-17T22:27:26.433ZWARNcrucible: Guest handle has been dropped
37839 Mar 17 22:27:26.499 INFO accepted connection, remote_addr: 127.0.0.1:44120, local_addr: 127.0.0.1:9998
37840 Main task has work to do, go find it
37841 got dsc Shutdown
37842 Shutdown
37843 Mar 17 22:27:26.500 INFO request completed, Shut it down
37844 latency_us[8810] Disable keep_running
37845 : 168, [8810] Got stop action so:false kr:false
37846 response_code: 204, uri: /shutdown, method: POST, req_id: e78d72df-fd6c-41e9-b8ba-9e9eeb6f150c, remote_addr: 127.0.0.1:44120, local_addr: 127.0.[8820] Disable keep_running
37847 0[8830] Disable keep_running
37848 .[8830] Got stop action so:false kr:false
37849 1:[8820] Got stop action so:false kr:false
37850 9998
37851 [8820] Exited with: ExitStatus(unix_wait_status(9))
37852 [8830] Exited with: ExitStatus(unix_wait_status(9))
37853 [8810] Exited with: ExitStatus(unix_wait_status(9))
37854 [8820][1] reports Exit
37855 [8830][2] reports Exit
37856 [8810][0] reports Exit
37857 March 17, 2025 at 10:27:31 PM UTC Test ends with 0