Max level shown:
1 starting March 17, 2025 at 10:16:08 PM UTC
2 Creating 3 downstairs regions
3 Creating region directory at: /var/tmp/dsc/region
4 Creating dsc directory at: /tmp/dsc
5 Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.27932337
6 Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.024509864
7 Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.023223653
8 Created 3 regions
9 Starting 3 downstairs
10 Using existing output directory "/tmp/dsc"
11 Update our region info with: Some(RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 })
12 start ds: 8810
13 start ds: 8820
14 start ds: 8830
15 start access at:127.0.0.1:9998
16 Starting downstairs at port 8830
17 Starting downstairs at port 8810
18 Make output file at "/tmp/dsc/downstairs-8810.txt"
19 Starting downstairs at port 8820
20 Make output file at "/tmp/dsc/downstairs-8830.txt"
21 Make output file at "/tmp/dsc/downstairs-8820.txt"
22 [8830][2] initial start wait reports Starting
23 [8810][0] initial start wait reports Starting
24 [8820][1] initial start wait reports Starting
25 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1196)
26 [8810][0] initial start wait reports Running
27 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1197)
28 [8830][2] initial start wait reports Running
29 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1198)
30 [8820][1] initial start wait reports Running
31 All downstairs are running
32 Mar 17 22:16:09.345 INFO listening, local_addr: 127.0.0.1:9998
33 Control access at:127.0.0.1:9998
34 Running initial fill
35 Mar 17 22:16:14.415 INFO accepted connection, remote_addr: 127.0.0.1:60000, local_addr: 127.0.0.1:9998
36 Mar 17 22:16:14.418 INFO request completed, latency_us: 716, response_code: 200, uri: /regioninfo, method: GET, req_id: 7f7ddca7-02bc-4447-8ee4-0f719cc457bd, remote_addr: 127.0.0.1:60000, local_addr: 127.0.0.1:9998
37 Mar 17 22:16:14.419 INFO request completed, latency_us: 116, response_code: 200, uri: /regioncount, method: GET, req_id: 671b5787-6e19-4c92-bdb6-7f0759a425cf, remote_addr: 127.0.0.1:60000, local_addr: 127.0.0.1:9998
382025-03-17T22:16:14.419ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
392025-03-17T22:16:14.420ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
40 Mar 17 22:16:14.420 INFO request completed, latency_us: 135, response_code: 200, uri: /port/cid/0, method: GET, req_id: 70945d02-3482-4d79-87fe-f08f79402d20, remote_addr: 127.0.0.1:60000, local_addr: 127.0.0.1:9998
41 Mar 17 22:16:14.421 INFO request completed, latency_us: 91, response_code: 200, uri: /port/cid/1, method: GET, req_id: bb4592df-2df3-4601-b7b6-5ce01c4c8ef4, remote_addr: 127.0.0.1:60000, local_addr: 127.0.0.1:9998
42 Mar 17 22:16:14.421 INFO request completed, latency_us: 89, response_code: 200, uri: /port/cid/2, method: GET, req_id: 6c6d43e0-4fbe-4009-9c6a-c85deb943396, remote_addr: 127.0.0.1:60000, local_addr: 127.0.0.1:9998
432025-03-17T22:16:14.422ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
442025-03-17T22:16:14.422ZINFOcrucible: Upstairs starts
452025-03-17T22:16:14.422ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "f72bfdb36aa576722b377a737f26a9b5ef54faf2", git_commit_timestamp: "2025-03-17T21:58:36.000000000Z", git_branch: "renovate/tempfile-3.x-lockfile", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
462025-03-17T22:16:14.422ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
472025-03-17T22:16:14.422ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
482025-03-17T22:16:14.423ZINFOcrucible: Crucible 2e5a0afc-bf1b-4320-b748-817adf400a63 has session id: 1df7c5e8-2894-4477-a1fd-95daeea5bfc0 session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
492025-03-17T22:16:14.423ZINFOcrucible: Upstairs opts: Upstairs UUID: 2e5a0afc-bf1b-4320-b748-817adf400a63, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
502025-03-17T22:16:14.423ZINFOcrucible: Crucible stats registered with UUID: 2e5a0afc-bf1b-4320-b748-817adf400a63 session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
512025-03-17T22:16:14.423ZWARNcrucible: could not find Downstairs address for Nexus session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
522025-03-17T22:16:14.425ZINFOcrucible: Downstairs in volume = 3 task = crutest
532025-03-17T22:16:14.425ZINFOcrucible: The guest has requested activation with gen:1
542025-03-17T22:16:14.425ZINFOcrucible: 2e5a0afc-bf1b-4320-b748-817adf400a63 active request set session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
552025-03-17T22:16:14.425ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
562025-03-17T22:16:14.425ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
572025-03-17T22:16:14.425ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
582025-03-17T22:16:14.425ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
592025-03-17T22:16:14.425ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
602025-03-17T22:16:14.425ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
612025-03-17T22:16:14.425ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
622025-03-17T22:16:14.425ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
632025-03-17T22:16:14.425ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
642025-03-17T22:16:14.425ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:43387) = io task client = 2 session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
652025-03-17T22:16:14.425ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:61536) = io task client = 1 session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
662025-03-17T22:16:14.425ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:63262) = io task client = 0 session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
672025-03-17T22:16:14.427ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
682025-03-17T22:16:14.427ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
692025-03-17T22:16:14.427ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
702025-03-17T22:16:14.428ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
712025-03-17T22:16:14.428ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
722025-03-17T22:16:14.428ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
732025-03-17T22:16:14.430ZINFOcrucible: Waiting for more clients to be ready session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
742025-03-17T22:16:14.430ZINFOcrucible: Waiting for more clients to be ready session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
752025-03-17T22:16:14.430ZINFOcrucible: [0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
762025-03-17T22:16:14.430ZINFOcrucible: [0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
772025-03-17T22:16:14.430ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
782025-03-17T22:16:14.430ZINFOcrucible: [1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
792025-03-17T22:16:14.430ZINFOcrucible: [1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
802025-03-17T22:16:14.430ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
812025-03-17T22:16:14.430ZINFOcrucible: [2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
822025-03-17T22:16:14.430ZINFOcrucible: [2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
832025-03-17T22:16:14.430ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
842025-03-17T22:16:14.430ZINFOcrucible: Max found gen is 1 = downstairs session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
852025-03-17T22:16:14.430ZINFOcrucible: Generation requested: 1 >= found:1 = downstairs session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
862025-03-17T22:16:14.430ZINFOcrucible: Next flush: 1 = downstairs session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
872025-03-17T22:16:14.430ZINFOcrucible: All extents match = downstairs session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
882025-03-17T22:16:14.430ZINFOcrucible: No downstairs reconciliation required session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
892025-03-17T22:16:14.430ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
902025-03-17T22:16:14.430ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
912025-03-17T22:16:14.430ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
92 {"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:16:14.430516089Z","Activate successful
93 hostname":Wait for a query_work_queue command to finish before sending IO
94 "w-01JPK2S810SFAA9787XEG8GFX5","pid":1202,"session_id":"1df7c5e8-2894-4477-a1fd-95daeea5bfc0"}
952025-03-17T22:16:14.430ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
962025-03-17T22:16:14.430ZINFOcrucible: 2e5a0afc-bf1b-4320-b748-817adf400a63 is now active with session: 1df7c5e8-2894-4477-a1fd-95daeea5bfc0 session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
972025-03-17T22:16:14.430ZINFOcrucible: Set Active after no reconciliation session_id = 1df7c5e8-2894-4477-a1fd-95daeea5bfc0
982025-03-17T22:16:14.430ZINFOcrucible: The guest has finished waiting for activation with:1
992025-03-17T22:16:14.430ZINFOcrucible: Activated sub_volume 2e5a0afc-bf1b-4320-b748-817adf400a63
100 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
101 Fill test
102 Read and Verify all blocks (0..5000 range:false)
103 Wrote out file "/tmp/test_replay_verify.log"
104 CLIENT: Tests done. All submitted work has been ACK'd
105 ----------------------------------------------------------------
106 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:20
107 States: Active Active Active
108 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
109 1021 Acked Read 256 Done Done Done false
110 1022 Acked Read 256 Done Done Done false
111 1023 Acked Read 256 Done Done Done false
112 1024 Acked Read 256 Done Done Done false
113 1025 Acked Read 256 Done Done Done false
114 1026 Acked Read 256 Done Done Done false
115 1027 Acked Read 256 Done Done Done false
116 1028 Acked Read 256 Done Done Done false
117 1029 Acked Read 256 Done Done Done false
118 1030 Acked Read 256 Done Done Done false
119 1031 Acked Read 256 Done Done Done false
120 1032 Acked Read 256 Done Done Done false
121 1033 Acked Read 256 Done Done Done false
122 1034 Acked Read 256 Done Done Done false
123 1035 Acked Read 256 Done Done Done false
124 1036 Acked Read 256 Done Done Done false
125 1037 Acked Read 256 Done Done Done false
126 1038 Acked Read 256 Done Done Done false
127 1039 Acked Read 256 Done Done Done false
128 1040 Acked Read 136 Done Done Done false
129 STATES DS:0 DS:1 DS:2 TOTAL
130 Sent 0 0 0 0
131 Done 20 20 20 60
132 Skipped 0 0 0 0
133 Error 0 0 0 0
134 Last Flush: Some(JobId(1020)) Some(JobId(1020)) Some(JobId(1020))
135 Downstairs last five completed: 1020 1019 1018 1017 1016
136 Upstairs last five completed: 1040 1039 1038 1037 1036
137 CLIENT: Up:0 ds:20 act:3
138 ----------------------------------------------------------------
139 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0
140 Downstairs last five completed: 1041 1040 1039 1038 1037
141 Upstairs last five completed: 1041 1040 1039 1038 1037
142 CLIENT: Up:0 ds:0 act:3
143 CLIENT: All crucible jobs finished, exiting program
1442025-03-17T22:16:19.017ZWARNcrucible: Guest handle has been dropped
145 Replay loop starts now March 17, 2025 at 10:16:19 PM UTC
146 Mar 17 22:16:19.090 INFO accepted connection, remote_addr: 127.0.0.1:52427, local_addr: 127.0.0.1:9998
147 Mar 17 22:16:19.091 INFO request completed, latency_us: 177, response_code: 200, uri: /regioninfo, method: GET, req_id: ebeee4d7-4052-4ee0-b65d-e602815d6bb4, remote_addr: 127.0.0.1:52427, local_addr: 127.0.0.1:9998
1482025-03-17T22:16:19.092ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
149 Mar 17 22:16:19.092 INFO request completed, latency_us: 108, response_code: 200, uri: /regioncount, method: GET, req_id: 929f5275-03ac-4009-9265-ecb0c99240ed, remote_addr: 127.0.0.1:52427, local_addr: 127.0.0.1:9998
1502025-03-17T22:16:19.093ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
151 Mar 17 22:16:19.093 INFO request completed, latency_us: 114, response_code: 200, uri: /port/cid/0, method: GET, req_id: c8c1d3b6-e429-44fd-bdd0-4f7f55820c52, remote_addr: 127.0.0.1:52427, local_addr: 127.0.0.1:9998
152 Mar 17 22:16:19.093 INFO request completed, latency_us: 102, response_code: 200, uri: /port/cid/1, method: GET, req_id: 21596d36-e7fe-42fd-a031-e131dce1296d, remote_addr: 127.0.0.1:52427, local_addr: 127.0.0.1:9998
153 Mar 17 22:16:19.094 INFO request completed, latency_us: 91, response_code: 200, uri: /port/cid/2, method: GET, req_id: a8524de3-299f-4dbf-9d79-6d3b6b3d7c4c, remote_addr: 127.0.0.1:52427, local_addr: 127.0.0.1:9998
1542025-03-17T22:16:19.094ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
1552025-03-17T22:16:19.094ZINFOcrucible: Upstairs starts
1562025-03-17T22:16:19.094ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "f72bfdb36aa576722b377a737f26a9b5ef54faf2", git_commit_timestamp: "2025-03-17T21:58:36.000000000Z", git_branch: "renovate/tempfile-3.x-lockfile", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
1572025-03-17T22:16:19.094ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
1582025-03-17T22:16:19.094ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
1592025-03-17T22:16:19.095ZINFOcrucible: Crucible d422572a-ba55-4c01-b96c-32bb507c77d1 has session id: 0f8310d7-099e-4ead-8484-91ddc162d307 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1602025-03-17T22:16:19.095ZINFOcrucible: Upstairs opts: Upstairs UUID: d422572a-ba55-4c01-b96c-32bb507c77d1, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1612025-03-17T22:16:19.095ZINFOcrucible: Crucible stats registered with UUID: d422572a-ba55-4c01-b96c-32bb507c77d1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1622025-03-17T22:16:19.095ZWARNcrucible: could not find Downstairs address for Nexus session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1632025-03-17T22:16:19.096ZINFOcrucible: Downstairs in volume = 3 task = crutest
1642025-03-17T22:16:19.096ZINFOcrucible: The guest has requested activation with gen:2
1652025-03-17T22:16:19.096ZINFOcrucible: d422572a-ba55-4c01-b96c-32bb507c77d1 active request set session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1662025-03-17T22:16:19.096ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1672025-03-17T22:16:19.096ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1682025-03-17T22:16:19.096ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1692025-03-17T22:16:19.096ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1702025-03-17T22:16:19.096ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1712025-03-17T22:16:19.096ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1722025-03-17T22:16:19.096ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1732025-03-17T22:16:19.096ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1742025-03-17T22:16:19.096ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1752025-03-17T22:16:19.096ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:43292) = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1762025-03-17T22:16:19.096ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:37500) = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1772025-03-17T22:16:19.096ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:38934) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1782025-03-17T22:16:19.097ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1792025-03-17T22:16:19.097ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1802025-03-17T22:16:19.097ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1812025-03-17T22:16:19.098ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1822025-03-17T22:16:19.098ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1832025-03-17T22:16:19.098ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1842025-03-17T22:16:19.098ZINFOcrucible: Waiting for more clients to be ready session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1852025-03-17T22:16:19.099ZINFOcrucible: Waiting for more clients to be ready session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1862025-03-17T22:16:19.099ZINFOcrucible: [0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1872025-03-17T22:16:19.099ZINFOcrucible: [0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1882025-03-17T22:16:19.099ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1892025-03-17T22:16:19.099ZINFOcrucible: [1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1902025-03-17T22:16:19.099ZINFOcrucible: [1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1912025-03-17T22:16:19.099ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1922025-03-17T22:16:19.099ZINFOcrucible: [2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1932025-03-17T22:16:19.099ZINFOcrucible: [2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1942025-03-17T22:16:19.099ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1952025-03-17T22:16:19.099ZINFOcrucible: Max found gen is 2 = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1962025-03-17T22:16:19.099ZINFOcrucible: Generation requested: 2 >= found:2 = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1972025-03-17T22:16:19.099ZINFOcrucible: Next flush: 2 = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1982025-03-17T22:16:19.099ZINFOcrucible: All extents match = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1992025-03-17T22:16:19.099ZINFOcrucible: No downstairs reconciliation required session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
200 {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Activate successful
201 ,"Wait for a query_work_queue command to finish before sending IO
202 time":"2025-03-17T22:16:19.099359779Z","hostname":"w-01JPK2S810SFAA9787XEG8GFX5","pid":1206,"client":"0","":"downstairs","session_id":"0f8310d7-099e-4ead-8484-91ddc162d307"}
2032025-03-17T22:16:19.099ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
2042025-03-17T22:16:19.099ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
2052025-03-17T22:16:19.099ZINFOcrucible: All required reconciliation work is completed session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
2062025-03-17T22:16:19.099ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
2072025-03-17T22:16:19.099ZINFOcrucible: d422572a-ba55-4c01-b96c-32bb507c77d1 is now active with session: 0f8310d7-099e-4ead-8484-91ddc162d307 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
2082025-03-17T22:16:19.099ZINFOcrucible: Set Active after no reconciliation session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
2092025-03-17T22:16:19.099ZINFOcrucible: The guest has finished waiting for activation with:2
2102025-03-17T22:16:19.099ZINFOcrucible: Activated sub_volume d422572a-ba55-4c01-b96c-32bb507c77d1
211 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
212 Loading write count information from file "/tmp/test_replay_verify.log"
213 Mar 17 22:16:19.132 INFO accepted connection, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
214 Main task has work to do, go find it
215 got dsc Stop(2)
216 stop 2
217 Mar 17 22:16:19.133 INFO request completed, latency_us: 156, response_code: 204, uri: /stop/cid/2, method: POST, req_id: c7dd11b9-f67a-45b9-86e1-ab7817a198cf, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998[8830] Got stop action so:false kr:false
218 
2192025-03-17T22:16:19.133ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
2202025-03-17T22:16:19.133ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
2212025-03-17T22:16:19.134ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
222 Mar 17 22:16:19.134 INFO request completed, latency_us: 157, response_code: 200, uri: /state/cid/2, method: GET, req_id: 8a4391e1-f72d-49a6-af94-7078f798610a, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
2232025-03-17T22:16:19.134ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
224 [8830] Exited with: ExitStatus(unix_wait_status(9))
225 [8830][2] reports Exit
226 Mar 17 22:16:23.136 INFO request completed, latency_us: 444, response_code: 200, uri: /state/cid/2, method: GET, req_id: 9b560cee-47f8-46bc-b57e-a034c090c6bf, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
227 001/300 Read block 3928 len 24576
228 002/300 Write block 652 len 20480 data: 2 2 2 2 2
229 003/300 Read block 2164 len 28672
230 004/300 Read block 3300 len 40960
231 005/300 Read block 151 len 16384
232 006/300 Write block 2987 len 20480 data: 2 2 2 2 2
233 007/300 Read block 2561 len 24576
234 008/300 Flush
235 009/300 Read block 1834 len 8192
236 010/300 Write block 2105 len 12288 data: 2 2 2
237 011/300 Read block 3056 len 4096
238 012/300 Write block 3102 len 36864 data: 2 2 2 2 2 2 2 2 2
239 013/300 Flush
240 014/300 Write block 1102 len 20480 data: 2 2 2 2 2
241 015/300 Flush
242 016/300 Read block 3760 len 8192
243 017/300 Flush
244 018/300 Write block 4733 len 40960 data: 2 2 2 2 2 2 2 2 2 2
245 019/300 Flush
246 020/300 Write block 3387 len 8192 data: 2 2
247 021/300 Read block 1549 len 28672
248 022/300 Read block 3540 len 4096
249 023/300 Write block 4484 len 40960 data: 2 2 2 2 2 2 2 2 2 2
250 024/300 Read block 3104 len 32768
251 025/300 Read block 18 len 4096
252 026/300 Write block 3867 len 40960 data: 2 2 2 2 2 2 2 2 2 2
253 027/300 Read block 1608 len 32768
254 028/300 Write block 1087 len 32768 data: 2 2 2 2 2 2 2 2
255 029/300 Read block 3409 len 16384
256 030/300 Read block 3522 len 20480
257 031/300 Write block 4337 len 8192 data: 2 2
258 032/300 Read block 4853 len 12288
259 033/300 Write block 2229 len 20480 data: 2 2 2 2 2
260 034/300 Read block 4990 len 28672
261 035/300 Read block 2328 len 12288
262 036/300 Read block 675 len 20480
263 037/300 Read block 2916 len 20480
264 038/300 Write block 4056 len 16384 data: 2 2 2 2
265 039/300 Read block 4502 len 24576
266 040/300 Read block 4541 len 20480
267 041/300 Flush
268 042/300 Read block 2196 len 16384
269 043/300 Read block 1658 len 8192
270 044/300 Write block 624 len 28672 data: 2 2 2 2 2 2 2
271 045/300 Read block 1091 len 20480
272 046/300 Read block 2518 len 4096
273 047/300 Flush
274 048/300 Write block 1693 len 36864 data: 2 2 2 2 2 2 2 2 2
275 049/300 Write block 1640 len 40960 data: 2 2 2 2 2 2 2 2 2 2
276 050/300 Write block 2930 len 36864 data: 2 2 2 2 2 2 2 2 2
277 051/300 Write block 1580 len 28672 data: 2 2 2 2 2 2 2
278 052/300 Read block 964 len 12288
279 053/300 Read block 3845 len 4096
280 054/300 Read block 944 len 20480
281 055/300 Read block 3069 len 28672
282 056/300 Read block 2110 len 28672
283 057/300 Read block 192 len 36864
284 058/300 Read block 2156 len 4096
285 059/300 Read block 937 len 40960
286 060/300 Flush
287 061/300 Read block 1374 len 40960
288 062/300 Read block 4380 len 20480
289 063/300 Write block 4335 len 32768 data: 2 2 3 3 2 2 2 2
290 064/300 Write block 2058 len 28672 data: 2 2 2 2 2 2 2
291 065/300 Write block 904 len 4096 data: 2
292 066/300 Write block 930 len 36864 data: 2 2 2 2 2 2 2 2 2
293 067/300 Read block 2001 len 32768
294 068/300 Read block 4422 len 12288
295 069/300 Read block 4701 len 20480
296 070/300 Write block 1465 len 36864 data: 2 2 2 2 2 2 2 2 2
297 071/300 Read block 2552 len 8192
298 072/300 Read block 4074 len 16384
299 073/300 Read block 4216 len 20480
300 074/300 Write block 3739 len 16384 data: 2 2 2 2
301 075/300 Write block 228 len 40960 data: 2 2 2 2 2 2 2 2 2 2
302 076/300 Read block 386 len 36864
303 077/300 Write block 2334 len 28672 data: 2 2 2 2 2 2 2
304 078/300 Write block 2215 len 32768 data: 2 2 2 2 2 2 2 2
305 079/300 Read block 4120 len 24576
306 080/300 Write block 527 len 16384 data: 2 2 2 2
307 081/300 Read block 454 len 8192
308 082/300 Read block 2842 len 24576
309 083/300 Write block 4301 len 20480 data: 2 2 2 2 2
310 084/300 Flush
311 085/300 Write block 3847 len 12288 data: 2 2 2
312 086/300 Read block 4359 len 16384
313 087/300 Read block 2565 len 28672
314 088/300 Write block 716 len 4096 data: 2
315 089/300 Write block 1763 len 32768 data: 2 2 2 2 2 2 2 2
316 090/300 Write block 270 len 40960 data: 2 2 2 2 2 2 2 2 2 2
317 091/300 Flush
318 092/300 Flush
319 093/300 Write block 4095 len 8192 data: 2 2
320 094/300 Read block 2360 len 36864
321 095/300 Read block 1197 len 12288
322 096/300 Read block 4899 len 24576
323 097/300 Write block 3347 len 40960 data: 2 2 2 2 2 2 2 2 2 2
324 098/300 Read block 639 len 16384
325 099/300 Flush
326 100/300 Read block 873 len 4096
327 101/300 Read block 2261 len 28672
328 102/300 Read block 2141 len 16384
329 103/300 Flush
330 104/300 Read block 3348 len 40960
331 105/300 Read block 1159 len 16384
332 106/300 Write block 3495 len 4096 data: 2
333 107/300 Read block 4173 len 8192
334 108/300 Flush
335 109/300 Read block 4674 len 32768
336 110/300 Read block 428 len 32768
337 111/300 Write block 2200 len 24576 data: 2 2 2 2 2 2
338 112/300 Write block 333 len 24576 data: 2 2 2 2 2 2
339 113/300 Read block 4719 len 16384
340 114/300 Flush
341 115/300 Read block 3492 len 28672
342 116/300 Flush
343 117/300 Write block 2089 len 20480 data: 2 2 2 2 2
344 118/300 Read block 1802 len 12288
345 119/300 Read block 1940 len 36864
346 120/300 Read block 1410 len 28672
347 121/300 Read block 63 len 28672
348 122/300 Flush
349 123/300 Read block 1120 len 28672
350 124/300 Read block 2650 len 32768
351 125/300 Write block 838 len 16384 data: 2 2 2 2
352 126/300 Write block 1606 len 4096 data: 2
353 127/300 Write block 4763 len 8192 data: 2 2
354 128/300 Read block 2568 len 4096
355 129/300 Read block 1606 len 32768
356 130/300 Write block 3957 len 16384 data: 2 2 2 2
357 131/300 Read block 1277 len 8192
358 132/300 Read block 108 len 4096
359 133/300 Flush
360 134/300 Write block 4211 len 4096 data: 2
361 135/300 Write block 2767 len 20480 data: 2 2 2 2 2
362 136/300 Write block 3378 len 12288 data: 2 2 2
363 137/300 Write block 4881 len 32768 data: 2 2 2 2 2 2 2 2
364 138/300 Read block 4838 len 36864
365 139/300 Read block 1032 len 12288
366 140/300 Write block 2599 len 32768 data: 2 2 2 2 2 2 2 2
367 141/300 Read block 2740 len 36864
368 142/300 Read block 2261 len 40960
369 143/300 Read block 836 len 16384
370 144/300 Read block 4077 len 20480
371 145/300 Write block 282 len 28672 data: 2 2 2 2 2 2 2
372 146/300 Read block 3099 len 40960
373 147/300 Read block 750 len 32768
374 148/300 Read block 3012 len 24576
375 149/300 Read block 214 len 32768
376 150/300 Read block 4392 len 12288
377 151/300 Flush
378 152/300 Read block 2646 len 32768
379 153/300 Read block 325 len 32768
380 154/300 Read block 2999 len 4096
381 155/300 Read block 2482 len 36864
382 156/300 Flush
383 157/300 Write block 2951 len 16384 data: 2 2 2 2
384 158/300 Flush
385 159/300 Read block 341 len 4096
386 160/300 Flush
387 161/300 Read block 1855 len 24576
388 162/300 Read block 4927 len 12288
389 163/300 Write block 3730 len 20480 data: 2 2 2 2 2
390 164/300 Write block 181 len 36864 data: 2 2 2 2 2 2 2 2 2
391 165/300 Read block 391 len 12288
392 166/300 Flush
393 167/300 Read block 3669 len 24576
394 168/300 Read block 1996 len 40960
395 169/300 Read block 2042 len 20480
396 170/300 Read block 835 len 8192
397 171/300 Write block 4715 len 12288 data: 2 2 2
398 172/300 Write block 502 len 24576 data: 2 2 2 2 2 2
399 173/300 Write block 1354 len 40960 data: 2 2 2 2 2 2 2 2 2 2
400 174/300 Read block 3161 len 20480
401 175/300 Write block 3392 len 32768 data: 2 2 2 2 2 2 2 2
402 176/300 Read block 3 len 24576
403 177/300 Read block 374 len 20480
404 178/300 Write block 1505 len 4096 data: 2
405 179/300 Read block 1284 len 28672
406 180/300 Read block 763 len 40960
407 181/300 Write block 681 len 16384 data: 2 2 2 2
408 182/300 Read block 3014 len 16384
409 183/300 Read block 3854 len 4096
410 184/300 Read block 3791 len 16384
411 185/300 Write block 2976 len 40960 data: 2 2 2 2 2 2 2 2 2 2
412 186/300 Write block 180 len 24576 data: 2 3 3 3 3 3
413 187/300 Read block 3014 len 24576
414 188/300 Write block 970 len 12288 data: 2 2 2
415 189/300 Write block 2897 len 16384 data: 2 2 2 2
416 190/300 Write block 330 len 32768 data: 2 2 2 3 3 3 3 3
417 191/300 Read block 3758 len 12288
418 192/300 Write block 1577 len 24576 data: 2 2 2 3 3 3
419 193/300 Read block 1824 len 4096
420 194/300 Read block 1508 len 28672
421 195/300 Read block 1029 len 40960
422 196/300 Read block 2843 len 20480
423 197/300 Write block 3133 len 40960 data: 2 2 2 2 2 2 2 2 2 2
424 198/300 Read block 4862 len 36864
425 199/300 Read block 4782 len 36864
426 200/300 Flush
427 201/300 Read block 3645 len 36864
428 202/300 Flush
429 203/300 Write block 2909 len 4096 data: 2
430 204/300 Write block 541 len 16384 data: 2 2 2 2
431 205/300 Write block 2097 len 12288 data: 2 2 2
432 206/300 Read block 938 len 36864
433 207/300 Read block 3382 len 20480
434 208/300 Read block 951 len 36864
435 209/300 Read block 4850 len 16384
436 210/300 Flush
437 211/300 Read block 791 len 32768
438 212/300 Write block 1243 len 20480 data: 2 2 2 2 2
439 213/300 Write block 4861 len 12288 data: 2 2 2
440 214/300 Flush
441 215/300 Flush
442 216/300 Read block 1290 len 4096
443 217/300 Write block 3143 len 32768 data: 2 2 2 2 2 2 2 2
444 218/300 Write block 2440 len 20480 data: 2 2 2 2 2
445 219/300 Write block 2295 len 24576 data: 2 2 2 2 2 2
446 220/300 Read block 3819 len 32768
447 221/300 Read block 2714 len 32768
448 222/300 Write block 4805 len 24576 data: 2 2 2 2 2 2
449 223/300 Write block 2472 len 32768 data: 2 2 2 2 2 2 2 2
450 224/300 Read block 4480 len 8192
451 225/300 Flush
452 226/300 Read block 4738 len 32768
453 227/300 Read block 4607 len 20480
454 228/300 Flush
455 229/300 Write block 4821 len 8192 data: 2 2
456 230/300 Read block 800 len 36864
457 231/300 Read block 3062 len 20480
458 232/300 Write block 1813 len 32768 data: 2 2 2 2 2 2 2 2
459 233/300 Write block 12 len 40960 data: 2 2 2 2 2 2 2 2 2 2
460 234/300 Read block 574 len 4096
461 235/300 Read block 663 len 24576
462 236/300 Write block 22 len 28672 data: 2 2 2 2 2 2 2
463 237/300 Write block 3718 len 12288 data: 2 2 2
464 238/300 Write block 4833 len 32768 data: 2 2 2 2 2 2 2 2
465 239/300 Flush
466 240/300 Write block 590 len 4096 data: 2
467 241/300 Write block 2054 len 24576 data: 2 2 2 2 3 3
468 242/300 Write block 3605 len 8192 data: 2 2
469 243/300 Read block 1517 len 12288
470 244/300 Read block 3955 len 32768
471 245/300 Read block 2252 len 40960
472 246/300 Read block 724 len 8192
473 247/300 Read block 4158 len 36864
474 248/300 Read block 305 len 32768
475 249/300 Read block 4420 len 12288
476 250/300 Read block 3782 len 8192
477 251/300 Read block 3357 len 28672
478 252/300 Read block 3028 len 24576
479 253/300 Read block 566 len 4096
480 254/300 Read block 207 len 20480
481 255/300 Write block 2377 len 36864 data: 2 2 2 2 2 2 2 2 2
482 256/300 Write block 924 len 36864 data: 2 2 2 2 2 2 3 3 3
483 257/300 Write block 2887 len 12288 data: 2 2 2
484 258/300 Write block 3278 len 24576 data: 2 2 2 2 2 2
485 259/300 Read block 2918 len 40960
486 260/300 Read block 3957 len 12288
487 261/300 Read block 4250 len 28672
488 262/300 Write block 3390 len 8192 data: 2 2
489 263/300 Write block 2555 len 16384 data: 2 2 2 2
490 264/300 Write block 3350 len 28672 data: 3 3 3 3 3 3 3
491 265/300 Flush
492 266/300 Write block 2377 len 40960 data: 3 3 3 3 3 3 3 3 3 2
493 267/300 Read block 3566 len 40960
494 268/300 Write block 3199 len 36864 data: 2 2 2 2 2 2 2 2 2
495 269/300 Read block 3460 len 20480
496 270/300 Read block 1234 len 40960
497 271/300 Read block 1949 len 32768
498 272/300 Write block 1063 len 8192 data: 2 2
499 273/300 Write block 2850 len 24576 data: 2 2 2 2 2 2
500 274/300 Write block 2406 len 12288 data: 2 2 2
501 275/300 Read block 1089 len 4096
502 276/300 Read block 1258 len 32768
503 277/300 Write block 3304 len 28672 data: 2 2 2 2 2 2 2
504 278/300 Read block 3809 len 4096
505 279/300 Read block 3204 len 4096
506 280/300 Read block 4789 len 24576
507 281/300 Write block 2876 len 4096 data: 2
508 282/300 Write block 979 len 20480 data: 2 2 2 2 2
509 283/300 Read block 4540 len 40960
510 284/300 Read block 190 len 24576
511 285/300 Flush
512 286/300 Write block 4391 len 40960 data: 2 2 2 2 2 2 2 2 2 2
513 287/300 Write block 2142 len 36864 data: 2 2 2 2 2 2 2 2 2
514 288/300 Flush
515 289/300 Read block 3188 len 12288
516 290/300 Read block 3112 len 40960
517 291/300 Write block 3780 len 16384 data: 2 2 2 2
518 292/300 Write block 2996 len 16384 data: 2 2 2 2
519 293/300 Read block 3475 len 16384
520 294/300 Write block 278 len 4096 data: 3
521 295/300 Read block 3996 len 4096
522 296/300 Read block 2294 len 40960
523 297/300 Read block 3759 len 36864
524 298/300 Flush
525 299/300 Read block 1833 len 40960
526 300/300 Read block 4646 len 4096
527 301/300 Write block 1006 len 4096 data: 2
528 Main task has work to do, go find it
529 got dsc Start(2)
530 start 2
531 Mar 17 22:16:23.396 INFO request completed, latency_us: 223, response_code: 204, uri: /start/cid/2, method: POST, req_id: 3571b71f-da11-44c7-9742-453a4651a55c, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
532 Replay: started 2, returned:Ok(())
533 ----------------------------------------------------------------
534 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
535 States: Active Active Offline
536 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
537 1000 Acked Read 6 Done Done Sent false
538 1001 Acked Write 5 Done Done Sent false
539 1002 Acked Read 7 Done Done Sent false
540 1003 Acked Read 10 Done Done Sent false
541 1004 Acked Read 4 Done Done Sent false
542 1005 Acked Write 5 Done Done Sent false
543 1006 Acked Read 6 Done Done Sent false
544 1007 Acked Flush 0 Done Done Sent false
545 1008 Acked Read 2 Done Done Sent false
546 1009 Acked Write 3 Done Done Sent false
547 1010 Acked Read 1 Done Done Sent false
548 1011 Acked Write 9 Done Done Sent false
549 1012 Acked Flush 0 Done Done Sent false
550 1013 Acked Write 5 Done Done Sent false
551 1014 Acked Flush 0 Done Done Sent false
552 1015 Acked Read 2 Done Done Sent false
553 1016 Acked Flush 0 Done Done Sent false
554 1017 Acked Write 10 Done Done Sent false
555 1018 Acked Flush 0 Done Done Sent false
556 1019 Acked Write 2 Done Done Sent false
557 1020 Acked Read 7 Done Done Sent false
558 1021 Acked Read 1 Done Done Sent false
559 1022 Acked Write 10 Done Done Sent false
560 1023 Acked Read 8 Done Done Sent false
561 1024 Acked Read 1 Done Done Sent false
562 1025 Acked Write 10 Done Done Sent false
563 1026 Acked Read 8 Done Done Sent false
564 1027 Acked Write 8 Done Done Sent false
565 1028 Acked Read 4 Done Done Sent false
566 1029 Acked Read 5 Done Done Sent false
567 1030 Acked Write 2 Done Done Sent false
568 1031 Acked Read 3 Done Done Sent false
569 1032 Acked Write 5 Done Done Sent false
570 1033 Acked Read 7 Done Done Sent false
571 1034 Acked Read 3 Done Done Sent false
572 1035 Acked Read 5 Done Done Sent false
573 1036 Acked Read 5 Done Done Sent false
574 1037 Acked Write 4 Done Done Sent false
575 1038 Acked Read 6 Done Done Sent false
576 1039 Acked Read 5 Done Done Sent false
577 1040 Acked Flush 0 Done Done Sent false
578 1041 Acked Read 4 Done Done Sent false
579 1042 Acked Read 2 Done Done Sent false
580 1043 Acked Write 7 Done Done Sent false
581 1044 Acked Read 5 Done Done Sent false
582 1045 Acked Read 1 Done Done Sent false
583 1046 Acked Flush 0 Done Done Sent false
584 1047 Acked Write 9 Done Done Sent false
585 1048 Acked Write 10 Done Done Sent false
586 1049 Acked Write 9 Done Done Sent false
587 1050 Acked Write 7 Done Done Sent false
588 1051 Acked Read 3 Done Done Sent false
589 1052 Acked Read 1 Done Done Sent false
590 1053 Acked Read 5 Done Done Sent false
591 1054 Acked Read 7 Done Done Sent false
592 1055 Acked Read 7 Done Done Sent false
593 1056 Acked Read 9 Done Done Sent false
594 1057 Acked Read 1 Done Done Sent false
595 1058 Acked Read 10 Done Done Sent false
596 1059 Acked Flush 0 Done Done Sent false
597 1060 Acked Read 10 Done Done Sent false
598 1061 Acked Read 5 Done Done Sent false
599 1062 Acked Write 8 Done Done Sent false
600 1063 Acked Write 7 Done Done Sent false
601 1064 Acked Write 1 Done Done Sent false
602 1065 Acked Write 9 Done Done Sent false
603 1066 Acked Read 8 Done Done Sent false
604 1067 Acked Read 3 Done Done Sent false
605 1068 Acked Read 5 Done Done Sent false
606 1069 Acked Write 9 Done Done Sent false
607 1070 Acked Read 2 Done Done Sent false
608 1071 Acked Read 4 Done Done Sent false
609 1072 Acked Read 5 Done Done Sent false
610 1073 Acked Write 4 Done Done Sent false
611 1074 Acked Write 10 Done Done Sent false
612 1075 Acked Read 9 Done Done Sent false
613 1076 Acked Write 7 Done Done Sent false
614 1077 Acked Write 8 Done Done Sent false
615 1078 Acked Read 6 Done Done Sent false
616 1079 Acked Write 4 Done Done Sent false
617 1080 Acked Read 2 Done Done Sent false
618 1081 Acked Read 6 Done Done Sent false
619 1082 Acked Write 5 Done Done Sent false
620 1083 Acked Flush 0 Done Done Sent false
621 1084 Acked Write 3 Done Done Sent false
622 1085 Acked Read 4 Done Done Sent false
623 1086 Acked Read 7 Done Done Sent false
624 1087 Acked Write 1 Done Done Sent false
625 1088 Acked Write 8 Done Done Sent false
626 1089 Acked Write 10 Done Done Sent false
627 1090 Acked Flush 0 Done Done Sent false
628 1091 Acked Flush 0 Done Done Sent false
629 1092 Acked Write 2 Done Done Sent false
630 1093 Acked Read 9 Done Done Sent false
631 1094 Acked Read 3 Done Done Sent false
632 1095 Acked Read 6 Done Done Sent false
633 1096 Acked Write 10 Done Done Sent false
634 1097 Acked Read 4 Done Done Sent false
635 1098 Acked Flush 0 Done Done Sent false
636 1099 Acked Read 1 Done Done Sent false
637 1100 Acked Read 7 Done Done Sent false
638 1101 Acked Read 4 Done Done Sent false
639 1102 Acked Flush 0 Done Done Sent false
640 1103 Acked Read 10 Done Done Sent false
641 1104 Acked Read 4 Done Done Sent false
642 1105 Acked Write 1 Done Done Sent false
643 1106 Acked Read 2 Done Done Sent false
644 1107 Acked Flush 0 Done Done Sent false
645 1108 Acked Read 8 Done Done Sent false
646 1109 Acked Read 8 Done Done Sent false
647 1110 Acked Write 6 Done Done Sent false
648 1111 Acked Write 6 Done Done Sent false
649 1112 Acked Read 4 Done Done Sent false
650 1113 Acked Flush 0 Done Done Sent false
651 1114 Acked Read 7 Done Done Sent false
652 1115 Acked Flush 0 Done Done Sent false
653 1116 Acked Write 5 Done Done Sent false
654 1117 Acked Read 3 Done Done Sent false
655 1118 Acked Read 9 Done Done Sent false
656 1119 Acked Read 7 Done Done Sent false
657 1120 Acked Read 7 Done Done Sent false
658 1121 Acked Flush 0 Done Done Sent false
659 1122 Acked Read 7 Done Done Sent false
660 1123 Acked Read 8 Done Done Sent false
661 1124 Acked Write 4 Done Done Sent false
662 1125 Acked Write 1 Done Done Sent false
663 1126 Acked Write 2 Done Done Sent false
664 1127 Acked Read 1 Done Done Sent false
665 1128 Acked Read 8 Done Done Sent false
666 1129 Acked Write 4 Done Done Sent false
667 1130 Acked Read 2 Done Done Sent false
668 1131 Acked Read 1 Done Done Sent false
669 1132 Acked Flush 0 Done Done Sent false
670 1133 Acked Write 1 Done Done Sent false
671 1134 Acked Write 5 Done Done Sent false
672 1135 Acked Write 3 Done Done Sent false
673 1136 Acked Write 8 Done Done Sent false
674 1137 Acked Read 9 Done Done Sent false
675 1138 Acked Read 3 Done Done Sent false
676 1139 Acked Write 8 Done Done Sent false
677 1140 Acked Read 9 Done Done Sent false
678 1141 Acked Read 10 Done Done Sent false
679 1142 Acked Read 4 Done Done Sent false
680 1143 Acked Read 5 Done Done Sent false
681 1144 Acked Write 7 Done Done Sent false
682 1145 Acked Read 10 Done Done Sent false
683 1146 Acked Read 8 Done Done Sent false
684 1147 Acked Read 6 Done Done Sent false
685 1148 Acked Read 8 Done Done Sent false
686 1149 Acked Read 3 Done Done Sent false
687 1150 Acked Flush 0 Done Done Sent false
688 1151 Acked Read 8 Done Done Sent false
689 1152 Acked Read 8 Done Done Sent false
690 1153 Acked Read 1 Done Done Sent false
691 1154 Acked Read 9 Done Done Sent false
692 1155 Acked Flush 0 Done Done Sent false
693 1156 Acked Write 4 Done Done Sent false
694 1157 Acked Flush 0 Done Done Sent false
695 1158 Acked Read 1 Done Done Sent false
696 1159 Acked Flush 0 Done Done Sent false
697 1160 Acked Read 6 Done Done Sent false
698 1161 Acked Read 3 Done Done Sent false
699 1162 Acked Write 5 Done Done Sent false
700 1163 Acked Write 9 Done Done Sent false
701 1164 Acked Read 3 Done Done Sent false
702 1165 Acked Flush 0 Done Done Sent false
703 1166 Acked Read 6 Done Done Sent false
704 1167 Acked Read 10 Done Done Sent false
705 1168 Acked Read 5 Done Done Sent false
706 1169 Acked Read 2 Done Done Sent false
707 1170 Acked Write 3 Done Done Sent false
708 1171 Acked Write 6 Done Done Sent false
709 1172 Acked Write 10 Done Done Sent false
710 1173 Acked Read 5 Done Done Sent false
711 1174 Acked Write 8 Done Done Sent false
712 1175 Acked Read 6 Done Done Sent false
713 1176 Acked Read 5 Done Done Sent false
714 1177 Acked Write 1 Done Done Sent false
715 1178 Acked Read 7 Done Done Sent false
716 1179 Acked Read 10 Done Done Sent false
717 1180 Acked Write 4 Done Done Sent false
718 1181 Acked Read 4 Done Done Sent false
719 1182 Acked Read 1 Done Done Sent false
720 1183 Acked Read 4 Done Done Sent false
721 1184 Acked Write 10 Done Done Sent false
722 1185 Acked Write 6 Done Done Sent false
723 1186 Acked Read 6 Done Done Sent false
724 1187 Acked Write 3 Done Done Sent false
725 1188 Acked Write 4 Done Done Sent false
726 1189 Acked Write 8 Done Done Sent false
727 1190 Acked Read 3 Done Done Sent false
728 1191 Acked Write 6 Done Done Sent false
729 1192 Acked Read 1 Done Done Sent false
730 1193 Acked Read 7 Done Done Sent false
731 1194 Acked Read 10 Done Done Sent false
732 1195 Acked Read 5 Done Done Sent false
733 1196 Acked Write 10 Done Done Sent false
734 1197 Acked Read 9 Done Done Sent false
735 1198 Acked Read 9 Done Done Sent false
736 1199 Acked Flush 0 Done Done Sent false
737 1200 Acked Read 9 Done Done Sent false
738 1201 Acked Flush 0 Done Done Sent false
739 1202 Acked Write 1 Done Done Sent false
740 1203 Acked Write 4 Done Done Sent false
741 1204 Acked Write 3 Done Done Sent false
742 1205 Acked Read 9 Done Done Sent false
743 1206 Acked Read 5 Done Done Sent false
744 1207 Acked Read 9 Done Done Sent false
745 1208 Acked Read 4 Done Done Sent false
746 1209 Acked Flush 0 Done Done Sent false
747 1210 Acked Read 8 Done Done Sent false
748 1211 Acked Write 5 Done Done Sent false
749 1212 Acked Write 3 Done Done Sent false
750 1213 Acked Flush 0 Done Done Sent false
751 1214 Acked Flush 0 Done Done Sent false
752 1215 Acked Read 1 Done Done Sent false
753 1216 Acked Write 8 Done Done Sent false
754 1217 Acked Write 5 Done Done Sent false
755 1218 Acked Write 6 Done Done Sent false
756 1219 Acked Read 8 Done Done Sent false
757 1220 Acked Read 8 Done Done Sent false
758 1221 Acked Write 6 Done Done Sent false
759 1222 Acked Write 8 Done Done Sent false
760 1223 Acked Read 2 Done Done Sent false
761 1224 Acked Flush 0 Done Done Sent false
762 1225 Acked Read 8 Done Done Sent false
763 1226 Acked Read 5 Done Done Sent false
764 1227 Acked Flush 0 Done Done Sent false
765 1228 Acked Write 2 Done Done Sent false
766 1229 Acked Read 9 Done Done Sent false
767 1230 Acked Read 5 Done Done Sent false
768 1231 Acked Write 8 Done Done Sent false
769 1232 Acked Write 10 Done Done Sent false
770 1233 Acked Read 1 Done Done Sent false
771 1234 Acked Read 6 Done Done Sent false
772 1235 Acked Write 7 Done Done Sent false
773 1236 Acked Write 3 Done Done Sent false
774 1237 Acked Write 8 Done Done Sent false
775 1238 Acked Flush 0 Done Done Sent false
776 1239 Acked Write 1 Done Done Sent false
777 1240 Acked Write 6 Done Done Sent false
778 1241 Acked Write 2 Done Done Sent false
779 1242 Acked Read 3 Done Done Sent false
780 1243 Acked Read 8 Done Done Sent false
781 1244 Acked Read 10 Done Done Sent false
782 1245 Acked Read 2 Done Done Sent false
783 1246 Acked Read 9 Done Done Sent false
784 1247 Acked Read 8 Done Done Sent false
785 1248 Acked Read 3 Done Done Sent false
786 1249 Acked Read 2 Done Done Sent false
787 1250 Acked Read 7 Done Done Sent false
788 1251 Acked Read 6 Done Done Sent false
789 1252 Acked Read 1 Done Done Sent false
790 1253 Acked Read 5 Done Done Sent false
791 1254 Acked Write 9 Done Done Sent false
792 1255 Acked Write 9 Done Done Sent false
793 1256 Acked Write 3 Done Done Sent false
794 1257 Acked Write 6 Done Done Sent false
795 1258 Acked Read 10 Done Done Sent false
796 1259 Acked Read 3 Done Done Sent false
797 1260 Acked Read 7 Done Done Sent false
798 1261 Acked Write 2 Done Done Sent false
799 1262 Acked Write 4 Done Done Sent false
800 1263 Acked Write 7 Done Done Sent false
801 1264 Acked Flush 0 Done Done Sent false
802 1265 Acked Write 10 Done Done Sent false
803 1266 Acked Read 10 Done Done Sent false
804 1267 Acked Write 9 Done Done Sent false
805 1268 Acked Read 5 Done Done Sent false
806 1269 Acked Read 10 Done Done Sent false
807 1270 Acked Read 8 Done Done Sent false
808 1271 Acked Write 2 Done Done Sent false
809 1272 Acked Write 6 Done Done Sent false
810 1273 Acked Write 3 Done Done Sent false
811 1274 Acked Read 1 Done Done Sent false
812 1275 Acked Read 8 Done Done Sent false
813 1276 Acked Write 7 Done Done Sent false
814 1277 Acked Read 1 Done Done Sent false
815 1278 Acked Read 1 Done Done Sent false
816 1279 Acked Read 6 Done Done Sent false
817 1280 Acked Write 1 Done Done Sent false
818 1281 Acked Write 5 Done Done Sent false
819 1282 Acked Read 10 Done Done Sent false
820 1283 Acked Read 6 Done Done Sent false
821 1284 Acked Flush 0 Done Done Sent false
822 1285 Acked Write 10 Done Done Sent false
823 1286 Acked Write 9 Done Done Sent false
824 1287 Acked Flush 0 Done Done Sent false
825 1288 Acked Read 3 Done Done Sent false
826 1289 Acked Read 10 Done Done Sent false
827 1290 Acked Write 4 Done Done Sent false
828 1291 Acked Write 4 Done Done Sent false
829 1292 Acked Read 4 Done Done Sent false
830 1293 Acked Write 1 Done Done Sent false
831 1294 Acked Read 1 Done Done Sent false
832 1295 Acked Read 10 Done Done Sent false
833 1296 Acked Read 9 Done Done Sent false
834 1297 Acked Flush 0 Done Done Sent false
835 1298 Acked Read 10 Done Done Sent false
836 1299 Acked Read 1 Done Done Sent false
837 1300 Acked Write 1 Done Done Sent false
838 STATES DS:0 DS:1 DS:2 TOTAL
839 Sent 0 0 301 301
840 Done 301 301 0 602
841 Skipped 0 0 0 0
842 Error 0 0 0 0
843 Last Flush: Some(JobId(1297)) Some(JobId(1297)) None
844 Downstairs last five completed:
845 Upstairs last five completed: 1300 1299 1298 1297 1296
846 CLIENT: Up:0 ds:301 act:2
8472025-03-17T22:16:23.897ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
8482025-03-17T22:16:24.604ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
849 [8830] got start action
850 [8830] I am going to restart
851 Starting downstairs at port 8830
852 Make output file at "/tmp/dsc/downstairs-8830.txt"
853 [8830][2] reports Starting
854 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1207)
855 [8830][2] reports Running
8562025-03-17T22:16:25.605ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
8572025-03-17T22:16:26.607ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
858 ----------------------------------------------------------------
859 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
860 States: Active Active Offline
861 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
862 1000 Acked Read 6 Done Done Sent false
863 1001 Acked Write 5 Done Done Sent false
864 1002 Acked Read 7 Done Done Sent false
865 1003 Acked Read 10 Done Done Sent false
866 1004 Acked Read 4 Done Done Sent false
867 1005 Acked Write 5 Done Done Sent false
868 1006 Acked Read 6 Done Done Sent false
869 1007 Acked Flush 0 Done Done Sent false
870 1008 Acked Read 2 Done Done Sent false
871 1009 Acked Write 3 Done Done Sent false
872 1010 Acked Read 1 Done Done Sent false
873 1011 Acked Write 9 Done Done Sent false
874 1012 Acked Flush 0 Done Done Sent false
875 1013 Acked Write 5 Done Done Sent false
876 1014 Acked Flush 0 Done Done Sent false
877 1015 Acked Read 2 Done Done Sent false
878 1016 Acked Flush 0 Done Done Sent false
879 1017 Acked Write 10 Done Done Sent false
880 1018 Acked Flush 0 Done Done Sent false
881 1019 Acked Write 2 Done Done Sent false
882 1020 Acked Read 7 Done Done Sent false
883 1021 Acked Read 1 Done Done Sent false
884 1022 Acked Write 10 Done Done Sent false
885 1023 Acked Read 8 Done Done Sent false
886 1024 Acked Read 1 Done Done Sent false
887 1025 Acked Write 10 Done Done Sent false
888 1026 Acked Read 8 Done Done Sent false
889 1027 Acked Write 8 Done Done Sent false
890 1028 Acked Read 4 Done Done Sent false
891 1029 Acked Read 5 Done Done Sent false
892 1030 Acked Write 2 Done Done Sent false
893 1031 Acked Read 3 Done Done Sent false
894 1032 Acked Write 5 Done Done Sent false
895 1033 Acked Read 7 Done Done Sent false
896 1034 Acked Read 3 Done Done Sent false
897 1035 Acked Read 5 Done Done Sent false
898 1036 Acked Read 5 Done Done Sent false
899 1037 Acked Write 4 Done Done Sent false
900 1038 Acked Read 6 Done Done Sent false
901 1039 Acked Read 5 Done Done Sent false
902 1040 Acked Flush 0 Done Done Sent false
903 1041 Acked Read 4 Done Done Sent false
904 1042 Acked Read 2 Done Done Sent false
905 1043 Acked Write 7 Done Done Sent false
906 1044 Acked Read 5 Done Done Sent false
907 1045 Acked Read 1 Done Done Sent false
908 1046 Acked Flush 0 Done Done Sent false
909 1047 Acked Write 9 Done Done Sent false
910 1048 Acked Write 10 Done Done Sent false
911 1049 Acked Write 9 Done Done Sent false
912 1050 Acked Write 7 Done Done Sent false
913 1051 Acked Read 3 Done Done Sent false
914 1052 Acked Read 1 Done Done Sent false
915 1053 Acked Read 5 Done Done Sent false
916 1054 Acked Read 7 Done Done Sent false
917 1055 Acked Read 7 Done Done Sent false
918 1056 Acked Read 9 Done Done Sent false
919 1057 Acked Read 1 Done Done Sent false
920 1058 Acked Read 10 Done Done Sent false
921 1059 Acked Flush 0 Done Done Sent false
922 1060 Acked Read 10 Done Done Sent false
923 1061 Acked Read 5 Done Done Sent false
924 1062 Acked Write 8 Done Done Sent false
925 1063 Acked Write 7 Done Done Sent false
926 1064 Acked Write 1 Done Done Sent false
927 1065 Acked Write 9 Done Done Sent false
928 1066 Acked Read 8 Done Done Sent false
929 1067 Acked Read 3 Done Done Sent false
930 1068 Acked Read 5 Done Done Sent false
931 1069 Acked Write 9 Done Done Sent false
932 1070 Acked Read 2 Done Done Sent false
933 1071 Acked Read 4 Done Done Sent false
934 1072 Acked Read 5 Done Done Sent false
935 1073 Acked Write 4 Done Done Sent false
936 1074 Acked Write 10 Done Done Sent false
937 1075 Acked Read 9 Done Done Sent false
938 1076 Acked Write 7 Done Done Sent false
939 1077 Acked Write 8 Done Done Sent false
940 1078 Acked Read 6 Done Done Sent false
941 1079 Acked Write 4 Done Done Sent false
942 1080 Acked Read 2 Done Done Sent false
943 1081 Acked Read 6 Done Done Sent false
944 1082 Acked Write 5 Done Done Sent false
945 1083 Acked Flush 0 Done Done Sent false
946 1084 Acked Write 3 Done Done Sent false
947 1085 Acked Read 4 Done Done Sent false
948 1086 Acked Read 7 Done Done Sent false
949 1087 Acked Write 1 Done Done Sent false
950 1088 Acked Write 8 Done Done Sent false
951 1089 Acked Write 10 Done Done Sent false
952 1090 Acked Flush 0 Done Done Sent false
953 1091 Acked Flush 0 Done Done Sent false
954 1092 Acked Write 2 Done Done Sent false
955 1093 Acked Read 9 Done Done Sent false
956 1094 Acked Read 3 Done Done Sent false
957 1095 Acked Read 6 Done Done Sent false
958 1096 Acked Write 10 Done Done Sent false
959 1097 Acked Read 4 Done Done Sent false
960 1098 Acked Flush 0 Done Done Sent false
961 1099 Acked Read 1 Done Done Sent false
962 1100 Acked Read 7 Done Done Sent false
963 1101 Acked Read 4 Done Done Sent false
964 1102 Acked Flush 0 Done Done Sent false
965 1103 Acked Read 10 Done Done Sent false
966 1104 Acked Read 4 Done Done Sent false
967 1105 Acked Write 1 Done Done Sent false
968 1106 Acked Read 2 Done Done Sent false
969 1107 Acked Flush 0 Done Done Sent false
970 1108 Acked Read 8 Done Done Sent false
971 1109 Acked Read 8 Done Done Sent false
972 1110 Acked Write 6 Done Done Sent false
973 1111 Acked Write 6 Done Done Sent false
974 1112 Acked Read 4 Done Done Sent false
975 1113 Acked Flush 0 Done Done Sent false
976 1114 Acked Read 7 Done Done Sent false
977 1115 Acked Flush 0 Done Done Sent false
978 1116 Acked Write 5 Done Done Sent false
979 1117 Acked Read 3 Done Done Sent false
980 1118 Acked Read 9 Done Done Sent false
981 1119 Acked Read 7 Done Done Sent false
982 1120 Acked Read 7 Done Done Sent false
983 1121 Acked Flush 0 Done Done Sent false
984 1122 Acked Read 7 Done Done Sent false
985 1123 Acked Read 8 Done Done Sent false
986 1124 Acked Write 4 Done Done Sent false
987 1125 Acked Write 1 Done Done Sent false
988 1126 Acked Write 2 Done Done Sent false
989 1127 Acked Read 1 Done Done Sent false
990 1128 Acked Read 8 Done Done Sent false
991 1129 Acked Write 4 Done Done Sent false
992 1130 Acked Read 2 Done Done Sent false
993 1131 Acked Read 1 Done Done Sent false
994 1132 Acked Flush 0 Done Done Sent false
995 1133 Acked Write 1 Done Done Sent false
996 1134 Acked Write 5 Done Done Sent false
997 1135 Acked Write 3 Done Done Sent false
998 1136 Acked Write 8 Done Done Sent false
999 1137 Acked Read 9 Done Done Sent false
1000 1138 Acked Read 3 Done Done Sent false
1001 1139 Acked Write 8 Done Done Sent false
1002 1140 Acked Read 9 Done Done Sent false
1003 1141 Acked Read 10 Done Done Sent false
1004 1142 Acked Read 4 Done Done Sent false
1005 1143 Acked Read 5 Done Done Sent false
1006 1144 Acked Write 7 Done Done Sent false
1007 1145 Acked Read 10 Done Done Sent false
1008 1146 Acked Read 8 Done Done Sent false
1009 1147 Acked Read 6 Done Done Sent false
1010 1148 Acked Read 8 Done Done Sent false
1011 1149 Acked Read 3 Done Done Sent false
1012 1150 Acked Flush 0 Done Done Sent false
1013 1151 Acked Read 8 Done Done Sent false
1014 1152 Acked Read 8 Done Done Sent false
1015 1153 Acked Read 1 Done Done Sent false
1016 1154 Acked Read 9 Done Done Sent false
1017 1155 Acked Flush 0 Done Done Sent false
1018 1156 Acked Write 4 Done Done Sent false
1019 1157 Acked Flush 0 Done Done Sent false
1020 1158 Acked Read 1 Done Done Sent false
1021 1159 Acked Flush 0 Done Done Sent false
1022 1160 Acked Read 6 Done Done Sent false
1023 1161 Acked Read 3 Done Done Sent false
1024 1162 Acked Write 5 Done Done Sent false
1025 1163 Acked Write 9 Done Done Sent false
1026 1164 Acked Read 3 Done Done Sent false
1027 1165 Acked Flush 0 Done Done Sent false
1028 1166 Acked Read 6 Done Done Sent false
1029 1167 Acked Read 10 Done Done Sent false
1030 1168 Acked Read 5 Done Done Sent false
1031 1169 Acked Read 2 Done Done Sent false
1032 1170 Acked Write 3 Done Done Sent false
1033 1171 Acked Write 6 Done Done Sent false
1034 1172 Acked Write 10 Done Done Sent false
1035 1173 Acked Read 5 Done Done Sent false
1036 1174 Acked Write 8 Done Done Sent false
1037 1175 Acked Read 6 Done Done Sent false
1038 1176 Acked Read 5 Done Done Sent false
1039 1177 Acked Write 1 Done Done Sent false
1040 1178 Acked Read 7 Done Done Sent false
1041 1179 Acked Read 10 Done Done Sent false
1042 1180 Acked Write 4 Done Done Sent false
1043 1181 Acked Read 4 Done Done Sent false
1044 1182 Acked Read 1 Done Done Sent false
1045 1183 Acked Read 4 Done Done Sent false
1046 1184 Acked Write 10 Done Done Sent false
1047 1185 Acked Write 6 Done Done Sent false
1048 1186 Acked Read 6 Done Done Sent false
1049 1187 Acked Write 3 Done Done Sent false
1050 1188 Acked Write 4 Done Done Sent false
1051 1189 Acked Write 8 Done Done Sent false
1052 1190 Acked Read 3 Done Done Sent false
1053 1191 Acked Write 6 Done Done Sent false
1054 1192 Acked Read 1 Done Done Sent false
1055 1193 Acked Read 7 Done Done Sent false
1056 1194 Acked Read 10 Done Done Sent false
1057 1195 Acked Read 5 Done Done Sent false
1058 1196 Acked Write 10 Done Done Sent false
1059 1197 Acked Read 9 Done Done Sent false
1060 1198 Acked Read 9 Done Done Sent false
1061 1199 Acked Flush 0 Done Done Sent false
1062 1200 Acked Read 9 Done Done Sent false
1063 1201 Acked Flush 0 Done Done Sent false
1064 1202 Acked Write 1 Done Done Sent false
1065 1203 Acked Write 4 Done Done Sent false
1066 1204 Acked Write 3 Done Done Sent false
1067 1205 Acked Read 9 Done Done Sent false
1068 1206 Acked Read 5 Done Done Sent false
1069 1207 Acked Read 9 Done Done Sent false
1070 1208 Acked Read 4 Done Done Sent false
1071 1209 Acked Flush 0 Done Done Sent false
1072 1210 Acked Read 8 Done Done Sent false
1073 1211 Acked Write 5 Done Done Sent false
1074 1212 Acked Write 3 Done Done Sent false
1075 1213 Acked Flush 0 Done Done Sent false
1076 1214 Acked Flush 0 Done Done Sent false
1077 1215 Acked Read 1 Done Done Sent false
1078 1216 Acked Write 8 Done Done Sent false
1079 1217 Acked Write 5 Done Done Sent false
1080 1218 Acked Write 6 Done Done Sent false
1081 1219 Acked Read 8 Done Done Sent false
1082 1220 Acked Read 8 Done Done Sent false
1083 1221 Acked Write 6 Done Done Sent false
1084 1222 Acked Write 8 Done Done Sent false
1085 1223 Acked Read 2 Done Done Sent false
1086 1224 Acked Flush 0 Done Done Sent false
1087 1225 Acked Read 8 Done Done Sent false
1088 1226 Acked Read 5 Done Done Sent false
1089 1227 Acked Flush 0 Done Done Sent false
1090 1228 Acked Write 2 Done Done Sent false
1091 1229 Acked Read 9 Done Done Sent false
1092 1230 Acked Read 5 Done Done Sent false
1093 1231 Acked Write 8 Done Done Sent false
1094 1232 Acked Write 10 Done Done Sent false
1095 1233 Acked Read 1 Done Done Sent false
1096 1234 Acked Read 6 Done Done Sent false
1097 1235 Acked Write 7 Done Done Sent false
1098 1236 Acked Write 3 Done Done Sent false
1099 1237 Acked Write 8 Done Done Sent false
1100 1238 Acked Flush 0 Done Done Sent false
1101 1239 Acked Write 1 Done Done Sent false
1102 1240 Acked Write 6 Done Done Sent false
1103 1241 Acked Write 2 Done Done Sent false
1104 1242 Acked Read 3 Done Done Sent false
1105 1243 Acked Read 8 Done Done Sent false
1106 1244 Acked Read 10 Done Done Sent false
1107 1245 Acked Read 2 Done Done Sent false
1108 1246 Acked Read 9 Done Done Sent false
1109 1247 Acked Read 8 Done Done Sent false
1110 1248 Acked Read 3 Done Done Sent false
1111 1249 Acked Read 2 Done Done Sent false
1112 1250 Acked Read 7 Done Done Sent false
1113 1251 Acked Read 6 Done Done Sent false
1114 1252 Acked Read 1 Done Done Sent false
1115 1253 Acked Read 5 Done Done Sent false
1116 1254 Acked Write 9 Done Done Sent false
1117 1255 Acked Write 9 Done Done Sent false
1118 1256 Acked Write 3 Done Done Sent false
1119 1257 Acked Write 6 Done Done Sent false
1120 1258 Acked Read 10 Done Done Sent false
1121 1259 Acked Read 3 Done Done Sent false
1122 1260 Acked Read 7 Done Done Sent false
1123 1261 Acked Write 2 Done Done Sent false
1124 1262 Acked Write 4 Done Done Sent false
1125 1263 Acked Write 7 Done Done Sent false
1126 1264 Acked Flush 0 Done Done Sent false
1127 1265 Acked Write 10 Done Done Sent false
1128 1266 Acked Read 10 Done Done Sent false
1129 1267 Acked Write 9 Done Done Sent false
1130 1268 Acked Read 5 Done Done Sent false
1131 1269 Acked Read 10 Done Done Sent false
1132 1270 Acked Read 8 Done Done Sent false
1133 1271 Acked Write 2 Done Done Sent false
1134 1272 Acked Write 6 Done Done Sent false
1135 1273 Acked Write 3 Done Done Sent false
1136 1274 Acked Read 1 Done Done Sent false
1137 1275 Acked Read 8 Done Done Sent false
1138 1276 Acked Write 7 Done Done Sent false
1139 1277 Acked Read 1 Done Done Sent false
1140 1278 Acked Read 1 Done Done Sent false
1141 1279 Acked Read 6 Done Done Sent false
1142 1280 Acked Write 1 Done Done Sent false
1143 1281 Acked Write 5 Done Done Sent false
1144 1282 Acked Read 10 Done Done Sent false
1145 1283 Acked Read 6 Done Done Sent false
1146 1284 Acked Flush 0 Done Done Sent false
1147 1285 Acked Write 10 Done Done Sent false
1148 1286 Acked Write 9 Done Done Sent false
1149 1287 Acked Flush 0 Done Done Sent false
1150 1288 Acked Read 3 Done Done Sent false
1151 1289 Acked Read 10 Done Done Sent false
1152 1290 Acked Write 4 Done Done Sent false
1153 1291 Acked Write 4 Done Done Sent false
1154 1292 Acked Read 4 Done Done Sent false
1155 1293 Acked Write 1 Done Done Sent false
1156 1294 Acked Read 1 Done Done Sent false
1157 1295 Acked Read 10 Done Done Sent false
1158 1296 Acked Read 9 Done Done Sent false
1159 1297 Acked Flush 0 Done Done Sent false
1160 1298 Acked Read 10 Done Done Sent false
1161 1299 Acked Read 1 Done Done Sent false
1162 1300 Acked Write 1 Done Done Sent false
1163 STATES DS:0 DS:1 DS:2 TOTAL
1164 Sent 0 0 301 301
1165 Done 301 301 0 602
1166 Skipped 0 0 0 0
1167 Error 0 0 0 0
1168 Last Flush: Some(JobId(1297)) Some(JobId(1297)) None
1169 Downstairs last five completed:
1170 Upstairs last five completed: 1300 1299 1298 1297 1296
1171 CLIENT: Up:0 ds:301 act:2
11722025-03-17T22:16:27.900ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
11732025-03-17T22:16:28.612ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
11742025-03-17T22:16:29.134ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
11752025-03-17T22:16:29.134ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58752) = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
11762025-03-17T22:16:29.135ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
11772025-03-17T22:16:29.136ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
11782025-03-17T22:16:29.136ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
11792025-03-17T22:16:29.136ZINFOcrucible: send last flush ID to this DS: None = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
11802025-03-17T22:16:29.136ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
11812025-03-17T22:16:29.136ZINFOcrucible: [2] client re-new 301 jobs since flush None = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
11822025-03-17T22:16:29.136ZINFOcrucible: [2] Replayed 301 jobs since flush: None = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1183 ----------------------------------------------------------------
1184 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
1185 Downstairs last five completed: 1301 1300 1299 1298 1297
1186 Upstairs last five completed: 1301 1300 1299 1298 1297
1187 CLIENT: Up:0 ds:0 act:3
1188 Replay: All jobs finished
1189 Main task has work to do, go find it
1190 got dsc Stop(1)
1191 stop 1
1192 Mar 17 22:16:31.400 INFO request completed, latency_us: 310, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 2bb3c366-46b7-4232-b531-0f7ff916fc47, remote_addr: 127.0.0.1:60423, [8820] Got stop action so:false kr:false
1193 local_addr: 127.0.0.1:9998
1194 Mar 17 22:16:31.401 INFO request completed, latency_us: 159, response_code: 200, uri: /state/cid/1, method: GET, req_id: 2c84c631-c5c7-4177-aa4c-a37d955915b9, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
11952025-03-17T22:16:31.401ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
11962025-03-17T22:16:31.401ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
11972025-03-17T22:16:31.401ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
11982025-03-17T22:16:31.401ZINFOcrucible: client task is exiting = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1199 [8820] Exited with: ExitStatus(unix_wait_status(9))
1200 [8820][1] reports Exit
1201 Mar 17 22:16:35.404 INFO request completed, latency_us: 299, response_code: 200, uri: /state/cid/1, method: GET, req_id: f3d826e3-7ace-4db0-a10e-f68cb4b80b82, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
1202 001/300 Read block 1970 len 8192
1203 002/300 Read block 1771 len 32768
1204 003/300 Read block 4349 len 8192
1205 004/300 Write block 3872 len 4096 data: 3
1206 005/300 Write block 497 len 40960 data: 2 2 2 2 2 3 3 3 3 3
1207 006/300 Read block 560 len 24576
1208 007/300 Write block 4757 len 32768 data: 2 2 2 2 2 2 3 3
1209 008/300 Write block 185 len 20480 data: 4 3 3 3 3
1210 009/300 Write block 4563 len 28672 data: 2 2 2 2 2 2 2
1211 010/300 Read block 1179 len 28672
1212 011/300 Flush
1213 012/300 Write block 2058 len 24576 data: 4 4 3 3 3 3
1214 013/300 Read block 3431 len 4096
1215 014/300 Write block 4848 len 20480 data: 2 2 2 2 2
1216 015/300 Read block 4746 len 12288
1217 016/300 Write block 4718 len 28672 data: 2 2 2 2 2 2 2
1218 017/300 Write block 2623 len 32768 data: 2 2 2 2 2 2 2 2
1219 018/300 Write block 2824 len 8192 data: 2 2
1220 019/300 Read block 1749 len 20480
1221 020/300 Write block 4729 len 8192 data: 2 2
1222 021/300 Read block 328 len 36864
1223 022/300 Write block 1637 len 24576 data: 2 2 2 3 3 3
1224 023/300 Read block 4659 len 36864
1225 024/300 Write block 890 len 28672 data: 2 2 2 2 2 2 2
1226 025/300 Read block 4125 len 4096
1227 026/300 Read block 2376 len 12288
1228 027/300 Flush
1229 028/300 Write block 4098 len 16384 data: 2 2 2 2
1230 029/300 Write block 2555 len 20480 data: 3 3 3 3 2
1231 030/300 Write block 3346 len 8192 data: 2 3
1232 031/300 Write block 654 len 28672 data: 3 3 3 2 2 2 2
1233 032/300 Write block 4602 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1234 033/300 Read block 4125 len 8192
1235 034/300 Flush
1236 035/300 Write block 4862 len 20480 data: 3 3 2 2 2
1237 036/300 Read block 3218 len 36864
1238 037/300 Read block 3298 len 36864
1239 038/300 Read block 1350 len 36864
1240 039/300 Write block 3987 len 4096 data: 2
1241 040/300 Write block 1215 len 32768 data: 2 2 2 2 2 2 2 2
1242 041/300 Read block 1417 len 12288
1243 042/300 Write block 264 len 4096 data: 2
1244 043/300 Write block 2366 len 24576 data: 2 2 2 2 2 2
1245 044/300 Read block 1100 len 8192
1246 045/300 Write block 4606 len 12288 data: 3 3 3
1247 046/300 Read block 3678 len 4096
1248 047/300 Read block 4268 len 40960
1249 048/300 Read block 3243 len 32768
1250 049/300 Write block 4288 len 28672 data: 2 2 2 2 2 2 2
1251 050/300 Read block 4370 len 24576
1252 051/300 Flush
1253 052/300 Write block 4547 len 4096 data: 2
1254 053/300 Read block 3460 len 16384
1255 054/300 Write block 1459 len 36864 data: 2 2 2 2 2 2 3 3 3
1256 055/300 Write block 4342 len 12288 data: 3 2 2
1257 056/300 Read block 2686 len 24576
1258 057/300 Read block 4487 len 32768
1259 058/300 Read block 4152 len 16384
1260 059/300 Read block 1486 len 16384
1261 060/300 Read block 3067 len 24576
1262 061/300 Write block 2016 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1263 062/300 Read block 4958 len 40960
1264 063/300 Write block 3323 len 24576 data: 2 2 2 2 2 2
1265 064/300 Write block 4470 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1266 065/300 Read block 2736 len 16384
1267 066/300 Write block 1798 len 16384 data: 2 2 2 2
1268 067/300 Read block 419 len 24576
1269 068/300 Read block 2531 len 36864
1270 069/300 Write block 2930 len 20480 data: 3 3 3 3 3
1271 070/300 Read block 3593 len 40960
1272 071/300 Read block 1728 len 4096
1273 072/300 Read block 3404 len 40960
1274 073/300 Write block 1928 len 4096 data: 2
1275 074/300 Flush
1276 075/300 Flush
1277 076/300 Flush
1278 077/300 Read block 2027 len 20480
1279 078/300 Read block 408 len 4096
1280 079/300 Read block 3795 len 8192
1281 080/300 Write block 4793 len 36864 data: 2 2 2 2 2 2 2 2 2
1282 081/300 Write block 4063 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1283 082/300 Write block 714 len 8192 data: 2 2
1284 083/300 Write block 2594 len 4096 data: 2
1285 084/300 Write block 4477 len 16384 data: 3 3 3 2
1286 085/300 Write block 3166 len 8192 data: 2 2
1287 086/300 Write block 4068 len 8192 data: 3 3
1288 087/300 Read block 133 len 36864
1289 088/300 Write block 4625 len 8192 data: 2 2
1290 089/300 Read block 1145 len 4096
1291 090/300 Read block 1327 len 36864
1292 091/300 Read block 4704 len 40960
1293 092/300 Read block 4372 len 16384
1294 093/300 Flush
1295 094/300 Read block 1123 len 16384
1296 095/300 Flush
1297 096/300 Read block 186 len 36864
1298 097/300 Write block 74 len 28672 data: 2 2 2 2 2 2 2
1299 098/300 Write block 2844 len 12288 data: 2 2 2
1300 099/300 Write block 1996 len 28672 data: 2 2 2 2 2 2 2
1301 100/300 Write block 1595 len 20480 data: 2 2 2 2 2
1302 101/300 Write block 1669 len 36864 data: 2 2 2 2 2 2 2 2 2
1303 102/300 Write block 3669 len 4096 data: 2
1304 103/300 Read block 989 len 40960
1305 104/300 Flush
1306 105/300 Write block 2323 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1307 106/300 Write block 4398 len 12288 data: 3 3 3
1308 107/300 Write block 201 len 12288 data: 2 2 2
1309 108/300 Read block 1905 len 28672
1310 109/300 Write block 4684 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1311 110/300 Write block 2001 len 12288 data: 3 3 2
1312 111/300 Flush
1313 112/300 Read block 2923 len 32768
1314 113/300 Read block 3109 len 4096
1315 114/300 Read block 3814 len 12288
1316 115/300 Flush
1317 116/300 Read block 576 len 28672
1318 117/300 Write block 3753 len 20480 data: 2 2 2 2 2
1319 118/300 Read block 1407 len 32768
1320 119/300 Flush
1321 120/300 Flush
1322 121/300 Flush
1323 122/300 Read block 1129 len 4096
1324 123/300 Read block 1909 len 8192
1325 124/300 Write block 3168 len 16384 data: 2 2 2 2
1326 125/300 Read block 3052 len 4096
1327 126/300 Write block 86 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1328 127/300 Read block 2188 len 8192
1329 128/300 Read block 3765 len 8192
1330 129/300 Write block 253 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1331 130/300 Read block 1218 len 40960
1332 131/300 Read block 1325 len 12288
1333 132/300 Write block 2110 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1334 133/300 Read block 2694 len 24576
1335 134/300 Write block 3269 len 32768 data: 2 2 2 2 2 2 2 2
1336 135/300 Read block 695 len 32768
1337 136/300 Read block 3989 len 20480
1338 137/300 Read block 4637 len 28672
1339 138/300 Read block 538 len 28672
1340 139/300 Write block 4900 len 16384 data: 2 2 2 2
1341 140/300 Read block 2590 len 28672
1342 141/300 Write block 265 len 8192 data: 2 2
1343 142/300 Read block 3230 len 20480
1344 143/300 Flush
1345 144/300 Read block 2658 len 40960
1346 145/300 Read block 2150 len 24576
1347 146/300 Write block 4324 len 20480 data: 2 2 2 2 2
1348 147/300 Read block 966 len 4096
1349 148/300 Read block 1452 len 4096
1350 149/300 Read block 4171 len 16384
1351 150/300 Write block 3896 len 28672 data: 2 2 2 2 2 2 2
1352 151/300 Write block 3104 len 32768 data: 3 3 3 3 3 3 3 2
1353 152/300 Write block 4160 len 20480 data: 2 2 2 2 2
1354 153/300 Read block 208 len 40960
1355 154/300 Write block 1492 len 8192 data: 2 2
1356 155/300 Write block 1275 len 16384 data: 2 2 2 2
1357 156/300 Read block 2203 len 40960
1358 157/300 Read block 1936 len 28672
1359 158/300 Write block 4280 len 28672 data: 2 2 2 2 2 2 2
1360 159/300 Read block 1544 len 24576
1361 160/300 Write block 3416 len 8192 data: 2 2
1362 161/300 Read block 4024 len 24576
1363 162/300 Write block 3042 len 28672 data: 2 2 2 2 2 2 2
1364 163/300 Write block 1113 len 28672 data: 2 2 2 2 2 2 2
1365 164/300 Write block 829 len 4096 data: 2
1366 165/300 Read block 2898 len 40960
1367 166/300 Read block 2016 len 24576
1368 167/300 Read block 2577 len 36864
1369 168/300 Write block 940 len 24576 data: 2 2 2 2 2 2
1370 169/300 Write block 703 len 4096 data: 2
1371 170/300 Write block 4296 len 36864 data: 2 2 2 2 2 3 3 3 3
1372 171/300 Flush
1373 172/300 Read block 92 len 8192
1374 173/300 Read block 2916 len 28672
1375 174/300 Read block 3051 len 28672
1376 175/300 Read block 2830 len 24576
1377 176/300 Read block 2192 len 36864
1378 177/300 Write block 418 len 12288 data: 2 2 2
1379 178/300 Write block 2277 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1380 179/300 Write block 2515 len 8192 data: 2 2
1381 180/300 Write block 2170 len 4096 data: 2
1382 181/300 Read block 2305 len 40960
1383 182/300 Write block 930 len 16384 data: 4 4 4 3
1384 183/300 Write block 4105 len 16384 data: 2 2 2 2
1385 184/300 Write block 4802 len 16384 data: 2 2 2 3
1386 185/300 Read block 4198 len 8192
1387 186/300 Flush
1388 187/300 Write block 3727 len 20480 data: 2 2 2 3 3
1389 188/300 Read block 403 len 20480
1390 189/300 Read block 56 len 32768
1391 190/300 Read block 4612 len 16384
1392 191/300 Write block 3275 len 8192 data: 3 3
1393 192/300 Write block 1985 len 8192 data: 2 2
1394 193/300 Read block 863 len 24576
1395 194/300 Read block 4952 len 28672
1396 195/300 Read block 1196 len 4096
1397 196/300 Read block 457 len 8192
1398 197/300 Read block 4625 len 16384
1399 198/300 Write block 3018 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1400 199/300 Read block 823 len 36864
1401 200/300 Read block 2822 len 16384
1402 201/300 Write block 2595 len 24576 data: 2 2 2 2 3 3
1403 202/300 Read block 2354 len 40960
1404 203/300 Write block 3091 len 32768 data: 2 2 2 2 2 2 2 2
1405 204/300 Write block 2885 len 12288 data: 2 2 3
1406 205/300 Write block 714 len 36864 data: 3 3 3 2 2 2 2 2 2
1407 206/300 Write block 896 len 20480 data: 3 2 2 2 2
1408 207/300 Read block 4200 len 8192
1409 208/300 Write block 3643 len 8192 data: 2 2
1410 209/300 Read block 478 len 8192
1411 210/300 Write block 2246 len 32768 data: 2 2 2 2 2 2 2 2
1412 211/300 Read block 2245 len 20480
1413 212/300 Read block 3367 len 12288
1414 213/300 Write block 56 len 4096 data: 2
1415 214/300 Read block 37 len 24576
1416 215/300 Read block 2699 len 8192
1417 216/300 Read block 1513 len 32768
1418 217/300 Flush
1419 218/300 Write block 1555 len 32768 data: 2 2 2 2 2 2 2 2
1420 219/300 Read block 3726 len 40960
1421 220/300 Read block 3120 len 28672
1422 221/300 Write block 2471 len 24576 data: 2 3 3 3 3 3
1423 222/300 Write block 4330 len 20480 data: 2 2 2 2 2
1424 223/300 Write block 4723 len 40960 data: 3 3 2 2 2 2 3 3 2 2
1425 224/300 Write block 2181 len 28672 data: 2 2 2 2 2 2 2
1426 225/300 Flush
1427 226/300 Read block 2247 len 40960
1428 227/300 Read block 2995 len 16384
1429 228/300 Write block 36 len 12288 data: 2 2 2
1430 229/300 Write block 501 len 36864 data: 3 4 4 4 4 4 3 2 2
1431 230/300 Write block 4406 len 16384 data: 2 2 2 2
1432 231/300 Flush
1433 232/300 Flush
1434 233/300 Write block 986 len 36864 data: 2 2 2 2 2 2 2 2 2
1435 234/300 Read block 2746 len 4096
1436 235/300 Read block 668 len 36864
1437 236/300 Read block 3766 len 32768
1438 237/300 Read block 4372 len 20480
1439 238/300 Write block 1882 len 4096 data: 2
1440 239/300 Read block 3554 len 24576
1441 240/300 Write block 3143 len 12288 data: 3 3 3
1442 241/300 Read block 766 len 24576
1443 242/300 Read block 623 len 40960
1444 243/300 Flush
1445 244/300 Read block 3978 len 28672
1446 245/300 Write block 2757 len 24576 data: 2 2 2 2 2 2
1447 246/300 Write block 2868 len 36864 data: 2 2 2 2 2 2 2 2 3
1448 247/300 Read block 3413 len 16384
1449 248/300 Read block 4716 len 40960
1450 249/300 Flush
1451 250/300 Read block 1045 len 40960
1452 251/300 Read block 4078 len 16384
1453 252/300 Flush
1454 253/300 Read block 1049 len 28672
1455 254/300 Flush
1456 255/300 Write block 4530 len 4096 data: 2
1457 256/300 Flush
1458 257/300 Write block 4057 len 32768 data: 3 3 3 2 2 2 3 3
1459 258/300 Flush
1460 259/300 Write block 3857 len 24576 data: 2 2 2 2 2 2
1461 260/300 Write block 997 len 8192 data: 2 2
1462 261/300 Read block 2757 len 24576
1463 262/300 Flush
1464 263/300 Write block 3880 len 16384 data: 2 2 2 2
1465 264/300 Read block 3906 len 40960
1466 265/300 Read block 2814 len 24576
1467 266/300 Read block 1244 len 12288
1468 267/300 Read block 1450 len 24576
1469 268/300 Flush
1470 269/300 Read block 3185 len 4096
1471 270/300 Flush
1472 271/300 Write block 2883 len 32768 data: 2 2 3 3 4 3 3 2
1473 272/300 Write block 1527 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1474 273/300 Read block 1702 len 40960
1475 274/300 Read block 236 len 12288
1476 275/300 Write block 1522 len 24576 data: 2 2 2 2 2 3
1477 276/300 Write block 4903 len 20480 data: 3 2 2 2 2
1478 277/300 Write block 4247 len 36864 data: 2 2 2 2 2 2 2 2 2
1479 278/300 Read block 4570 len 28672
1480 279/300 Write block 2358 len 8192 data: 2 2
1481 280/300 Read block 3371 len 36864
1482 281/300 Read block 2561 len 16384
1483 282/300 Read block 3007 len 20480
1484 283/300 Write block 1308 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1485 284/300 Read block 1239 len 16384
1486 285/300 Read block 4493 len 12288
1487 286/300 Flush
1488 287/300 Read block 198 len 40960
1489 288/300 Write block 1866 len 36864 data: 2 2 2 2 2 2 2 2 2
1490 289/300 Read block 4538 len 4096
1491 290/300 Write block 1731 len 20480 data: 2 2 2 2 2
1492 291/300 Read block 1484 len 12288
1493 292/300 Flush
1494 293/300 Read block 1747 len 36864
1495 294/300 Read block 860 len 28672
1496 295/300 Read block 3541 len 36864
1497 296/300 Write block 3444 len 12288 data: 2 2 2
1498 297/300 Read block 2821 len 36864
1499 298/300 Write block 2911 len 16384 data: 2 2 2 2
1500 299/300 Write block 80 len 12288 data: 3 2 2
1501 300/300 Read block 1042 len 8192
1502 301/300 Write block 2617 len 20480 data: 2 2 2 2 2
1503 Main task has work to do, go find it
1504 got dsc Start(1)
1505 start 1
1506 Mar 17 22:16:35.648 INFO request completed, latency_us: 228, response_code: 204, uri: /start/cid/1, method: POST, req_id: d50c8e6a-a0d0-42e9-a7e1-36620bbe6df8, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
1507 Replay: started 1, returned:Ok(())
1508 ----------------------------------------------------------------
1509 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1510 States: Active Offline Active
1511 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1512 1302 Acked Read 2 Done Sent Done false
1513 1303 Acked Read 8 Done Sent Done false
1514 1304 Acked Read 2 Done Sent Done false
1515 1305 Acked Write 1 Done Sent Done false
1516 1306 Acked Write 10 Done Sent Done false
1517 1307 Acked Read 6 Done Sent Done false
1518 1308 Acked Write 8 Done Sent Done false
1519 1309 Acked Write 5 Done Sent Done false
1520 1310 Acked Write 7 Done Sent Done false
1521 1311 Acked Read 7 Done Sent Done false
1522 1312 Acked Flush 0 Done Sent Done false
1523 1313 Acked Write 6 Done Sent Done false
1524 1314 Acked Read 1 Done Sent Done false
1525 1315 Acked Write 5 Done Sent Done false
1526 1316 Acked Read 3 Done Sent Done false
1527 1317 Acked Write 7 Done Sent Done false
1528 1318 Acked Write 8 Done Sent Done false
1529 1319 Acked Write 2 Done Sent Done false
1530 1320 Acked Read 5 Done Sent Done false
1531 1321 Acked Write 2 Done Sent Done false
1532 1322 Acked Read 9 Done Sent Done false
1533 1323 Acked Write 6 Done Sent Done false
1534 1324 Acked Read 9 Done Sent Done false
1535 1325 Acked Write 7 Done Sent Done false
1536 1326 Acked Read 1 Done Sent Done false
1537 1327 Acked Read 3 Done Sent Done false
1538 1328 Acked Flush 0 Done Sent Done false
1539 1329 Acked Write 4 Done Sent Done false
1540 1330 Acked Write 5 Done Sent Done false
1541 1331 Acked Write 2 Done Sent Done false
1542 1332 Acked Write 7 Done Sent Done false
1543 1333 Acked Write 10 Done Sent Done false
1544 1334 Acked Read 2 Done Sent Done false
1545 1335 Acked Flush 0 Done Sent Done false
1546 1336 Acked Write 5 Done Sent Done false
1547 1337 Acked Read 9 Done Sent Done false
1548 1338 Acked Read 9 Done Sent Done false
1549 1339 Acked Read 9 Done Sent Done false
1550 1340 Acked Write 1 Done Sent Done false
1551 1341 Acked Write 8 Done Sent Done false
1552 1342 Acked Read 3 Done Sent Done false
1553 1343 Acked Write 1 Done Sent Done false
1554 1344 Acked Write 6 Done Sent Done false
1555 1345 Acked Read 2 Done Sent Done false
1556 1346 Acked Write 3 Done Sent Done false
1557 1347 Acked Read 1 Done Sent Done false
1558 1348 Acked Read 10 Done Sent Done false
1559 1349 Acked Read 8 Done Sent Done false
1560 1350 Acked Write 7 Done Sent Done false
1561 1351 Acked Read 6 Done Sent Done false
1562 1352 Acked Flush 0 Done Sent Done false
1563 1353 Acked Write 1 Done Sent Done false
1564 1354 Acked Read 4 Done Sent Done false
1565 1355 Acked Write 9 Done Sent Done false
1566 1356 Acked Write 3 Done Sent Done false
1567 1357 Acked Read 6 Done Sent Done false
1568 1358 Acked Read 8 Done Sent Done false
1569 1359 Acked Read 4 Done Sent Done false
1570 1360 Acked Read 4 Done Sent Done false
1571 1361 Acked Read 6 Done Sent Done false
1572 1362 Acked Write 10 Done Sent Done false
1573 1363 Acked Read 10 Done Sent Done false
1574 1364 Acked Write 6 Done Sent Done false
1575 1365 Acked Write 10 Done Sent Done false
1576 1366 Acked Read 4 Done Sent Done false
1577 1367 Acked Write 4 Done Sent Done false
1578 1368 Acked Read 6 Done Sent Done false
1579 1369 Acked Read 9 Done Sent Done false
1580 1370 Acked Write 5 Done Sent Done false
1581 1371 Acked Read 10 Done Sent Done false
1582 1372 Acked Read 1 Done Sent Done false
1583 1373 Acked Read 10 Done Sent Done false
1584 1374 Acked Write 1 Done Sent Done false
1585 1375 Acked Flush 0 Done Sent Done false
1586 1376 Acked Flush 0 Done Sent Done false
1587 1377 Acked Flush 0 Done Sent Done false
1588 1378 Acked Read 5 Done Sent Done false
1589 1379 Acked Read 1 Done Sent Done false
1590 1380 Acked Read 2 Done Sent Done false
1591 1381 Acked Write 9 Done Sent Done false
1592 1382 Acked Write 10 Done Sent Done false
1593 1383 Acked Write 2 Done Sent Done false
1594 1384 Acked Write 1 Done Sent Done false
1595 1385 Acked Write 4 Done Sent Done false
1596 1386 Acked Write 2 Done Sent Done false
1597 1387 Acked Write 2 Done Sent Done false
1598 1388 Acked Read 9 Done Sent Done false
1599 1389 Acked Write 2 Done Sent Done false
1600 1390 Acked Read 1 Done Sent Done false
1601 1391 Acked Read 9 Done Sent Done false
1602 1392 Acked Read 10 Done Sent Done false
1603 1393 Acked Read 4 Done Sent Done false
1604 1394 Acked Flush 0 Done Sent Done false
1605 1395 Acked Read 4 Done Sent Done false
1606 1396 Acked Flush 0 Done Sent Done false
1607 1397 Acked Read 9 Done Sent Done false
1608 1398 Acked Write 7 Done Sent Done false
1609 1399 Acked Write 3 Done Sent Done false
1610 1400 Acked Write 7 Done Sent Done false
1611 1401 Acked Write 5 Done Sent Done false
1612 1402 Acked Write 9 Done Sent Done false
1613 1403 Acked Write 1 Done Sent Done false
1614 1404 Acked Read 10 Done Sent Done false
1615 1405 Acked Flush 0 Done Sent Done false
1616 1406 Acked Write 10 Done Sent Done false
1617 1407 Acked Write 3 Done Sent Done false
1618 1408 Acked Write 3 Done Sent Done false
1619 1409 Acked Read 7 Done Sent Done false
1620 1410 Acked Write 10 Done Sent Done false
1621 1411 Acked Write 3 Done Sent Done false
1622 1412 Acked Flush 0 Done Sent Done false
1623 1413 Acked Read 8 Done Sent Done false
1624 1414 Acked Read 1 Done Sent Done false
1625 1415 Acked Read 3 Done Sent Done false
1626 1416 Acked Flush 0 Done Sent Done false
1627 1417 Acked Read 7 Done Sent Done false
1628 1418 Acked Write 5 Done Sent Done false
1629 1419 Acked Read 8 Done Sent Done false
1630 1420 Acked Flush 0 Done Sent Done false
1631 1421 Acked Flush 0 Done Sent Done false
1632 1422 Acked Flush 0 Done Sent Done false
1633 1423 Acked Read 1 Done Sent Done false
1634 1424 Acked Read 2 Done Sent Done false
1635 1425 Acked Write 4 Done Sent Done false
1636 1426 Acked Read 1 Done Sent Done false
1637 1427 Acked Write 10 Done Sent Done false
1638 1428 Acked Read 2 Done Sent Done false
1639 1429 Acked Read 2 Done Sent Done false
1640 1430 Acked Write 10 Done Sent Done false
1641 1431 Acked Read 10 Done Sent Done false
1642 1432 Acked Read 3 Done Sent Done false
1643 1433 Acked Write 10 Done Sent Done false
1644 1434 Acked Read 6 Done Sent Done false
1645 1435 Acked Write 8 Done Sent Done false
1646 1436 Acked Read 8 Done Sent Done false
1647 1437 Acked Read 5 Done Sent Done false
1648 1438 Acked Read 7 Done Sent Done false
1649 1439 Acked Read 7 Done Sent Done false
1650 1440 Acked Write 4 Done Sent Done false
1651 1441 Acked Read 7 Done Sent Done false
1652 1442 Acked Write 2 Done Sent Done false
1653 1443 Acked Read 5 Done Sent Done false
1654 1444 Acked Flush 0 Done Sent Done false
1655 1445 Acked Read 10 Done Sent Done false
1656 1446 Acked Read 6 Done Sent Done false
1657 1447 Acked Write 5 Done Sent Done false
1658 1448 Acked Read 1 Done Sent Done false
1659 1449 Acked Read 1 Done Sent Done false
1660 1450 Acked Read 4 Done Sent Done false
1661 1451 Acked Write 7 Done Sent Done false
1662 1452 Acked Write 8 Done Sent Done false
1663 1453 Acked Write 5 Done Sent Done false
1664 1454 Acked Read 10 Done Sent Done false
1665 1455 Acked Write 2 Done Sent Done false
1666 1456 Acked Write 4 Done Sent Done false
1667 1457 Acked Read 10 Done Sent Done false
1668 1458 Acked Read 7 Done Sent Done false
1669 1459 Acked Write 7 Done Sent Done false
1670 1460 Acked Read 6 Done Sent Done false
1671 1461 Acked Write 2 Done Sent Done false
1672 1462 Acked Read 6 Done Sent Done false
1673 1463 Acked Write 7 Done Sent Done false
1674 1464 Acked Write 7 Done Sent Done false
1675 1465 Acked Write 1 Done Sent Done false
1676 1466 Acked Read 10 Done Sent Done false
1677 1467 Acked Read 6 Done Sent Done false
1678 1468 Acked Read 9 Done Sent Done false
1679 1469 Acked Write 6 Done Sent Done false
1680 1470 Acked Write 1 Done Sent Done false
1681 1471 Acked Write 9 Done Sent Done false
1682 1472 Acked Flush 0 Done Sent Done false
1683 1473 Acked Read 2 Done Sent Done false
1684 1474 Acked Read 7 Done Sent Done false
1685 1475 Acked Read 7 Done Sent Done false
1686 1476 Acked Read 6 Done Sent Done false
1687 1477 Acked Read 9 Done Sent Done false
1688 1478 Acked Write 3 Done Sent Done false
1689 1479 Acked Write 10 Done Sent Done false
1690 1480 Acked Write 2 Done Sent Done false
1691 1481 Acked Write 1 Done Sent Done false
1692 1482 Acked Read 10 Done Sent Done false
1693 1483 Acked Write 4 Done Sent Done false
1694 1484 Acked Write 4 Done Sent Done false
1695 1485 Acked Write 4 Done Sent Done false
1696 1486 Acked Read 2 Done Sent Done false
1697 1487 Acked Flush 0 Done Sent Done false
1698 1488 Acked Write 5 Done Sent Done false
1699 1489 Acked Read 5 Done Sent Done false
1700 1490 Acked Read 8 Done Sent Done false
1701 1491 Acked Read 4 Done Sent Done false
1702 1492 Acked Write 2 Done Sent Done false
1703 1493 Acked Write 2 Done Sent Done false
1704 1494 Acked Read 6 Done Sent Done false
1705 1495 Acked Read 7 Done Sent Done false
1706 1496 Acked Read 1 Done Sent Done false
1707 1497 Acked Read 2 Done Sent Done false
1708 1498 Acked Read 4 Done Sent Done false
1709 1499 Acked Write 10 Done Sent Done false
1710 1500 Acked Read 9 Done Sent Done false
1711 1501 Acked Read 4 Done Sent Done false
1712 1502 Acked Write 6 Done Sent Done false
1713 1503 Acked Read 10 Done Sent Done false
1714 1504 Acked Write 8 Done Sent Done false
1715 1505 Acked Write 3 Done Sent Done false
1716 1506 Acked Write 9 Done Sent Done false
1717 1507 Acked Write 5 Done Sent Done false
1718 1508 Acked Read 2 Done Sent Done false
1719 1509 Acked Write 2 Done Sent Done false
1720 1510 Acked Read 2 Done Sent Done false
1721 1511 Acked Write 8 Done Sent Done false
1722 1512 Acked Read 5 Done Sent Done false
1723 1513 Acked Read 3 Done Sent Done false
1724 1514 Acked Write 1 Done Sent Done false
1725 1515 Acked Read 6 Done Sent Done false
1726 1516 Acked Read 2 Done Sent Done false
1727 1517 Acked Read 8 Done Sent Done false
1728 1518 Acked Flush 0 Done Sent Done false
1729 1519 Acked Write 8 Done Sent Done false
1730 1520 Acked Read 10 Done Sent Done false
1731 1521 Acked Read 7 Done Sent Done false
1732 1522 Acked Write 6 Done Sent Done false
1733 1523 Acked Write 5 Done Sent Done false
1734 1524 Acked Write 10 Done Sent Done false
1735 1525 Acked Write 7 Done Sent Done false
1736 1526 Acked Flush 0 Done Sent Done false
1737 1527 Acked Read 10 Done Sent Done false
1738 1528 Acked Read 4 Done Sent Done false
1739 1529 Acked Write 3 Done Sent Done false
1740 1530 Acked Write 9 Done Sent Done false
1741 1531 Acked Write 4 Done Sent Done false
1742 1532 Acked Flush 0 Done Sent Done false
1743 1533 Acked Flush 0 Done Sent Done false
1744 1534 Acked Write 9 Done Sent Done false
1745 1535 Acked Read 1 Done Sent Done false
1746 1536 Acked Read 9 Done Sent Done false
1747 1537 Acked Read 8 Done Sent Done false
1748 1538 Acked Read 5 Done Sent Done false
1749 1539 Acked Write 1 Done Sent Done false
1750 1540 Acked Read 6 Done Sent Done false
1751 1541 Acked Write 3 Done Sent Done false
1752 1542 Acked Read 6 Done Sent Done false
1753 1543 Acked Read 10 Done Sent Done false
1754 1544 Acked Flush 0 Done Sent Done false
1755 1545 Acked Read 7 Done Sent Done false
1756 1546 Acked Write 6 Done Sent Done false
1757 1547 Acked Write 9 Done Sent Done false
1758 1548 Acked Read 4 Done Sent Done false
1759 1549 Acked Read 10 Done Sent Done false
1760 1550 Acked Flush 0 Done Sent Done false
1761 1551 Acked Read 10 Done Sent Done false
1762 1552 Acked Read 4 Done Sent Done false
1763 1553 Acked Flush 0 Done Sent Done false
1764 1554 Acked Read 7 Done Sent Done false
1765 1555 Acked Flush 0 Done Sent Done false
1766 1556 Acked Write 1 Done Sent Done false
1767 1557 Acked Flush 0 Done Sent Done false
1768 1558 Acked Write 8 Done Sent Done false
1769 1559 Acked Flush 0 Done Sent Done false
1770 1560 Acked Write 6 Done Sent Done false
1771 1561 Acked Write 2 Done Sent Done false
1772 1562 Acked Read 6 Done Sent Done false
1773 1563 Acked Flush 0 Done Sent Done false
1774 1564 Acked Write 4 Done Sent Done false
1775 1565 Acked Read 10 Done Sent Done false
1776 1566 Acked Read 6 Done Sent Done false
1777 1567 Acked Read 3 Done Sent Done false
1778 1568 Acked Read 6 Done Sent Done false
1779 1569 Acked Flush 0 Done Sent Done false
1780 1570 Acked Read 1 Done Sent Done false
1781 1571 Acked Flush 0 Done Sent Done false
1782 1572 Acked Write 8 Done Sent Done false
1783 1573 Acked Write 10 Done Sent Done false
1784 1574 Acked Read 10 Done Sent Done false
1785 1575 Acked Read 3 Done Sent Done false
1786 1576 Acked Write 6 Done Sent Done false
1787 1577 Acked Write 5 Done Sent Done false
1788 1578 Acked Write 9 Done Sent Done false
1789 1579 Acked Read 7 Done Sent Done false
1790 1580 Acked Write 2 Done Sent Done false
1791 1581 Acked Read 9 Done Sent Done false
1792 1582 Acked Read 4 Done Sent Done false
1793 1583 Acked Read 5 Done Sent Done false
1794 1584 Acked Write 10 Done Sent Done false
1795 1585 Acked Read 4 Done Sent Done false
1796 1586 Acked Read 3 Done Sent Done false
1797 1587 Acked Flush 0 Done Sent Done false
1798 1588 Acked Read 10 Done Sent Done false
1799 1589 Acked Write 9 Done Sent Done false
1800 1590 Acked Read 1 Done Sent Done false
1801 1591 Acked Write 5 Done Sent Done false
1802 1592 Acked Read 3 Done Sent Done false
1803 1593 Acked Flush 0 Done Sent Done false
1804 1594 Acked Read 9 Done Sent Done false
1805 1595 Acked Read 7 Done Sent Done false
1806 1596 Acked Read 9 Done Sent Done false
1807 1597 Acked Write 3 Done Sent Done false
1808 1598 Acked Read 9 Done Sent Done false
1809 1599 Acked Write 4 Done Sent Done false
1810 1600 Acked Write 3 Done Sent Done false
1811 1601 Acked Read 2 Done Sent Done false
1812 1602 Acked Write 5 Done Sent Done false
1813 STATES DS:0 DS:1 DS:2 TOTAL
1814 Sent 0 301 0 301
1815 Done 301 0 301 602
1816 Skipped 0 0 0 0
1817 Error 0 0 0 0
1818 Last Flush: Some(JobId(1593)) Some(JobId(1301)) Some(JobId(1593))
1819 Downstairs last five completed: 1301 1300 1299 1298 1297
1820 Upstairs last five completed: 1602 1601 1600 1599 1598
1821 CLIENT: Up:0 ds:301 act:2
18222025-03-17T22:16:36.622ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1823 [8820] got start action
1824 [8820] I am going to restart
1825 Starting downstairs at port 8820
1826 Make output file at "/tmp/dsc/downstairs-8820.txt"
1827 [8820][1] reports Starting
1828 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1208)
1829 [8820][1] reports Running
18302025-03-17T22:16:37.623ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
18312025-03-17T22:16:38.625ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
18322025-03-17T22:16:39.638ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
1833 ----------------------------------------------------------------
1834 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1835 States: Active Offline Active
1836 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
1837 1302 Acked Read 2 Done Sent Done false
1838 1303 Acked Read 8 Done Sent Done false
1839 1304 Acked Read 2 Done Sent Done false
1840 1305 Acked Write 1 Done Sent Done false
1841 1306 Acked Write 10 Done Sent Done false
1842 1307 Acked Read 6 Done Sent Done false
1843 1308 Acked Write 8 Done Sent Done false
1844 1309 Acked Write 5 Done Sent Done false
1845 1310 Acked Write 7 Done Sent Done false
1846 1311 Acked Read 7 Done Sent Done false
1847 1312 Acked Flush 0 Done Sent Done false
1848 1313 Acked Write 6 Done Sent Done false
1849 1314 Acked Read 1 Done Sent Done false
1850 1315 Acked Write 5 Done Sent Done false
1851 1316 Acked Read 3 Done Sent Done false
1852 1317 Acked Write 7 Done Sent Done false
1853 1318 Acked Write 8 Done Sent Done false
1854 1319 Acked Write 2 Done Sent Done false
1855 1320 Acked Read 5 Done Sent Done false
1856 1321 Acked Write 2 Done Sent Done false
1857 1322 Acked Read 9 Done Sent Done false
1858 1323 Acked Write 6 Done Sent Done false
1859 1324 Acked Read 9 Done Sent Done false
1860 1325 Acked Write 7 Done Sent Done false
1861 1326 Acked Read 1 Done Sent Done false
1862 1327 Acked Read 3 Done Sent Done false
1863 1328 Acked Flush 0 Done Sent Done false
1864 1329 Acked Write 4 Done Sent Done false
1865 1330 Acked Write 5 Done Sent Done false
1866 1331 Acked Write 2 Done Sent Done false
1867 1332 Acked Write 7 Done Sent Done false
1868 1333 Acked Write 10 Done Sent Done false
1869 1334 Acked Read 2 Done Sent Done false
1870 1335 Acked Flush 0 Done Sent Done false
1871 1336 Acked Write 5 Done Sent Done false
1872 1337 Acked Read 9 Done Sent Done false
1873 1338 Acked Read 9 Done Sent Done false
1874 1339 Acked Read 9 Done Sent Done false
1875 1340 Acked Write 1 Done Sent Done false
1876 1341 Acked Write 8 Done Sent Done false
1877 1342 Acked Read 3 Done Sent Done false
1878 1343 Acked Write 1 Done Sent Done false
1879 1344 Acked Write 6 Done Sent Done false
1880 1345 Acked Read 2 Done Sent Done false
1881 1346 Acked Write 3 Done Sent Done false
1882 1347 Acked Read 1 Done Sent Done false
1883 1348 Acked Read 10 Done Sent Done false
1884 1349 Acked Read 8 Done Sent Done false
1885 1350 Acked Write 7 Done Sent Done false
1886 1351 Acked Read 6 Done Sent Done false
1887 1352 Acked Flush 0 Done Sent Done false
1888 1353 Acked Write 1 Done Sent Done false
1889 1354 Acked Read 4 Done Sent Done false
1890 1355 Acked Write 9 Done Sent Done false
1891 1356 Acked Write 3 Done Sent Done false
1892 1357 Acked Read 6 Done Sent Done false
1893 1358 Acked Read 8 Done Sent Done false
1894 1359 Acked Read 4 Done Sent Done false
1895 1360 Acked Read 4 Done Sent Done false
1896 1361 Acked Read 6 Done Sent Done false
1897 1362 Acked Write 10 Done Sent Done false
1898 1363 Acked Read 10 Done Sent Done false
1899 1364 Acked Write 6 Done Sent Done false
1900 1365 Acked Write 10 Done Sent Done false
1901 1366 Acked Read 4 Done Sent Done false
1902 1367 Acked Write 4 Done Sent Done false
1903 1368 Acked Read 6 Done Sent Done false
1904 1369 Acked Read 9 Done Sent Done false
1905 1370 Acked Write 5 Done Sent Done false
1906 1371 Acked Read 10 Done Sent Done false
1907 1372 Acked Read 1 Done Sent Done false
1908 1373 Acked Read 10 Done Sent Done false
1909 1374 Acked Write 1 Done Sent Done false
1910 1375 Acked Flush 0 Done Sent Done false
1911 1376 Acked Flush 0 Done Sent Done false
1912 1377 Acked Flush 0 Done Sent Done false
1913 1378 Acked Read 5 Done Sent Done false
1914 1379 Acked Read 1 Done Sent Done false
1915 1380 Acked Read 2 Done Sent Done false
1916 1381 Acked Write 9 Done Sent Done false
1917 1382 Acked Write 10 Done Sent Done false
1918 1383 Acked Write 2 Done Sent Done false
1919 1384 Acked Write 1 Done Sent Done false
1920 1385 Acked Write 4 Done Sent Done false
1921 1386 Acked Write 2 Done Sent Done false
1922 1387 Acked Write 2 Done Sent Done false
1923 1388 Acked Read 9 Done Sent Done false
1924 1389 Acked Write 2 Done Sent Done false
1925 1390 Acked Read 1 Done Sent Done false
1926 1391 Acked Read 9 Done Sent Done false
1927 1392 Acked Read 10 Done Sent Done false
1928 1393 Acked Read 4 Done Sent Done false
1929 1394 Acked Flush 0 Done Sent Done false
1930 1395 Acked Read 4 Done Sent Done false
1931 1396 Acked Flush 0 Done Sent Done false
1932 1397 Acked Read 9 Done Sent Done false
1933 1398 Acked Write 7 Done Sent Done false
1934 1399 Acked Write 3 Done Sent Done false
1935 1400 Acked Write 7 Done Sent Done false
1936 1401 Acked Write 5 Done Sent Done false
1937 1402 Acked Write 9 Done Sent Done false
1938 1403 Acked Write 1 Done Sent Done false
1939 1404 Acked Read 10 Done Sent Done false
1940 1405 Acked Flush 0 Done Sent Done false
1941 1406 Acked Write 10 Done Sent Done false
1942 1407 Acked Write 3 Done Sent Done false
1943 1408 Acked Write 3 Done Sent Done false
1944 1409 Acked Read 7 Done Sent Done false
1945 1410 Acked Write 10 Done Sent Done false
1946 1411 Acked Write 3 Done Sent Done false
1947 1412 Acked Flush 0 Done Sent Done false
1948 1413 Acked Read 8 Done Sent Done false
1949 1414 Acked Read 1 Done Sent Done false
1950 1415 Acked Read 3 Done Sent Done false
1951 1416 Acked Flush 0 Done Sent Done false
1952 1417 Acked Read 7 Done Sent Done false
1953 1418 Acked Write 5 Done Sent Done false
1954 1419 Acked Read 8 Done Sent Done false
1955 1420 Acked Flush 0 Done Sent Done false
1956 1421 Acked Flush 0 Done Sent Done false
1957 1422 Acked Flush 0 Done Sent Done false
1958 1423 Acked Read 1 Done Sent Done false
1959 1424 Acked Read 2 Done Sent Done false
1960 1425 Acked Write 4 Done Sent Done false
1961 1426 Acked Read 1 Done Sent Done false
1962 1427 Acked Write 10 Done Sent Done false
1963 1428 Acked Read 2 Done Sent Done false
1964 1429 Acked Read 2 Done Sent Done false
1965 1430 Acked Write 10 Done Sent Done false
1966 1431 Acked Read 10 Done Sent Done false
1967 1432 Acked Read 3 Done Sent Done false
1968 1433 Acked Write 10 Done Sent Done false
1969 1434 Acked Read 6 Done Sent Done false
1970 1435 Acked Write 8 Done Sent Done false
1971 1436 Acked Read 8 Done Sent Done false
1972 1437 Acked Read 5 Done Sent Done false
1973 1438 Acked Read 7 Done Sent Done false
1974 1439 Acked Read 7 Done Sent Done false
1975 1440 Acked Write 4 Done Sent Done false
1976 1441 Acked Read 7 Done Sent Done false
1977 1442 Acked Write 2 Done Sent Done false
1978 1443 Acked Read 5 Done Sent Done false
1979 1444 Acked Flush 0 Done Sent Done false
1980 1445 Acked Read 10 Done Sent Done false
1981 1446 Acked Read 6 Done Sent Done false
1982 1447 Acked Write 5 Done Sent Done false
1983 1448 Acked Read 1 Done Sent Done false
1984 1449 Acked Read 1 Done Sent Done false
1985 1450 Acked Read 4 Done Sent Done false
1986 1451 Acked Write 7 Done Sent Done false
1987 1452 Acked Write 8 Done Sent Done false
1988 1453 Acked Write 5 Done Sent Done false
1989 1454 Acked Read 10 Done Sent Done false
1990 1455 Acked Write 2 Done Sent Done false
1991 1456 Acked Write 4 Done Sent Done false
1992 1457 Acked Read 10 Done Sent Done false
1993 1458 Acked Read 7 Done Sent Done false
1994 1459 Acked Write 7 Done Sent Done false
1995 1460 Acked Read 6 Done Sent Done false
1996 1461 Acked Write 2 Done Sent Done false
1997 1462 Acked Read 6 Done Sent Done false
1998 1463 Acked Write 7 Done Sent Done false
1999 1464 Acked Write 7 Done Sent Done false
2000 1465 Acked Write 1 Done Sent Done false
2001 1466 Acked Read 10 Done Sent Done false
2002 1467 Acked Read 6 Done Sent Done false
2003 1468 Acked Read 9 Done Sent Done false
2004 1469 Acked Write 6 Done Sent Done false
2005 1470 Acked Write 1 Done Sent Done false
2006 1471 Acked Write 9 Done Sent Done false
2007 1472 Acked Flush 0 Done Sent Done false
2008 1473 Acked Read 2 Done Sent Done false
2009 1474 Acked Read 7 Done Sent Done false
2010 1475 Acked Read 7 Done Sent Done false
2011 1476 Acked Read 6 Done Sent Done false
2012 1477 Acked Read 9 Done Sent Done false
2013 1478 Acked Write 3 Done Sent Done false
2014 1479 Acked Write 10 Done Sent Done false
2015 1480 Acked Write 2 Done Sent Done false
2016 1481 Acked Write 1 Done Sent Done false
2017 1482 Acked Read 10 Done Sent Done false
2018 1483 Acked Write 4 Done Sent Done false
2019 1484 Acked Write 4 Done Sent Done false
2020 1485 Acked Write 4 Done Sent Done false
2021 1486 Acked Read 2 Done Sent Done false
2022 1487 Acked Flush 0 Done Sent Done false
2023 1488 Acked Write 5 Done Sent Done false
2024 1489 Acked Read 5 Done Sent Done false
2025 1490 Acked Read 8 Done Sent Done false
2026 1491 Acked Read 4 Done Sent Done false
2027 1492 Acked Write 2 Done Sent Done false
2028 1493 Acked Write 2 Done Sent Done false
2029 1494 Acked Read 6 Done Sent Done false
2030 1495 Acked Read 7 Done Sent Done false
2031 1496 Acked Read 1 Done Sent Done false
2032 1497 Acked Read 2 Done Sent Done false
2033 1498 Acked Read 4 Done Sent Done false
2034 1499 Acked Write 10 Done Sent Done false
2035 1500 Acked Read 9 Done Sent Done false
2036 1501 Acked Read 4 Done Sent Done false
2037 1502 Acked Write 6 Done Sent Done false
2038 1503 Acked Read 10 Done Sent Done false
2039 1504 Acked Write 8 Done Sent Done false
2040 1505 Acked Write 3 Done Sent Done false
2041 1506 Acked Write 9 Done Sent Done false
2042 1507 Acked Write 5 Done Sent Done false
2043 1508 Acked Read 2 Done Sent Done false
2044 1509 Acked Write 2 Done Sent Done false
2045 1510 Acked Read 2 Done Sent Done false
2046 1511 Acked Write 8 Done Sent Done false
2047 1512 Acked Read 5 Done Sent Done false
2048 1513 Acked Read 3 Done Sent Done false
2049 1514 Acked Write 1 Done Sent Done false
2050 1515 Acked Read 6 Done Sent Done false
2051 1516 Acked Read 2 Done Sent Done false
2052 1517 Acked Read 8 Done Sent Done false
2053 1518 Acked Flush 0 Done Sent Done false
2054 1519 Acked Write 8 Done Sent Done false
2055 1520 Acked Read 10 Done Sent Done false
2056 1521 Acked Read 7 Done Sent Done false
2057 1522 Acked Write 6 Done Sent Done false
2058 1523 Acked Write 5 Done Sent Done false
2059 1524 Acked Write 10 Done Sent Done false
2060 1525 Acked Write 7 Done Sent Done false
2061 1526 Acked Flush 0 Done Sent Done false
2062 1527 Acked Read 10 Done Sent Done false
2063 1528 Acked Read 4 Done Sent Done false
2064 1529 Acked Write 3 Done Sent Done false
2065 1530 Acked Write 9 Done Sent Done false
2066 1531 Acked Write 4 Done Sent Done false
2067 1532 Acked Flush 0 Done Sent Done false
2068 1533 Acked Flush 0 Done Sent Done false
2069 1534 Acked Write 9 Done Sent Done false
2070 1535 Acked Read 1 Done Sent Done false
2071 1536 Acked Read 9 Done Sent Done false
2072 1537 Acked Read 8 Done Sent Done false
2073 1538 Acked Read 5 Done Sent Done false
2074 1539 Acked Write 1 Done Sent Done false
2075 1540 Acked Read 6 Done Sent Done false
2076 1541 Acked Write 3 Done Sent Done false
2077 1542 Acked Read 6 Done Sent Done false
2078 1543 Acked Read 10 Done Sent Done false
2079 1544 Acked Flush 0 Done Sent Done false
2080 1545 Acked Read 7 Done Sent Done false
2081 1546 Acked Write 6 Done Sent Done false
2082 1547 Acked Write 9 Done Sent Done false
2083 1548 Acked Read 4 Done Sent Done false
2084 1549 Acked Read 10 Done Sent Done false
2085 1550 Acked Flush 0 Done Sent Done false
2086 1551 Acked Read 10 Done Sent Done false
2087 1552 Acked Read 4 Done Sent Done false
2088 1553 Acked Flush 0 Done Sent Done false
2089 1554 Acked Read 7 Done Sent Done false
2090 1555 Acked Flush 0 Done Sent Done false
2091 1556 Acked Write 1 Done Sent Done false
2092 1557 Acked Flush 0 Done Sent Done false
2093 1558 Acked Write 8 Done Sent Done false
2094 1559 Acked Flush 0 Done Sent Done false
2095 1560 Acked Write 6 Done Sent Done false
2096 1561 Acked Write 2 Done Sent Done false
2097 1562 Acked Read 6 Done Sent Done false
2098 1563 Acked Flush 0 Done Sent Done false
2099 1564 Acked Write 4 Done Sent Done false
2100 1565 Acked Read 10 Done Sent Done false
2101 1566 Acked Read 6 Done Sent Done false
2102 1567 Acked Read 3 Done Sent Done false
2103 1568 Acked Read 6 Done Sent Done false
2104 1569 Acked Flush 0 Done Sent Done false
2105 1570 Acked Read 1 Done Sent Done false
2106 1571 Acked Flush 0 Done Sent Done false
2107 1572 Acked Write 8 Done Sent Done false
2108 1573 Acked Write 10 Done Sent Done false
2109 1574 Acked Read 10 Done Sent Done false
2110 1575 Acked Read 3 Done Sent Done false
2111 1576 Acked Write 6 Done Sent Done false
2112 1577 Acked Write 5 Done Sent Done false
2113 1578 Acked Write 9 Done Sent Done false
2114 1579 Acked Read 7 Done Sent Done false
2115 1580 Acked Write 2 Done Sent Done false
2116 1581 Acked Read 9 Done Sent Done false
2117 1582 Acked Read 4 Done Sent Done false
2118 1583 Acked Read 5 Done Sent Done false
2119 1584 Acked Write 10 Done Sent Done false
2120 1585 Acked Read 4 Done Sent Done false
2121 1586 Acked Read 3 Done Sent Done false
2122 1587 Acked Flush 0 Done Sent Done false
2123 1588 Acked Read 10 Done Sent Done false
2124 1589 Acked Write 9 Done Sent Done false
2125 1590 Acked Read 1 Done Sent Done false
2126 1591 Acked Write 5 Done Sent Done false
2127 1592 Acked Read 3 Done Sent Done false
2128 1593 Acked Flush 0 Done Sent Done false
2129 1594 Acked Read 9 Done Sent Done false
2130 1595 Acked Read 7 Done Sent Done false
2131 1596 Acked Read 9 Done Sent Done false
2132 1597 Acked Write 3 Done Sent Done false
2133 1598 Acked Read 9 Done Sent Done false
2134 1599 Acked Write 4 Done Sent Done false
2135 1600 Acked Write 3 Done Sent Done false
2136 1601 Acked Read 2 Done Sent Done false
2137 1602 Acked Write 5 Done Sent Done false
2138 STATES DS:0 DS:1 DS:2 TOTAL
2139 Sent 0 301 0 301
2140 Done 301 0 301 602
2141 Skipped 0 0 0 0
2142 Error 0 0 0 0
2143 Last Flush: Some(JobId(1593)) Some(JobId(1301)) Some(JobId(1593))
2144 Downstairs last five completed: 1301 1300 1299 1298 1297
2145 Upstairs last five completed: 1602 1601 1600 1599 1598
2146 CLIENT: Up:0 ds:301 act:2
21472025-03-17T22:16:40.628ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
21482025-03-17T22:16:41.402ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
21492025-03-17T22:16:41.403ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:62584) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
21502025-03-17T22:16:41.404ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
21512025-03-17T22:16:41.404ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
21522025-03-17T22:16:41.404ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
21532025-03-17T22:16:41.404ZINFOcrucible: send last flush ID to this DS: Some(JobId(1301)) = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
21542025-03-17T22:16:41.404ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
21552025-03-17T22:16:41.404ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(1301)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
21562025-03-17T22:16:41.405ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(1301)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
2157 ----------------------------------------------------------------
2158 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
2159 Downstairs last five completed: 1603 1602 1601 1600 1599
2160 Upstairs last five completed: 1603 1602 1601 1600 1599
2161 CLIENT: Up:0 ds:0 act:3
2162 Replay: All jobs finished
2163 Main task has work to do, go find it
2164 got dsc Stop(1)
2165 stop 1
2166 Mar 17 22:16:43.653 INFO request completed, latency_us: 357, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 3ee95b4e-1c45-4725-b649-53ef8f588b79, remote_addr: 127.0.0.1:60423[8820] Got stop action so:false kr:false
2167 , local_addr: 127.0.0.1:9998
21682025-03-17T22:16:43.654ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
21692025-03-17T22:16:43.654ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
2170 Mar 17 22:16:43.654 INFO request completed, latency_us: 161{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:16:43.654776906Z","hostname":"w-01JPK2S810SFAA9787XEG8GFX5","pid":1206,"session_id":"0f8310d7-099e-4ead-8484-91ddc162d307"}
2171 , response_code: 200, uri: /state/cid/1, method: GET{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:16:43.654802554Z","hostname":"w-01JPK2S810SFAA9787XEG8GFX5","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"0f8310d7-099e-4ead-8484-91ddc162d307"}
2172 , req_id: 536e677a-9446-4508-9bef-37b420f0a907, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
2173 [8820] Exited with: ExitStatus(unix_wait_status(9))
2174 [8820][1] reports Exit
2175 Mar 17 22:16:47.657 INFO request completed, latency_us: 240, response_code: 200, uri: /state/cid/1, method: GET, req_id: d799c792-2343-4fe3-92a9-7bee99e36bdc, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
2176 001/300 Read block 2027 len 12288
2177 002/300 Flush
2178 003/300 Read block 4987 len 32768
2179 004/300 Write block 4107 len 16384 data: 3 3 2 2
2180 005/300 Read block 1659 len 8192
2181 006/300 Flush
2182 007/300 Read block 4643 len 8192
2183 008/300 Read block 3835 len 40960
2184 009/300 Read block 1578 len 16384
2185 010/300 Write block 233 len 36864 data: 3 3 3 3 3 2 2 2 2
2186 011/300 Read block 3659 len 28672
2187 012/300 Write block 2967 len 8192 data: 2 2
2188 013/300 Read block 1619 len 12288
2189 014/300 Read block 3728 len 8192
2190 015/300 Read block 3804 len 24576
2191 016/300 Read block 2945 len 40960
2192 017/300 Write block 4779 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2193 018/300 Write block 4562 len 4096 data: 2
2194 019/300 Write block 1533 len 24576 data: 3 3 3 3 2 2
2195 020/300 Flush
2196 021/300 Read block 3494 len 16384
2197 022/300 Read block 1783 len 36864
2198 023/300 Read block 4132 len 12288
2199 024/300 Read block 1104 len 8192
2200 025/300 Read block 687 len 28672
2201 026/300 Write block 2826 len 8192 data: 2 2
2202 027/300 Read block 344 len 36864
2203 028/300 Write block 3036 len 24576 data: 2 2 2 2 2 2
2204 029/300 Read block 1543 len 28672
2205 030/300 Read block 2924 len 12288
2206 031/300 Read block 2153 len 32768
2207 032/300 Read block 2617 len 8192
2208 033/300 Write block 1680 len 24576 data: 2 2 2 2 2 2
2209 034/300 Read block 401 len 40960
2210 035/300 Write block 4468 len 28672 data: 2 2 3 3 3 3 3
2211 036/300 Read block 3583 len 24576
2212 037/300 Read block 1367 len 24576
2213 038/300 Read block 1257 len 28672
2214 039/300 Write block 1049 len 12288 data: 2 2 2
2215 040/300 Write block 3723 len 32768 data: 2 2 2 2 3 3 3 4
2216 041/300 Write block 2886 len 40960 data: 4 5 4 4 3 2 2 2 2 2
2217 042/300 Flush
2218 043/300 Write block 1670 len 4096 data: 3
2219 044/300 Write block 3806 len 36864 data: 2 2 2 2 2 2 2 2 2
2220 045/300 Flush
2221 046/300 Read block 2495 len 40960
2222 047/300 Write block 3710 len 32768 data: 2 2 2 2 2 2 2 2
2223 048/300 Write block 556 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2224 049/300 Read block 334 len 24576
2225 050/300 Write block 4888 len 24576 data: 3 2 2 2 2 2
2226 051/300 Write block 644 len 28672 data: 2 2 2 2 2 2 2
2227 052/300 Write block 1759 len 12288 data: 2 2 2
2228 053/300 Write block 135 len 8192 data: 2 2
2229 054/300 Read block 1528 len 40960
2230 055/300 Read block 2441 len 12288
2231 056/300 Write block 76 len 12288 data: 3 3 3
2232 057/300 Write block 1597 len 36864 data: 3 3 3 2 2 2 2 2 2
2233 058/300 Read block 4930 len 24576
2234 059/300 Read block 4567 len 20480
2235 060/300 Write block 3642 len 36864 data: 2 3 3 2 2 2 2 2 2
2236 061/300 Read block 2622 len 12288
2237 062/300 Read block 2649 len 12288
2238 063/300 Read block 973 len 16384
2239 064/300 Write block 2358 len 8192 data: 3 3
2240 065/300 Flush
2241 066/300 Read block 2645 len 24576
2242 067/300 Flush
2243 068/300 Read block 1828 len 24576
2244 069/300 Read block 1420 len 20480
2245 070/300 Read block 4 len 32768
2246 071/300 Flush
2247 072/300 Write block 2879 len 12288 data: 2 2 2
2248 073/300 Write block 2177 len 8192 data: 2 2
2249 074/300 Read block 2929 len 12288
2250 075/300 Write block 4689 len 32768 data: 3 3 3 3 3 2 2 2
2251 076/300 Write block 4433 len 36864 data: 2 2 2 2 2 2 2 2 2
2252 077/300 Write block 304 len 24576 data: 2 2 2 2 2 2
2253 078/300 Write block 3 len 24576 data: 2 2 2 2 2 2
2254 079/300 Read block 558 len 32768
2255 080/300 Read block 4057 len 4096
2256 081/300 Read block 2568 len 16384
2257 082/300 Write block 4208 len 28672 data: 2 2 2 3 2 2 2
2258 083/300 Write block 4860 len 36864 data: 2 3 4 4 3 3 3 2 2
2259 084/300 Flush
2260 085/300 Write block 3492 len 24576 data: 2 2 2 3 2 2
2261 086/300 Write block 4756 len 40960 data: 2 3 3 3 3 3 3 4 4 2
2262 087/300 Write block 3073 len 4096 data: 2
2263 088/300 Flush
2264 089/300 Write block 4604 len 24576 data: 3 3 4 4 4 3
2265 090/300 Read block 3631 len 8192
2266 091/300 Read block 4241 len 40960
2267 092/300 Write block 2278 len 24576 data: 3 3 3 3 3 3
2268 093/300 Read block 4540 len 40960
2269 094/300 Write block 3760 len 12288 data: 2 2 2
2270 095/300 Write block 3456 len 24576 data: 2 2 2 2 2 2
2271 096/300 Write block 3004 len 8192 data: 2 2
2272 097/300 Write block 1085 len 8192 data: 2 2
2273 098/300 Read block 284 len 12288
2274 099/300 Write block 1433 len 8192 data: 2 2
2275 100/300 Read block 3313 len 24576
2276 101/300 Write block 2214 len 28672 data: 2 3 3 3 3 3 3
2277 102/300 Read block 1161 len 36864
2278 103/300 Read block 559 len 32768
2279 104/300 Write block 1599 len 4096 data: 4
2280 105/300 Read block 3673 len 40960
2281 106/300 Read block 1051 len 12288
2282 107/300 Read block 4458 len 8192
2283 108/300 Write block 359 len 24576 data: 2 2 2 2 2 2
2284 109/300 Write block 1761 len 16384 data: 3 2 3 3
2285 110/300 Read block 4321 len 8192
2286 111/300 Read block 2806 len 24576
2287 112/300 Read block 4696 len 4096
2288 113/300 Write block 4859 len 12288 data: 2 3 4
2289 114/300 Read block 1037 len 28672
2290 115/300 Write block 679 len 36864 data: 2 2 3 3 3 3 2 2 2
2291 116/300 Read block 838 len 12288
2292 117/300 Write block 3828 len 32768 data: 2 2 2 2 2 2 2 2
2293 118/300 Read block 3650 len 16384
2294 119/300 Write block 2309 len 20480 data: 2 2 2 2 2
2295 120/300 Read block 273 len 28672
2296 121/300 Read block 1232 len 36864
2297 122/300 Write block 3793 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2298 123/300 Write block 2965 len 20480 data: 2 2 3 3 2
2299 124/300 Flush
2300 125/300 Write block 1524 len 16384 data: 3 3 3 4
2301 126/300 Read block 3287 len 24576
2302 127/300 Read block 779 len 28672
2303 128/300 Write block 2871 len 40960 data: 3 3 3 3 3 4 2 2 3 3
2304 129/300 Write block 4807 len 28672 data: 3 3 3 3 2 2 2
2305 130/300 Read block 1105 len 36864
2306 131/300 Write block 4573 len 36864 data: 2 2 2 2 2 2 2 2 2
2307 132/300 Write block 3879 len 36864 data: 2 3 3 3 3 2 2 2 2
2308 133/300 Read block 1357 len 4096
2309 134/300 Read block 1803 len 32768
2310 135/300 Flush
2311 136/300 Write block 3730 len 20480 data: 5 4 3 3 3
2312 137/300 Write block 2672 len 20480 data: 2 2 2 2 2
2313 138/300 Read block 203 len 28672
2314 139/300 Read block 1314 len 40960
2315 140/300 Read block 481 len 8192
2316 141/300 Read block 142 len 8192
2317 142/300 Write block 1663 len 28672 data: 2 2 2 2 2 2 3
2318 143/300 Read block 2007 len 4096
2319 144/300 Read block 3704 len 12288
2320 145/300 Read block 3458 len 20480
2321 146/300 Write block 4304 len 8192 data: 4 3
2322 147/300 Read block 2750 len 12288
2323 148/300 Read block 148 len 24576
2324 149/300 Read block 1687 len 8192
2325 150/300 Write block 3287 len 28672 data: 2 2 2 2 2 2 2
2326 151/300 Flush
2327 152/300 Write block 557 len 40960 data: 3 3 3 3 3 3 3 3 3 2
2328 153/300 Read block 717 len 8192
2329 154/300 Read block 619 len 24576
2330 155/300 Read block 1390 len 24576
2331 156/300 Flush
2332 157/300 Write block 1824 len 4096 data: 2
2333 158/300 Read block 836 len 24576
2334 159/300 Read block 880 len 40960
2335 160/300 Read block 2518 len 16384
2336 161/300 Write block 3139 len 8192 data: 3 3
2337 162/300 Read block 1473 len 12288
2338 163/300 Read block 2550 len 12288
2339 164/300 Write block 126 len 24576 data: 2 2 2 2 2 2
2340 165/300 Read block 3483 len 40960
2341 166/300 Read block 2455 len 32768
2342 167/300 Write block 3769 len 32768 data: 2 2 2 2 2 2 2 2
2343 168/300 Write block 1858 len 16384 data: 2 2 2 2
2344 169/300 Write block 4449 len 16384 data: 2 2 2 2
2345 170/300 Write block 124 len 36864 data: 2 2 3 3 3 3 3 3 2
2346 171/300 Flush
2347 172/300 Read block 3214 len 4096
2348 173/300 Flush
2349 174/300 Read block 2308 len 40960
2350 175/300 Read block 780 len 8192
2351 176/300 Read block 4886 len 8192
2352 177/300 Write block 1698 len 40960 data: 3 3 3 3 2 2 2 2 2 2
2353 178/300 Read block 1336 len 8192
2354 179/300 Read block 4226 len 8192
2355 180/300 Read block 1484 len 40960
2356 181/300 Read block 4168 len 16384
2357 182/300 Read block 1304 len 36864
2358 183/300 Write block 1033 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2359 184/300 Write block 2606 len 28672 data: 3 2 2 2 2 2 2
2360 185/300 Read block 2855 len 40960
2361 186/300 Read block 1448 len 24576
2362 187/300 Write block 988 len 32768 data: 3 3 3 3 3 3 3 2
2363 188/300 Read block 331 len 40960
2364 189/300 Write block 4047 len 8192 data: 2 2
2365 190/300 Flush
2366 191/300 Read block 1695 len 16384
2367 192/300 Write block 2217 len 32768 data: 4 4 4 4 3 3 2 2
2368 193/300 Read block 2462 len 24576
2369 194/300 Write block 4258 len 16384 data: 2 2 2 2
2370 195/300 Read block 4067 len 28672
2371 196/300 Write block 849 len 12288 data: 2 2 2
2372 197/300 Write block 2874 len 20480 data: 4 4 5 3 3
2373 198/300 Read block 1441 len 24576
2374 199/300 Read block 969 len 4096
2375 200/300 Read block 1974 len 12288
2376 201/300 Read block 1755 len 16384
2377 202/300 Read block 3764 len 32768
2378 203/300 Read block 2431 len 4096
2379 204/300 Read block 620 len 40960
2380 205/300 Write block 2768 len 32768 data: 3 3 3 3 2 2 2 2
2381 206/300 Read block 2572 len 8192
2382 207/300 Write block 2993 len 16384 data: 2 2 2 3
2383 208/300 Flush
2384 209/300 Read block 4346 len 32768
2385 210/300 Write block 3118 len 8192 data: 2 2
2386 211/300 Read block 4758 len 32768
2387 212/300 Read block 2438 len 4096
2388 213/300 Read block 454 len 8192
2389 214/300 Read block 1340 len 8192
2390 215/300 Flush
2391 216/300 Write block 2666 len 12288 data: 2 2 2
2392 217/300 Read block 2606 len 12288
2393 218/300 Write block 887 len 28672 data: 2 2 2 3 3 3 3
2394 219/300 Write block 1340 len 20480 data: 2 2 2 2 2
2395 220/300 Write block 1210 len 12288 data: 2 2 2
2396 221/300 Read block 579 len 20480
2397 222/300 Read block 4638 len 32768
2398 223/300 Flush
2399 224/300 Read block 4920 len 24576
2400 225/300 Read block 3207 len 4096
2401 226/300 Flush
2402 227/300 Read block 1337 len 12288
2403 228/300 Write block 3250 len 8192 data: 2 2
2404 229/300 Flush
2405 230/300 Write block 1477 len 36864 data: 2 2 2 2 2 2 2 2 2
2406 231/300 Read block 2172 len 16384
2407 232/300 Write block 1747 len 32768 data: 2 2 2 2 2 2 2 2
2408 233/300 Read block 2114 len 40960
2409 234/300 Write block 3338 len 12288 data: 2 2 2
2410 235/300 Flush
2411 236/300 Read block 1306 len 16384
2412 237/300 Write block 2861 len 24576 data: 2 2 2 2 2 2
2413 238/300 Write block 396 len 24576 data: 2 2 2 2 2 2
2414 239/300 Read block 1951 len 28672
2415 240/300 Read block 3772 len 28672
2416 241/300 Write block 1857 len 16384 data: 2 3 3 3
2417 242/300 Write block 2825 len 28672 data: 3 3 3 2 2 2 2
2418 243/300 Read block 3678 len 16384
2419 244/300 Write block 2429 len 8192 data: 2 2
2420 245/300 Read block 2052 len 16384
2421 246/300 Write block 1956 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2422 247/300 Flush
2423 248/300 Write block 1393 len 20480 data: 2 2 2 2 2
2424 249/300 Read block 2287 len 12288
2425 250/300 Read block 1640 len 12288
2426 251/300 Write block 1567 len 16384 data: 2 2 2 2
2427 252/300 Read block 2965 len 40960
2428 253/300 Flush
2429 254/300 Write block 1525 len 40960 data: 4 4 5 3 3 3 3 3 4 4
2430 255/300 Read block 3303 len 36864
2431 256/300 Read block 1049 len 28672
2432 257/300 Write block 2870 len 16384 data: 3 4 4 4
2433 258/300 Read block 1891 len 36864
2434 259/300 Read block 306 len 36864
2435 260/300 Read block 3439 len 16384
2436 261/300 Read block 1086 len 24576
2437 262/300 Write block 429 len 4096 data: 2
2438 263/300 Write block 1996 len 40960 data: 3 3 3 3 3 4 4 3 2 2
2439 264/300 Flush
2440 265/300 Write block 3147 len 28672 data: 3 3 3 3 2 2 2
2441 266/300 Read block 4234 len 40960
2442 267/300 Read block 852 len 20480
2443 268/300 Write block 4299 len 16384 data: 3 3 4 4
2444 269/300 Write block 3353 len 36864 data: 4 4 4 4 2 2 2 2 2
2445 270/300 Read block 2395 len 40960
2446 271/300 Read block 1819 len 4096
2447 272/300 Write block 3820 len 8192 data: 2 2
2448 273/300 Read block 3121 len 12288
2449 274/300 Read block 300 len 16384
2450 275/300 Read block 4652 len 8192
2451 276/300 Read block 729 len 40960
2452 277/300 Flush
2453 278/300 Read block 3754 len 16384
2454 279/300 Read block 2606 len 4096
2455 280/300 Read block 3300 len 28672
2456 281/300 Write block 2946 len 12288 data: 2 2 2
2457 282/300 Read block 1636 len 36864
2458 283/300 Read block 1607 len 36864
2459 284/300 Write block 2838 len 40960 data: 2 2 2 2 2 2 3 3 3 2
2460 285/300 Read block 863 len 36864
2461 286/300 Read block 2055 len 36864
2462 287/300 Flush
2463 288/300 Read block 4568 len 24576
2464 289/300 Write block 2992 len 8192 data: 2 3
2465 290/300 Write block 1072 len 36864 data: 2 2 2 2 2 2 2 2 2
2466 291/300 Read block 2025 len 12288
2467 292/300 Read block 4854 len 40960
2468 293/300 Read block 3551 len 32768
2469 294/300 Read block 3318 len 20480
2470 295/300 Read block 702 len 36864
2471 296/300 Read block 4882 len 24576
2472 297/300 Write block 2486 len 24576 data: 2 2 2 2 2 2
2473 298/300 Read block 3966 len 8192
2474 299/300 Read block 1667 len 8192
2475 300/300 Read block 124 len 24576
2476 301/300 Read block 422 len 24576
2477 Main task has work to do, go find it
2478 got dsc Start(1)
2479 start 1
2480 Mar 17 22:16:47.910 INFO request completed, latency_us: 227, response_code: 204, uri: /start/cid/1, method: POST, req_id: 4fe00d6f-459a-4700-be2c-3e57dd3d2693, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
2481 Replay: started 1, returned:Ok(())
2482 ----------------------------------------------------------------
2483 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2484 States: Active Offline Active
2485 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2486 1604 Acked Read 3 Done Sent Done false
2487 1605 Acked Flush 0 Done Sent Done false
2488 1606 Acked Read 8 Done Sent Done false
2489 1607 Acked Write 4 Done Sent Done false
2490 1608 Acked Read 2 Done Sent Done false
2491 1609 Acked Flush 0 Done Sent Done false
2492 1610 Acked Read 2 Done Sent Done false
2493 1611 Acked Read 10 Done Sent Done false
2494 1612 Acked Read 4 Done Sent Done false
2495 1613 Acked Write 9 Done Sent Done false
2496 1614 Acked Read 7 Done Sent Done false
2497 1615 Acked Write 2 Done Sent Done false
2498 1616 Acked Read 3 Done Sent Done false
2499 1617 Acked Read 2 Done Sent Done false
2500 1618 Acked Read 6 Done Sent Done false
2501 1619 Acked Read 10 Done Sent Done false
2502 1620 Acked Write 10 Done Sent Done false
2503 1621 Acked Write 1 Done Sent Done false
2504 1622 Acked Write 6 Done Sent Done false
2505 1623 Acked Flush 0 Done Sent Done false
2506 1624 Acked Read 4 Done Sent Done false
2507 1625 Acked Read 9 Done Sent Done false
2508 1626 Acked Read 3 Done Sent Done false
2509 1627 Acked Read 2 Done Sent Done false
2510 1628 Acked Read 7 Done Sent Done false
2511 1629 Acked Write 2 Done Sent Done false
2512 1630 Acked Read 9 Done Sent Done false
2513 1631 Acked Write 6 Done Sent Done false
2514 1632 Acked Read 7 Done Sent Done false
2515 1633 Acked Read 3 Done Sent Done false
2516 1634 Acked Read 8 Done Sent Done false
2517 1635 Acked Read 2 Done Sent Done false
2518 1636 Acked Write 6 Done Sent Done false
2519 1637 Acked Read 10 Done Sent Done false
2520 1638 Acked Write 7 Done Sent Done false
2521 1639 Acked Read 6 Done Sent Done false
2522 1640 Acked Read 6 Done Sent Done false
2523 1641 Acked Read 7 Done Sent Done false
2524 1642 Acked Write 3 Done Sent Done false
2525 1643 Acked Write 8 Done Sent Done false
2526 1644 Acked Write 10 Done Sent Done false
2527 1645 Acked Flush 0 Done Sent Done false
2528 1646 Acked Write 1 Done Sent Done false
2529 1647 Acked Write 9 Done Sent Done false
2530 1648 Acked Flush 0 Done Sent Done false
2531 1649 Acked Read 10 Done Sent Done false
2532 1650 Acked Write 8 Done Sent Done false
2533 1651 Acked Write 10 Done Sent Done false
2534 1652 Acked Read 6 Done Sent Done false
2535 1653 Acked Write 6 Done Sent Done false
2536 1654 Acked Write 7 Done Sent Done false
2537 1655 Acked Write 3 Done Sent Done false
2538 1656 Acked Write 2 Done Sent Done false
2539 1657 Acked Read 10 Done Sent Done false
2540 1658 Acked Read 3 Done Sent Done false
2541 1659 Acked Write 3 Done Sent Done false
2542 1660 Acked Write 9 Done Sent Done false
2543 1661 Acked Read 6 Done Sent Done false
2544 1662 Acked Read 5 Done Sent Done false
2545 1663 Acked Write 9 Done Sent Done false
2546 1664 Acked Read 3 Done Sent Done false
2547 1665 Acked Read 3 Done Sent Done false
2548 1666 Acked Read 4 Done Sent Done false
2549 1667 Acked Write 2 Done Sent Done false
2550 1668 Acked Flush 0 Done Sent Done false
2551 1669 Acked Read 6 Done Sent Done false
2552 1670 Acked Flush 0 Done Sent Done false
2553 1671 Acked Read 6 Done Sent Done false
2554 1672 Acked Read 5 Done Sent Done false
2555 1673 Acked Read 8 Done Sent Done false
2556 1674 Acked Flush 0 Done Sent Done false
2557 1675 Acked Write 3 Done Sent Done false
2558 1676 Acked Write 2 Done Sent Done false
2559 1677 Acked Read 3 Done Sent Done false
2560 1678 Acked Write 8 Done Sent Done false
2561 1679 Acked Write 9 Done Sent Done false
2562 1680 Acked Write 6 Done Sent Done false
2563 1681 Acked Write 6 Done Sent Done false
2564 1682 Acked Read 8 Done Sent Done false
2565 1683 Acked Read 1 Done Sent Done false
2566 1684 Acked Read 4 Done Sent Done false
2567 1685 Acked Write 7 Done Sent Done false
2568 1686 Acked Write 9 Done Sent Done false
2569 1687 Acked Flush 0 Done Sent Done false
2570 1688 Acked Write 6 Done Sent Done false
2571 1689 Acked Write 10 Done Sent Done false
2572 1690 Acked Write 1 Done Sent Done false
2573 1691 Acked Flush 0 Done Sent Done false
2574 1692 Acked Write 6 Done Sent Done false
2575 1693 Acked Read 2 Done Sent Done false
2576 1694 Acked Read 10 Done Sent Done false
2577 1695 Acked Write 6 Done Sent Done false
2578 1696 Acked Read 10 Done Sent Done false
2579 1697 Acked Write 3 Done Sent Done false
2580 1698 Acked Write 6 Done Sent Done false
2581 1699 Acked Write 2 Done Sent Done false
2582 1700 Acked Write 2 Done Sent Done false
2583 1701 Acked Read 3 Done Sent Done false
2584 1702 Acked Write 2 Done Sent Done false
2585 1703 Acked Read 6 Done Sent Done false
2586 1704 Acked Write 7 Done Sent Done false
2587 1705 Acked Read 9 Done Sent Done false
2588 1706 Acked Read 8 Done Sent Done false
2589 1707 Acked Write 1 Done Sent Done false
2590 1708 Acked Read 10 Done Sent Done false
2591 1709 Acked Read 3 Done Sent Done false
2592 1710 Acked Read 2 Done Sent Done false
2593 1711 Acked Write 6 Done Sent Done false
2594 1712 Acked Write 4 Done Sent Done false
2595 1713 Acked Read 2 Done Sent Done false
2596 1714 Acked Read 6 Done Sent Done false
2597 1715 Acked Read 1 Done Sent Done false
2598 1716 Acked Write 3 Done Sent Done false
2599 1717 Acked Read 7 Done Sent Done false
2600 1718 Acked Write 9 Done Sent Done false
2601 1719 Acked Read 3 Done Sent Done false
2602 1720 Acked Write 8 Done Sent Done false
2603 1721 Acked Read 4 Done Sent Done false
2604 1722 Acked Write 5 Done Sent Done false
2605 1723 Acked Read 7 Done Sent Done false
2606 1724 Acked Read 9 Done Sent Done false
2607 1725 Acked Write 10 Done Sent Done false
2608 1726 Acked Write 5 Done Sent Done false
2609 1727 Acked Flush 0 Done Sent Done false
2610 1728 Acked Write 4 Done Sent Done false
2611 1729 Acked Read 6 Done Sent Done false
2612 1730 Acked Read 7 Done Sent Done false
2613 1731 Acked Write 10 Done Sent Done false
2614 1732 Acked Write 7 Done Sent Done false
2615 1733 Acked Read 9 Done Sent Done false
2616 1734 Acked Write 9 Done Sent Done false
2617 1735 Acked Write 9 Done Sent Done false
2618 1736 Acked Read 1 Done Sent Done false
2619 1737 Acked Read 8 Done Sent Done false
2620 1738 Acked Flush 0 Done Sent Done false
2621 1739 Acked Write 5 Done Sent Done false
2622 1740 Acked Write 5 Done Sent Done false
2623 1741 Acked Read 7 Done Sent Done false
2624 1742 Acked Read 10 Done Sent Done false
2625 1743 Acked Read 2 Done Sent Done false
2626 1744 Acked Read 2 Done Sent Done false
2627 1745 Acked Write 7 Done Sent Done false
2628 1746 Acked Read 1 Done Sent Done false
2629 1747 Acked Read 3 Done Sent Done false
2630 1748 Acked Read 5 Done Sent Done false
2631 1749 Acked Write 2 Done Sent Done false
2632 1750 Acked Read 3 Done Sent Done false
2633 1751 Acked Read 6 Done Sent Done false
2634 1752 Acked Read 2 Done Sent Done false
2635 1753 Acked Write 7 Done Sent Done false
2636 1754 Acked Flush 0 Done Sent Done false
2637 1755 Acked Write 10 Done Sent Done false
2638 1756 Acked Read 2 Done Sent Done false
2639 1757 Acked Read 6 Done Sent Done false
2640 1758 Acked Read 6 Done Sent Done false
2641 1759 Acked Flush 0 Done Sent Done false
2642 1760 Acked Write 1 Done Sent Done false
2643 1761 Acked Read 6 Done Sent Done false
2644 1762 Acked Read 10 Done Sent Done false
2645 1763 Acked Read 4 Done Sent Done false
2646 1764 Acked Write 2 Done Sent Done false
2647 1765 Acked Read 3 Done Sent Done false
2648 1766 Acked Read 3 Done Sent Done false
2649 1767 Acked Write 6 Done Sent Done false
2650 1768 Acked Read 10 Done Sent Done false
2651 1769 Acked Read 8 Done Sent Done false
2652 1770 Acked Write 8 Done Sent Done false
2653 1771 Acked Write 4 Done Sent Done false
2654 1772 Acked Write 4 Done Sent Done false
2655 1773 Acked Write 9 Done Sent Done false
2656 1774 Acked Flush 0 Done Sent Done false
2657 1775 Acked Read 1 Done Sent Done false
2658 1776 Acked Flush 0 Done Sent Done false
2659 1777 Acked Read 10 Done Sent Done false
2660 1778 Acked Read 2 Done Sent Done false
2661 1779 Acked Read 2 Done Sent Done false
2662 1780 Acked Write 10 Done Sent Done false
2663 1781 Acked Read 2 Done Sent Done false
2664 1782 Acked Read 2 Done Sent Done false
2665 1783 Acked Read 10 Done Sent Done false
2666 1784 Acked Read 4 Done Sent Done false
2667 1785 Acked Read 9 Done Sent Done false
2668 1786 Acked Write 10 Done Sent Done false
2669 1787 Acked Write 7 Done Sent Done false
2670 1788 Acked Read 10 Done Sent Done false
2671 1789 Acked Read 6 Done Sent Done false
2672 1790 Acked Write 8 Done Sent Done false
2673 1791 Acked Read 10 Done Sent Done false
2674 1792 Acked Write 2 Done Sent Done false
2675 1793 Acked Flush 0 Done Sent Done false
2676 1794 Acked Read 4 Done Sent Done false
2677 1795 Acked Write 8 Done Sent Done false
2678 1796 Acked Read 6 Done Sent Done false
2679 1797 Acked Write 4 Done Sent Done false
2680 1798 Acked Read 7 Done Sent Done false
2681 1799 Acked Write 3 Done Sent Done false
2682 1800 Acked Write 5 Done Sent Done false
2683 1801 Acked Read 6 Done Sent Done false
2684 1802 Acked Read 1 Done Sent Done false
2685 1803 Acked Read 3 Done Sent Done false
2686 1804 Acked Read 4 Done Sent Done false
2687 1805 Acked Read 8 Done Sent Done false
2688 1806 Acked Read 1 Done Sent Done false
2689 1807 Acked Read 10 Done Sent Done false
2690 1808 Acked Write 8 Done Sent Done false
2691 1809 Acked Read 2 Done Sent Done false
2692 1810 Acked Write 4 Done Sent Done false
2693 1811 Acked Flush 0 Done Sent Done false
2694 1812 Acked Read 8 Done Sent Done false
2695 1813 Acked Write 2 Done Sent Done false
2696 1814 Acked Read 8 Done Sent Done false
2697 1815 Acked Read 1 Done Sent Done false
2698 1816 Acked Read 2 Done Sent Done false
2699 1817 Acked Read 2 Done Sent Done false
2700 1818 Acked Flush 0 Done Sent Done false
2701 1819 Acked Write 3 Done Sent Done false
2702 1820 Acked Read 3 Done Sent Done false
2703 1821 Acked Write 7 Done Sent Done false
2704 1822 Acked Write 5 Done Sent Done false
2705 1823 Acked Write 3 Done Sent Done false
2706 1824 Acked Read 5 Done Sent Done false
2707 1825 Acked Read 8 Done Sent Done false
2708 1826 Acked Flush 0 Done Sent Done false
2709 1827 Acked Read 6 Done Sent Done false
2710 1828 Acked Read 1 Done Sent Done false
2711 1829 Acked Flush 0 Done Sent Done false
2712 1830 Acked Read 3 Done Sent Done false
2713 1831 Acked Write 2 Done Sent Done false
2714 1832 Acked Flush 0 Done Sent Done false
2715 1833 Acked Write 9 Done Sent Done false
2716 1834 Acked Read 4 Done Sent Done false
2717 1835 Acked Write 8 Done Sent Done false
2718 1836 Acked Read 10 Done Sent Done false
2719 1837 Acked Write 3 Done Sent Done false
2720 1838 Acked Flush 0 Done Sent Done false
2721 1839 Acked Read 4 Done Sent Done false
2722 1840 Acked Write 6 Done Sent Done false
2723 1841 Acked Write 6 Done Sent Done false
2724 1842 Acked Read 7 Done Sent Done false
2725 1843 Acked Read 7 Done Sent Done false
2726 1844 Acked Write 4 Done Sent Done false
2727 1845 Acked Write 7 Done Sent Done false
2728 1846 Acked Read 4 Done Sent Done false
2729 1847 Acked Write 2 Done Sent Done false
2730 1848 Acked Read 4 Done Sent Done false
2731 1849 Acked Write 10 Done Sent Done false
2732 1850 Acked Flush 0 Done Sent Done false
2733 1851 Acked Write 5 Done Sent Done false
2734 1852 Acked Read 3 Done Sent Done false
2735 1853 Acked Read 3 Done Sent Done false
2736 1854 Acked Write 4 Done Sent Done false
2737 1855 Acked Read 10 Done Sent Done false
2738 1856 Acked Flush 0 Done Sent Done false
2739 1857 Acked Write 10 Done Sent Done false
2740 1858 Acked Read 9 Done Sent Done false
2741 1859 Acked Read 7 Done Sent Done false
2742 1860 Acked Write 4 Done Sent Done false
2743 1861 Acked Read 9 Done Sent Done false
2744 1862 Acked Read 9 Done Sent Done false
2745 1863 Acked Read 4 Done Sent Done false
2746 1864 Acked Read 6 Done Sent Done false
2747 1865 Acked Write 1 Done Sent Done false
2748 1866 Acked Write 10 Done Sent Done false
2749 1867 Acked Flush 0 Done Sent Done false
2750 1868 Acked Write 7 Done Sent Done false
2751 1869 Acked Read 10 Done Sent Done false
2752 1870 Acked Read 5 Done Sent Done false
2753 1871 Acked Write 4 Done Sent Done false
2754 1872 Acked Write 9 Done Sent Done false
2755 1873 Acked Read 10 Done Sent Done false
2756 1874 Acked Read 1 Done Sent Done false
2757 1875 Acked Write 2 Done Sent Done false
2758 1876 Acked Read 3 Done Sent Done false
2759 1877 Acked Read 4 Done Sent Done false
2760 1878 Acked Read 2 Done Sent Done false
2761 1879 Acked Read 10 Done Sent Done false
2762 1880 Acked Flush 0 Done Sent Done false
2763 1881 Acked Read 4 Done Sent Done false
2764 1882 Acked Read 1 Done Sent Done false
2765 1883 Acked Read 7 Done Sent Done false
2766 1884 Acked Write 3 Done Sent Done false
2767 1885 Acked Read 9 Done Sent Done false
2768 1886 Acked Read 9 Done Sent Done false
2769 1887 Acked Write 10 Done Sent Done false
2770 1888 Acked Read 9 Done Sent Done false
2771 1889 Acked Read 9 Done Sent Done false
2772 1890 Acked Flush 0 Done Sent Done false
2773 1891 Acked Read 6 Done Sent Done false
2774 1892 Acked Write 2 Done Sent Done false
2775 1893 Acked Write 9 Done Sent Done false
2776 1894 Acked Read 3 Done Sent Done false
2777 1895 Acked Read 10 Done Sent Done false
2778 1896 Acked Read 8 Done Sent Done false
2779 1897 Acked Read 5 Done Sent Done false
2780 1898 Acked Read 9 Done Sent Done false
2781 1899 Acked Read 6 Done Sent Done false
2782 1900 Acked Write 6 Done Sent Done false
2783 1901 Acked Read 2 Done Sent Done false
2784 1902 Acked Read 2 Done Sent Done false
2785 1903 Acked Read 6 Done Sent Done false
2786 1904 Acked Read 6 Done Sent Done false
2787 STATES DS:0 DS:1 DS:2 TOTAL
2788 Sent 0 301 0 301
2789 Done 301 0 301 602
2790 Skipped 0 0 0 0
2791 Error 0 0 0 0
2792 Last Flush: Some(JobId(1890)) Some(JobId(1603)) Some(JobId(1890))
2793 Downstairs last five completed: 1603 1602 1601 1600 1599
2794 Upstairs last five completed: 1904 1903 1902 1901 1900
2795 CLIENT: Up:0 ds:301 act:2
27962025-03-17T22:16:48.642ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
27972025-03-17T22:16:49.643ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
27982025-03-17T22:16:50.644ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
27992025-03-17T22:16:51.645ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
2800 ----------------------------------------------------------------
2801 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2802 States: Active Offline Active
2803 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
2804 1604 Acked Read 3 Done Sent Done false
2805 1605 Acked Flush 0 Done Sent Done false
2806 1606 Acked Read 8 Done Sent Done false
2807 1607 Acked Write 4 Done Sent Done false
2808 1608 Acked Read 2 Done Sent Done false
2809 1609 Acked Flush 0 Done Sent Done false
2810 1610 Acked Read 2 Done Sent Done false
2811 1611 Acked Read 10 Done Sent Done false
2812 1612 Acked Read 4 Done Sent Done false
2813 1613 Acked Write 9 Done Sent Done false
2814 1614 Acked Read 7 Done Sent Done false
2815 1615 Acked Write 2 Done Sent Done false
2816 1616 Acked Read 3 Done Sent Done false
2817 1617 Acked Read 2 Done Sent Done false
2818 1618 Acked Read 6 Done Sent Done false
2819 1619 Acked Read 10 Done Sent Done false
2820 1620 Acked Write 10 Done Sent Done false
2821 1621 Acked Write 1 Done Sent Done false
2822 1622 Acked Write 6 Done Sent Done false
2823 1623 Acked Flush 0 Done Sent Done false
2824 1624 Acked Read 4 Done Sent Done false
2825 1625 Acked Read 9 Done Sent Done false
2826 1626 Acked Read 3 Done Sent Done false
2827 1627 Acked Read 2 Done Sent Done false
2828 1628 Acked Read 7 Done Sent Done false
2829 1629 Acked Write 2 Done Sent Done false
2830 1630 Acked Read 9 Done Sent Done false
2831 1631 Acked Write 6 Done Sent Done false
2832 1632 Acked Read 7 Done Sent Done false
2833 1633 Acked Read 3 Done Sent Done false
2834 1634 Acked Read 8 Done Sent Done false
2835 1635 Acked Read 2 Done Sent Done false
2836 1636 Acked Write 6 Done Sent Done false
2837 1637 Acked Read 10 Done Sent Done false
2838 1638 Acked Write 7 Done Sent Done false
2839 1639 Acked Read 6 Done Sent Done false
2840 1640 Acked Read 6 Done Sent Done false
2841 1641 Acked Read 7 Done Sent Done false
2842 1642 Acked Write 3 Done Sent Done false
2843 1643 Acked Write 8 Done Sent Done false
2844 1644 Acked Write 10 Done Sent Done false
2845 1645 Acked Flush 0 Done Sent Done false
2846 1646 Acked Write 1 Done Sent Done false
2847 1647 Acked Write 9 Done Sent Done false
2848 1648 Acked Flush 0 Done Sent Done false
2849 1649 Acked Read 10 Done Sent Done false
2850 1650 Acked Write 8 Done Sent Done false
2851 1651 Acked Write 10 Done Sent Done false
2852 1652 Acked Read 6 Done Sent Done false
2853 1653 Acked Write 6 Done Sent Done false
2854 1654 Acked Write 7 Done Sent Done false
2855 1655 Acked Write 3 Done Sent Done false
2856 1656 Acked Write 2 Done Sent Done false
2857 1657 Acked Read 10 Done Sent Done false
2858 1658 Acked Read 3 Done Sent Done false
2859 1659 Acked Write 3 Done Sent Done false
2860 1660 Acked Write 9 Done Sent Done false
2861 1661 Acked Read 6 Done Sent Done false
2862 1662 Acked Read 5 Done Sent Done false
2863 1663 Acked Write 9 Done Sent Done false
2864 1664 Acked Read 3 Done Sent Done false
2865 1665 Acked Read 3 Done Sent Done false
2866 1666 Acked Read 4 Done Sent Done false
2867 1667 Acked Write 2 Done Sent Done false
2868 1668 Acked Flush 0 Done Sent Done false
2869 1669 Acked Read 6 Done Sent Done false
2870 1670 Acked Flush 0 Done Sent Done false
2871 1671 Acked Read 6 Done Sent Done false
2872 1672 Acked Read 5 Done Sent Done false
2873 1673 Acked Read 8 Done Sent Done false
2874 1674 Acked Flush 0 Done Sent Done false
2875 1675 Acked Write 3 Done Sent Done false
2876 1676 Acked Write 2 Done Sent Done false
2877 1677 Acked Read 3 Done Sent Done false
2878 1678 Acked Write 8 Done Sent Done false
2879 1679 Acked Write 9 Done Sent Done false
2880 1680 Acked Write 6 Done Sent Done false
2881 1681 Acked Write 6 Done Sent Done false
2882 1682 Acked Read 8 Done Sent Done false
2883 1683 Acked Read 1 Done Sent Done false
2884 1684 Acked Read 4 Done Sent Done false
2885 1685 Acked Write 7 Done Sent Done false
2886 1686 Acked Write 9 Done Sent Done false
2887 1687 Acked Flush 0 Done Sent Done false
2888 1688 Acked Write 6 Done Sent Done false
2889 1689 Acked Write 10 Done Sent Done false
2890 1690 Acked Write 1 Done Sent Done false
2891 1691 Acked Flush 0 Done Sent Done false
2892 1692 Acked Write 6 Done Sent Done false
2893 1693 Acked Read 2 Done Sent Done false
2894 1694 Acked Read 10 Done Sent Done false
2895 1695 Acked Write 6 Done Sent Done false
2896 1696 Acked Read 10 Done Sent Done false
2897 1697 Acked Write 3 Done Sent Done false
2898 1698 Acked Write 6 Done Sent Done false
2899 1699 Acked Write 2 Done Sent Done false
2900 1700 Acked Write 2 Done Sent Done false
2901 1701 Acked Read 3 Done Sent Done false
2902 1702 Acked Write 2 Done Sent Done false
2903 1703 Acked Read 6 Done Sent Done false
2904 1704 Acked Write 7 Done Sent Done false
2905 1705 Acked Read 9 Done Sent Done false
2906 1706 Acked Read 8 Done Sent Done false
2907 1707 Acked Write 1 Done Sent Done false
2908 1708 Acked Read 10 Done Sent Done false
2909 1709 Acked Read 3 Done Sent Done false
2910 1710 Acked Read 2 Done Sent Done false
2911 1711 Acked Write 6 Done Sent Done false
2912 1712 Acked Write 4 Done Sent Done false
2913 1713 Acked Read 2 Done Sent Done false
2914 1714 Acked Read 6 Done Sent Done false
2915 1715 Acked Read 1 Done Sent Done false
2916 1716 Acked Write 3 Done Sent Done false
2917 1717 Acked Read 7 Done Sent Done false
2918 1718 Acked Write 9 Done Sent Done false
2919 1719 Acked Read 3 Done Sent Done false
2920 1720 Acked Write 8 Done Sent Done false
2921 1721 Acked Read 4 Done Sent Done false
2922 1722 Acked Write 5 Done Sent Done false
2923 1723 Acked Read 7 Done Sent Done false
2924 1724 Acked Read 9 Done Sent Done false
2925 1725 Acked Write 10 Done Sent Done false
2926 1726 Acked Write 5 Done Sent Done false
2927 1727 Acked Flush 0 Done Sent Done false
2928 1728 Acked Write 4 Done Sent Done false
2929 1729 Acked Read 6 Done Sent Done false
2930 1730 Acked Read 7 Done Sent Done false
2931 1731 Acked Write 10 Done Sent Done false
2932 1732 Acked Write 7 Done Sent Done false
2933 1733 Acked Read 9 Done Sent Done false
2934 1734 Acked Write 9 Done Sent Done false
2935 1735 Acked Write 9 Done Sent Done false
2936 1736 Acked Read 1 Done Sent Done false
2937 1737 Acked Read 8 Done Sent Done false
2938 1738 Acked Flush 0 Done Sent Done false
2939 1739 Acked Write 5 Done Sent Done false
2940 1740 Acked Write 5 Done Sent Done false
2941 1741 Acked Read 7 Done Sent Done false
2942 1742 Acked Read 10 Done Sent Done false
2943 1743 Acked Read 2 Done Sent Done false
2944 1744 Acked Read 2 Done Sent Done false
2945 1745 Acked Write 7 Done Sent Done false
2946 1746 Acked Read 1 Done Sent Done false
2947 1747 Acked Read 3 Done Sent Done false
2948 1748 Acked Read 5 Done Sent Done false
2949 1749 Acked Write 2 Done Sent Done false
2950 1750 Acked Read 3 Done Sent Done false
2951 1751 Acked Read 6 Done Sent Done false
2952 1752 Acked Read 2 Done Sent Done false
2953 1753 Acked Write 7 Done Sent Done false
2954 1754 Acked Flush 0 Done Sent Done false
2955 1755 Acked Write 10 Done Sent Done false
2956 1756 Acked Read 2 Done Sent Done false
2957 1757 Acked Read 6 Done Sent Done false
2958 1758 Acked Read 6 Done Sent Done false
2959 1759 Acked Flush 0 Done Sent Done false
2960 1760 Acked Write 1 Done Sent Done false
2961 1761 Acked Read 6 Done Sent Done false
2962 1762 Acked Read 10 Done Sent Done false
2963 1763 Acked Read 4 Done Sent Done false
2964 1764 Acked Write 2 Done Sent Done false
2965 1765 Acked Read 3 Done Sent Done false
2966 1766 Acked Read 3 Done Sent Done false
2967 1767 Acked Write 6 Done Sent Done false
2968 1768 Acked Read 10 Done Sent Done false
2969 1769 Acked Read 8 Done Sent Done false
2970 1770 Acked Write 8 Done Sent Done false
2971 1771 Acked Write 4 Done Sent Done false
2972 1772 Acked Write 4 Done Sent Done false
2973 1773 Acked Write 9 Done Sent Done false
2974 1774 Acked Flush 0 Done Sent Done false
2975 1775 Acked Read 1 Done Sent Done false
2976 1776 Acked Flush 0 Done Sent Done false
2977 1777 Acked Read 10 Done Sent Done false
2978 1778 Acked Read 2 Done Sent Done false
2979 1779 Acked Read 2 Done Sent Done false
2980 1780 Acked Write 10 Done Sent Done false
2981 1781 Acked Read 2 Done Sent Done false
2982 1782 Acked Read 2 Done Sent Done false
2983 1783 Acked Read 10 Done Sent Done false
2984 1784 Acked Read 4 Done Sent Done false
2985 1785 Acked Read 9 Done Sent Done false
2986 1786 Acked Write 10 Done Sent Done false
2987 1787 Acked Write 7 Done Sent Done false
2988 1788 Acked Read 10 Done Sent Done false
2989 1789 Acked Read 6 Done Sent Done false
2990 1790 Acked Write 8 Done Sent Done false
2991 1791 Acked Read 10 Done Sent Done false
2992 1792 Acked Write 2 Done Sent Done false
2993 1793 Acked Flush 0 Done Sent Done false
2994 1794 Acked Read 4 Done Sent Done false
2995 1795 Acked Write 8 Done Sent Done false
2996 1796 Acked Read 6 Done Sent Done false
2997 1797 Acked Write 4 Done Sent Done false
2998 1798 Acked Read 7 Done Sent Done false
2999 1799 Acked Write 3 Done Sent Done false
3000 1800 Acked Write 5 Done Sent Done false
3001 1801 Acked Read 6 Done Sent Done false
3002 1802 Acked Read 1 Done Sent Done false
3003 1803 Acked Read 3 Done Sent Done false
3004 1804 Acked Read 4 Done Sent Done false
3005 1805 Acked Read 8 Done Sent Done false
3006 1806 Acked Read 1 Done Sent Done false
3007 1807 Acked Read 10 Done Sent Done false
3008 1808 Acked Write 8 Done Sent Done false
3009 1809 Acked Read 2 Done Sent Done false
3010 1810 Acked Write 4 Done Sent Done false
3011 1811 Acked Flush 0 Done Sent Done false
3012 1812 Acked Read 8 Done Sent Done false
3013 1813 Acked Write 2 Done Sent Done false
3014 1814 Acked Read 8 Done Sent Done false
3015 1815 Acked Read 1 Done Sent Done false
3016 1816 Acked Read 2 Done Sent Done false
3017 1817 Acked Read 2 Done Sent Done false
3018 1818 Acked Flush 0 Done Sent Done false
3019 1819 Acked Write 3 Done Sent Done false
3020 1820 Acked Read 3 Done Sent Done false
3021 1821 Acked Write 7 Done Sent Done false
3022 1822 Acked Write 5 Done Sent Done false
3023 1823 Acked Write 3 Done Sent Done false
3024 1824 Acked Read 5 Done Sent Done false
3025 1825 Acked Read 8 Done Sent Done false
3026 1826 Acked Flush 0 Done Sent Done false
3027 1827 Acked Read 6 Done Sent Done false
3028 1828 Acked Read 1 Done Sent Done false
3029 1829 Acked Flush 0 Done Sent Done false
3030 1830 Acked Read 3 Done Sent Done false
3031 1831 Acked Write 2 Done Sent Done false
3032 1832 Acked Flush 0 Done Sent Done false
3033 1833 Acked Write 9 Done Sent Done false
3034 1834 Acked Read 4 Done Sent Done false
3035 1835 Acked Write 8 Done Sent Done false
3036 1836 Acked Read 10 Done Sent Done false
3037 1837 Acked Write 3 Done Sent Done false
3038 1838 Acked Flush 0 Done Sent Done false
3039 1839 Acked Read 4 Done Sent Done false
3040 1840 Acked Write 6 Done Sent Done false
3041 1841 Acked Write 6 Done Sent Done false
3042 1842 Acked Read 7 Done Sent Done false
3043 1843 Acked Read 7 Done Sent Done false
3044 1844 Acked Write 4 Done Sent Done false
3045 1845 Acked Write 7 Done Sent Done false
3046 1846 Acked Read 4 Done Sent Done false
3047 1847 Acked Write 2 Done Sent Done false
3048 1848 Acked Read 4 Done Sent Done false
3049 1849 Acked Write 10 Done Sent Done false
3050 1850 Acked Flush 0 Done Sent Done false
3051 1851 Acked Write 5 Done Sent Done false
3052 1852 Acked Read 3 Done Sent Done false
3053 1853 Acked Read 3 Done Sent Done false
3054 1854 Acked Write 4 Done Sent Done false
3055 1855 Acked Read 10 Done Sent Done false
3056 1856 Acked Flush 0 Done Sent Done false
3057 1857 Acked Write 10 Done Sent Done false
3058 1858 Acked Read 9 Done Sent Done false
3059 1859 Acked Read 7 Done Sent Done false
3060 1860 Acked Write 4 Done Sent Done false
3061 1861 Acked Read 9 Done Sent Done false
3062 1862 Acked Read 9 Done Sent Done false
3063 1863 Acked Read 4 Done Sent Done false
3064 1864 Acked Read 6 Done Sent Done false
3065 1865 Acked Write 1 Done Sent Done false
3066 1866 Acked Write 10 Done Sent Done false
3067 1867 Acked Flush 0 Done Sent Done false
3068 1868 Acked Write 7 Done Sent Done false
3069 1869 Acked Read 10 Done Sent Done false
3070 1870 Acked Read 5 Done Sent Done false
3071 1871 Acked Write 4 Done Sent Done false
3072 1872 Acked Write 9 Done Sent Done false
3073 1873 Acked Read 10 Done Sent Done false
3074 1874 Acked Read 1 Done Sent Done false
3075 1875 Acked Write 2 Done Sent Done false
3076 1876 Acked Read 3 Done Sent Done false
3077 1877 Acked Read 4 Done Sent Done false
3078 1878 Acked Read 2 Done Sent Done false
3079 1879 Acked Read 10 Done Sent Done false
3080 1880 Acked Flush 0 Done Sent Done false
3081 1881 Acked Read 4 Done Sent Done false
3082 1882 Acked Read 1 Done Sent Done false
3083 1883 Acked Read 7 Done Sent Done false
3084 1884 Acked Write 3 Done Sent Done false
3085 1885 Acked Read 9 Done Sent Done false
3086 1886 Acked Read 9 Done Sent Done false
3087 1887 Acked Write 10 Done Sent Done false
3088 1888 Acked Read 9 Done Sent Done false
3089 1889 Acked Read 9 Done Sent Done false
3090 1890 Acked Flush 0 Done Sent Done false
3091 1891 Acked Read 6 Done Sent Done false
3092 1892 Acked Write 2 Done Sent Done false
3093 1893 Acked Write 9 Done Sent Done false
3094 1894 Acked Read 3 Done Sent Done false
3095 1895 Acked Read 10 Done Sent Done false
3096 1896 Acked Read 8 Done Sent Done false
3097 1897 Acked Read 5 Done Sent Done false
3098 1898 Acked Read 9 Done Sent Done false
3099 1899 Acked Read 6 Done Sent Done false
3100 1900 Acked Write 6 Done Sent Done false
3101 1901 Acked Read 2 Done Sent Done false
3102 1902 Acked Read 2 Done Sent Done false
3103 1903 Acked Read 6 Done Sent Done false
3104 1904 Acked Read 6 Done Sent Done false
3105 STATES DS:0 DS:1 DS:2 TOTAL
3106 Sent 0 301 0 301
3107 Done 301 0 301 602
3108 Skipped 0 0 0 0
3109 Error 0 0 0 0
3110 Last Flush: Some(JobId(1890)) Some(JobId(1603)) Some(JobId(1890))
3111 Downstairs last five completed: 1603 1602 1601 1600 1599
3112 Upstairs last five completed: 1904 1903 1902 1901 1900
3113 CLIENT: Up:0 ds:301 act:2
31142025-03-17T22:16:52.648ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
31152025-03-17T22:16:53.649ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
31162025-03-17T22:16:53.656ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
31172025-03-17T22:16:53.656ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
31182025-03-17T22:16:53.656ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
31192025-03-17T22:16:53.656ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
31202025-03-17T22:16:53.656ZINFOcrucible: client task is exiting = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
31212025-03-17T22:16:54.652ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
31222025-03-17T22:16:55.653ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
3123 [8820] got start action
3124 [8820] I am going to restart
3125 Starting downstairs at port 8820
3126 Make output file at "/tmp/dsc/downstairs-8820.txt"
3127 [8820][1] reports Starting
3128 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1209)
3129 [8820][1] reports Running
3130 ----------------------------------------------------------------
3131 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3132 States: Active Offline Active
3133 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3134 1604 Acked Read 3 Done Sent Done false
3135 1605 Acked Flush 0 Done Sent Done false
3136 1606 Acked Read 8 Done Sent Done false
3137 1607 Acked Write 4 Done Sent Done false
3138 1608 Acked Read 2 Done Sent Done false
3139 1609 Acked Flush 0 Done Sent Done false
3140 1610 Acked Read 2 Done Sent Done false
3141 1611 Acked Read 10 Done Sent Done false
3142 1612 Acked Read 4 Done Sent Done false
3143 1613 Acked Write 9 Done Sent Done false
3144 1614 Acked Read 7 Done Sent Done false
3145 1615 Acked Write 2 Done Sent Done false
3146 1616 Acked Read 3 Done Sent Done false
3147 1617 Acked Read 2 Done Sent Done false
3148 1618 Acked Read 6 Done Sent Done false
3149 1619 Acked Read 10 Done Sent Done false
3150 1620 Acked Write 10 Done Sent Done false
3151 1621 Acked Write 1 Done Sent Done false
3152 1622 Acked Write 6 Done Sent Done false
3153 1623 Acked Flush 0 Done Sent Done false
3154 1624 Acked Read 4 Done Sent Done false
3155 1625 Acked Read 9 Done Sent Done false
3156 1626 Acked Read 3 Done Sent Done false
3157 1627 Acked Read 2 Done Sent Done false
3158 1628 Acked Read 7 Done Sent Done false
3159 1629 Acked Write 2 Done Sent Done false
3160 1630 Acked Read 9 Done Sent Done false
3161 1631 Acked Write 6 Done Sent Done false
3162 1632 Acked Read 7 Done Sent Done false
3163 1633 Acked Read 3 Done Sent Done false
3164 1634 Acked Read 8 Done Sent Done false
3165 1635 Acked Read 2 Done Sent Done false
3166 1636 Acked Write 6 Done Sent Done false
3167 1637 Acked Read 10 Done Sent Done false
3168 1638 Acked Write 7 Done Sent Done false
3169 1639 Acked Read 6 Done Sent Done false
3170 1640 Acked Read 6 Done Sent Done false
3171 1641 Acked Read 7 Done Sent Done false
3172 1642 Acked Write 3 Done Sent Done false
3173 1643 Acked Write 8 Done Sent Done false
3174 1644 Acked Write 10 Done Sent Done false
3175 1645 Acked Flush 0 Done Sent Done false
3176 1646 Acked Write 1 Done Sent Done false
3177 1647 Acked Write 9 Done Sent Done false
3178 1648 Acked Flush 0 Done Sent Done false
3179 1649 Acked Read 10 Done Sent Done false
3180 1650 Acked Write 8 Done Sent Done false
3181 1651 Acked Write 10 Done Sent Done false
3182 1652 Acked Read 6 Done Sent Done false
3183 1653 Acked Write 6 Done Sent Done false
3184 1654 Acked Write 7 Done Sent Done false
3185 1655 Acked Write 3 Done Sent Done false
3186 1656 Acked Write 2 Done Sent Done false
3187 1657 Acked Read 10 Done Sent Done false
3188 1658 Acked Read 3 Done Sent Done false
3189 1659 Acked Write 3 Done Sent Done false
3190 1660 Acked Write 9 Done Sent Done false
3191 1661 Acked Read 6 Done Sent Done false
3192 1662 Acked Read 5 Done Sent Done false
3193 1663 Acked Write 9 Done Sent Done false
3194 1664 Acked Read 3 Done Sent Done false
3195 1665 Acked Read 3 Done Sent Done false
3196 1666 Acked Read 4 Done Sent Done false
3197 1667 Acked Write 2 Done Sent Done false
3198 1668 Acked Flush 0 Done Sent Done false
3199 1669 Acked Read 6 Done Sent Done false
3200 1670 Acked Flush 0 Done Sent Done false
3201 1671 Acked Read 6 Done Sent Done false
3202 1672 Acked Read 5 Done Sent Done false
3203 1673 Acked Read 8 Done Sent Done false
3204 1674 Acked Flush 0 Done Sent Done false
3205 1675 Acked Write 3 Done Sent Done false
3206 1676 Acked Write 2 Done Sent Done false
3207 1677 Acked Read 3 Done Sent Done false
3208 1678 Acked Write 8 Done Sent Done false
3209 1679 Acked Write 9 Done Sent Done false
3210 1680 Acked Write 6 Done Sent Done false
3211 1681 Acked Write 6 Done Sent Done false
3212 1682 Acked Read 8 Done Sent Done false
3213 1683 Acked Read 1 Done Sent Done false
3214 1684 Acked Read 4 Done Sent Done false
3215 1685 Acked Write 7 Done Sent Done false
3216 1686 Acked Write 9 Done Sent Done false
3217 1687 Acked Flush 0 Done Sent Done false
3218 1688 Acked Write 6 Done Sent Done false
3219 1689 Acked Write 10 Done Sent Done false
3220 1690 Acked Write 1 Done Sent Done false
3221 1691 Acked Flush 0 Done Sent Done false
3222 1692 Acked Write 6 Done Sent Done false
3223 1693 Acked Read 2 Done Sent Done false
3224 1694 Acked Read 10 Done Sent Done false
3225 1695 Acked Write 6 Done Sent Done false
3226 1696 Acked Read 10 Done Sent Done false
3227 1697 Acked Write 3 Done Sent Done false
3228 1698 Acked Write 6 Done Sent Done false
3229 1699 Acked Write 2 Done Sent Done false
3230 1700 Acked Write 2 Done Sent Done false
3231 1701 Acked Read 3 Done Sent Done false
3232 1702 Acked Write 2 Done Sent Done false
3233 1703 Acked Read 6 Done Sent Done false
3234 1704 Acked Write 7 Done Sent Done false
3235 1705 Acked Read 9 Done Sent Done false
3236 1706 Acked Read 8 Done Sent Done false
3237 1707 Acked Write 1 Done Sent Done false
3238 1708 Acked Read 10 Done Sent Done false
3239 1709 Acked Read 3 Done Sent Done false
3240 1710 Acked Read 2 Done Sent Done false
3241 1711 Acked Write 6 Done Sent Done false
3242 1712 Acked Write 4 Done Sent Done false
3243 1713 Acked Read 2 Done Sent Done false
3244 1714 Acked Read 6 Done Sent Done false
3245 1715 Acked Read 1 Done Sent Done false
3246 1716 Acked Write 3 Done Sent Done false
3247 1717 Acked Read 7 Done Sent Done false
3248 1718 Acked Write 9 Done Sent Done false
3249 1719 Acked Read 3 Done Sent Done false
3250 1720 Acked Write 8 Done Sent Done false
3251 1721 Acked Read 4 Done Sent Done false
3252 1722 Acked Write 5 Done Sent Done false
3253 1723 Acked Read 7 Done Sent Done false
3254 1724 Acked Read 9 Done Sent Done false
3255 1725 Acked Write 10 Done Sent Done false
3256 1726 Acked Write 5 Done Sent Done false
3257 1727 Acked Flush 0 Done Sent Done false
3258 1728 Acked Write 4 Done Sent Done false
3259 1729 Acked Read 6 Done Sent Done false
3260 1730 Acked Read 7 Done Sent Done false
3261 1731 Acked Write 10 Done Sent Done false
3262 1732 Acked Write 7 Done Sent Done false
3263 1733 Acked Read 9 Done Sent Done false
3264 1734 Acked Write 9 Done Sent Done false
3265 1735 Acked Write 9 Done Sent Done false
3266 1736 Acked Read 1 Done Sent Done false
3267 1737 Acked Read 8 Done Sent Done false
3268 1738 Acked Flush 0 Done Sent Done false
3269 1739 Acked Write 5 Done Sent Done false
3270 1740 Acked Write 5 Done Sent Done false
3271 1741 Acked Read 7 Done Sent Done false
3272 1742 Acked Read 10 Done Sent Done false
3273 1743 Acked Read 2 Done Sent Done false
3274 1744 Acked Read 2 Done Sent Done false
3275 1745 Acked Write 7 Done Sent Done false
3276 1746 Acked Read 1 Done Sent Done false
3277 1747 Acked Read 3 Done Sent Done false
3278 1748 Acked Read 5 Done Sent Done false
3279 1749 Acked Write 2 Done Sent Done false
3280 1750 Acked Read 3 Done Sent Done false
3281 1751 Acked Read 6 Done Sent Done false
3282 1752 Acked Read 2 Done Sent Done false
3283 1753 Acked Write 7 Done Sent Done false
3284 1754 Acked Flush 0 Done Sent Done false
3285 1755 Acked Write 10 Done Sent Done false
3286 1756 Acked Read 2 Done Sent Done false
3287 1757 Acked Read 6 Done Sent Done false
3288 1758 Acked Read 6 Done Sent Done false
3289 1759 Acked Flush 0 Done Sent Done false
3290 1760 Acked Write 1 Done Sent Done false
3291 1761 Acked Read 6 Done Sent Done false
3292 1762 Acked Read 10 Done Sent Done false
3293 1763 Acked Read 4 Done Sent Done false
3294 1764 Acked Write 2 Done Sent Done false
3295 1765 Acked Read 3 Done Sent Done false
3296 1766 Acked Read 3 Done Sent Done false
3297 1767 Acked Write 6 Done Sent Done false
3298 1768 Acked Read 10 Done Sent Done false
3299 1769 Acked Read 8 Done Sent Done false
3300 1770 Acked Write 8 Done Sent Done false
3301 1771 Acked Write 4 Done Sent Done false
3302 1772 Acked Write 4 Done Sent Done false
3303 1773 Acked Write 9 Done Sent Done false
3304 1774 Acked Flush 0 Done Sent Done false
3305 1775 Acked Read 1 Done Sent Done false
3306 1776 Acked Flush 0 Done Sent Done false
3307 1777 Acked Read 10 Done Sent Done false
3308 1778 Acked Read 2 Done Sent Done false
3309 1779 Acked Read 2 Done Sent Done false
3310 1780 Acked Write 10 Done Sent Done false
3311 1781 Acked Read 2 Done Sent Done false
3312 1782 Acked Read 2 Done Sent Done false
3313 1783 Acked Read 10 Done Sent Done false
3314 1784 Acked Read 4 Done Sent Done false
3315 1785 Acked Read 9 Done Sent Done false
3316 1786 Acked Write 10 Done Sent Done false
3317 1787 Acked Write 7 Done Sent Done false
3318 1788 Acked Read 10 Done Sent Done false
3319 1789 Acked Read 6 Done Sent Done false
3320 1790 Acked Write 8 Done Sent Done false
3321 1791 Acked Read 10 Done Sent Done false
3322 1792 Acked Write 2 Done Sent Done false
3323 1793 Acked Flush 0 Done Sent Done false
3324 1794 Acked Read 4 Done Sent Done false
3325 1795 Acked Write 8 Done Sent Done false
3326 1796 Acked Read 6 Done Sent Done false
3327 1797 Acked Write 4 Done Sent Done false
3328 1798 Acked Read 7 Done Sent Done false
3329 1799 Acked Write 3 Done Sent Done false
3330 1800 Acked Write 5 Done Sent Done false
3331 1801 Acked Read 6 Done Sent Done false
3332 1802 Acked Read 1 Done Sent Done false
3333 1803 Acked Read 3 Done Sent Done false
3334 1804 Acked Read 4 Done Sent Done false
3335 1805 Acked Read 8 Done Sent Done false
3336 1806 Acked Read 1 Done Sent Done false
3337 1807 Acked Read 10 Done Sent Done false
3338 1808 Acked Write 8 Done Sent Done false
3339 1809 Acked Read 2 Done Sent Done false
3340 1810 Acked Write 4 Done Sent Done false
3341 1811 Acked Flush 0 Done Sent Done false
3342 1812 Acked Read 8 Done Sent Done false
3343 1813 Acked Write 2 Done Sent Done false
3344 1814 Acked Read 8 Done Sent Done false
3345 1815 Acked Read 1 Done Sent Done false
3346 1816 Acked Read 2 Done Sent Done false
3347 1817 Acked Read 2 Done Sent Done false
3348 1818 Acked Flush 0 Done Sent Done false
3349 1819 Acked Write 3 Done Sent Done false
3350 1820 Acked Read 3 Done Sent Done false
3351 1821 Acked Write 7 Done Sent Done false
3352 1822 Acked Write 5 Done Sent Done false
3353 1823 Acked Write 3 Done Sent Done false
3354 1824 Acked Read 5 Done Sent Done false
3355 1825 Acked Read 8 Done Sent Done false
3356 1826 Acked Flush 0 Done Sent Done false
3357 1827 Acked Read 6 Done Sent Done false
3358 1828 Acked Read 1 Done Sent Done false
3359 1829 Acked Flush 0 Done Sent Done false
3360 1830 Acked Read 3 Done Sent Done false
3361 1831 Acked Write 2 Done Sent Done false
3362 1832 Acked Flush 0 Done Sent Done false
3363 1833 Acked Write 9 Done Sent Done false
3364 1834 Acked Read 4 Done Sent Done false
3365 1835 Acked Write 8 Done Sent Done false
3366 1836 Acked Read 10 Done Sent Done false
3367 1837 Acked Write 3 Done Sent Done false
3368 1838 Acked Flush 0 Done Sent Done false
3369 1839 Acked Read 4 Done Sent Done false
3370 1840 Acked Write 6 Done Sent Done false
3371 1841 Acked Write 6 Done Sent Done false
3372 1842 Acked Read 7 Done Sent Done false
3373 1843 Acked Read 7 Done Sent Done false
3374 1844 Acked Write 4 Done Sent Done false
3375 1845 Acked Write 7 Done Sent Done false
3376 1846 Acked Read 4 Done Sent Done false
3377 1847 Acked Write 2 Done Sent Done false
3378 1848 Acked Read 4 Done Sent Done false
3379 1849 Acked Write 10 Done Sent Done false
3380 1850 Acked Flush 0 Done Sent Done false
3381 1851 Acked Write 5 Done Sent Done false
3382 1852 Acked Read 3 Done Sent Done false
3383 1853 Acked Read 3 Done Sent Done false
3384 1854 Acked Write 4 Done Sent Done false
3385 1855 Acked Read 10 Done Sent Done false
3386 1856 Acked Flush 0 Done Sent Done false
3387 1857 Acked Write 10 Done Sent Done false
3388 1858 Acked Read 9 Done Sent Done false
3389 1859 Acked Read 7 Done Sent Done false
3390 1860 Acked Write 4 Done Sent Done false
3391 1861 Acked Read 9 Done Sent Done false
3392 1862 Acked Read 9 Done Sent Done false
3393 1863 Acked Read 4 Done Sent Done false
3394 1864 Acked Read 6 Done Sent Done false
3395 1865 Acked Write 1 Done Sent Done false
3396 1866 Acked Write 10 Done Sent Done false
3397 1867 Acked Flush 0 Done Sent Done false
3398 1868 Acked Write 7 Done Sent Done false
3399 1869 Acked Read 10 Done Sent Done false
3400 1870 Acked Read 5 Done Sent Done false
3401 1871 Acked Write 4 Done Sent Done false
3402 1872 Acked Write 9 Done Sent Done false
3403 1873 Acked Read 10 Done Sent Done false
3404 1874 Acked Read 1 Done Sent Done false
3405 1875 Acked Write 2 Done Sent Done false
3406 1876 Acked Read 3 Done Sent Done false
3407 1877 Acked Read 4 Done Sent Done false
3408 1878 Acked Read 2 Done Sent Done false
3409 1879 Acked Read 10 Done Sent Done false
3410 1880 Acked Flush 0 Done Sent Done false
3411 1881 Acked Read 4 Done Sent Done false
3412 1882 Acked Read 1 Done Sent Done false
3413 1883 Acked Read 7 Done Sent Done false
3414 1884 Acked Write 3 Done Sent Done false
3415 1885 Acked Read 9 Done Sent Done false
3416 1886 Acked Read 9 Done Sent Done false
3417 1887 Acked Write 10 Done Sent Done false
3418 1888 Acked Read 9 Done Sent Done false
3419 1889 Acked Read 9 Done Sent Done false
3420 1890 Acked Flush 0 Done Sent Done false
3421 1891 Acked Read 6 Done Sent Done false
3422 1892 Acked Write 2 Done Sent Done false
3423 1893 Acked Write 9 Done Sent Done false
3424 1894 Acked Read 3 Done Sent Done false
3425 1895 Acked Read 10 Done Sent Done false
3426 1896 Acked Read 8 Done Sent Done false
3427 1897 Acked Read 5 Done Sent Done false
3428 1898 Acked Read 9 Done Sent Done false
3429 1899 Acked Read 6 Done Sent Done false
3430 1900 Acked Write 6 Done Sent Done false
3431 1901 Acked Read 2 Done Sent Done false
3432 1902 Acked Read 2 Done Sent Done false
3433 1903 Acked Read 6 Done Sent Done false
3434 1904 Acked Read 6 Done Sent Done false
3435 STATES DS:0 DS:1 DS:2 TOTAL
3436 Sent 0 301 0 301
3437 Done 301 0 301 602
3438 Skipped 0 0 0 0
3439 Error 0 0 0 0
3440 Last Flush: Some(JobId(1890)) Some(JobId(1603)) Some(JobId(1890))
3441 Downstairs last five completed: 1603 1602 1601 1600 1599
3442 Upstairs last five completed: 1904 1903 1902 1901 1900
3443 CLIENT: Up:0 ds:301 act:2
34442025-03-17T22:16:56.655ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
34452025-03-17T22:16:57.657ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
34462025-03-17T22:16:58.659ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
34472025-03-17T22:16:59.661ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
3448 ----------------------------------------------------------------
3449 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3450 States: Active Offline Active
3451 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3452 1604 Acked Read 3 Done Sent Done false
3453 1605 Acked Flush 0 Done Sent Done false
3454 1606 Acked Read 8 Done Sent Done false
3455 1607 Acked Write 4 Done Sent Done false
3456 1608 Acked Read 2 Done Sent Done false
3457 1609 Acked Flush 0 Done Sent Done false
3458 1610 Acked Read 2 Done Sent Done false
3459 1611 Acked Read 10 Done Sent Done false
3460 1612 Acked Read 4 Done Sent Done false
3461 1613 Acked Write 9 Done Sent Done false
3462 1614 Acked Read 7 Done Sent Done false
3463 1615 Acked Write 2 Done Sent Done false
3464 1616 Acked Read 3 Done Sent Done false
3465 1617 Acked Read 2 Done Sent Done false
3466 1618 Acked Read 6 Done Sent Done false
3467 1619 Acked Read 10 Done Sent Done false
3468 1620 Acked Write 10 Done Sent Done false
3469 1621 Acked Write 1 Done Sent Done false
3470 1622 Acked Write 6 Done Sent Done false
3471 1623 Acked Flush 0 Done Sent Done false
3472 1624 Acked Read 4 Done Sent Done false
3473 1625 Acked Read 9 Done Sent Done false
3474 1626 Acked Read 3 Done Sent Done false
3475 1627 Acked Read 2 Done Sent Done false
3476 1628 Acked Read 7 Done Sent Done false
3477 1629 Acked Write 2 Done Sent Done false
3478 1630 Acked Read 9 Done Sent Done false
3479 1631 Acked Write 6 Done Sent Done false
3480 1632 Acked Read 7 Done Sent Done false
3481 1633 Acked Read 3 Done Sent Done false
3482 1634 Acked Read 8 Done Sent Done false
3483 1635 Acked Read 2 Done Sent Done false
3484 1636 Acked Write 6 Done Sent Done false
3485 1637 Acked Read 10 Done Sent Done false
3486 1638 Acked Write 7 Done Sent Done false
3487 1639 Acked Read 6 Done Sent Done false
3488 1640 Acked Read 6 Done Sent Done false
3489 1641 Acked Read 7 Done Sent Done false
3490 1642 Acked Write 3 Done Sent Done false
3491 1643 Acked Write 8 Done Sent Done false
3492 1644 Acked Write 10 Done Sent Done false
3493 1645 Acked Flush 0 Done Sent Done false
3494 1646 Acked Write 1 Done Sent Done false
3495 1647 Acked Write 9 Done Sent Done false
3496 1648 Acked Flush 0 Done Sent Done false
3497 1649 Acked Read 10 Done Sent Done false
3498 1650 Acked Write 8 Done Sent Done false
3499 1651 Acked Write 10 Done Sent Done false
3500 1652 Acked Read 6 Done Sent Done false
3501 1653 Acked Write 6 Done Sent Done false
3502 1654 Acked Write 7 Done Sent Done false
3503 1655 Acked Write 3 Done Sent Done false
3504 1656 Acked Write 2 Done Sent Done false
3505 1657 Acked Read 10 Done Sent Done false
3506 1658 Acked Read 3 Done Sent Done false
3507 1659 Acked Write 3 Done Sent Done false
3508 1660 Acked Write 9 Done Sent Done false
3509 1661 Acked Read 6 Done Sent Done false
3510 1662 Acked Read 5 Done Sent Done false
3511 1663 Acked Write 9 Done Sent Done false
3512 1664 Acked Read 3 Done Sent Done false
3513 1665 Acked Read 3 Done Sent Done false
3514 1666 Acked Read 4 Done Sent Done false
3515 1667 Acked Write 2 Done Sent Done false
3516 1668 Acked Flush 0 Done Sent Done false
3517 1669 Acked Read 6 Done Sent Done false
3518 1670 Acked Flush 0 Done Sent Done false
3519 1671 Acked Read 6 Done Sent Done false
3520 1672 Acked Read 5 Done Sent Done false
3521 1673 Acked Read 8 Done Sent Done false
3522 1674 Acked Flush 0 Done Sent Done false
3523 1675 Acked Write 3 Done Sent Done false
3524 1676 Acked Write 2 Done Sent Done false
3525 1677 Acked Read 3 Done Sent Done false
3526 1678 Acked Write 8 Done Sent Done false
3527 1679 Acked Write 9 Done Sent Done false
3528 1680 Acked Write 6 Done Sent Done false
3529 1681 Acked Write 6 Done Sent Done false
3530 1682 Acked Read 8 Done Sent Done false
3531 1683 Acked Read 1 Done Sent Done false
3532 1684 Acked Read 4 Done Sent Done false
3533 1685 Acked Write 7 Done Sent Done false
3534 1686 Acked Write 9 Done Sent Done false
3535 1687 Acked Flush 0 Done Sent Done false
3536 1688 Acked Write 6 Done Sent Done false
3537 1689 Acked Write 10 Done Sent Done false
3538 1690 Acked Write 1 Done Sent Done false
3539 1691 Acked Flush 0 Done Sent Done false
3540 1692 Acked Write 6 Done Sent Done false
3541 1693 Acked Read 2 Done Sent Done false
3542 1694 Acked Read 10 Done Sent Done false
3543 1695 Acked Write 6 Done Sent Done false
3544 1696 Acked Read 10 Done Sent Done false
3545 1697 Acked Write 3 Done Sent Done false
3546 1698 Acked Write 6 Done Sent Done false
3547 1699 Acked Write 2 Done Sent Done false
3548 1700 Acked Write 2 Done Sent Done false
3549 1701 Acked Read 3 Done Sent Done false
3550 1702 Acked Write 2 Done Sent Done false
3551 1703 Acked Read 6 Done Sent Done false
3552 1704 Acked Write 7 Done Sent Done false
3553 1705 Acked Read 9 Done Sent Done false
3554 1706 Acked Read 8 Done Sent Done false
3555 1707 Acked Write 1 Done Sent Done false
3556 1708 Acked Read 10 Done Sent Done false
3557 1709 Acked Read 3 Done Sent Done false
3558 1710 Acked Read 2 Done Sent Done false
3559 1711 Acked Write 6 Done Sent Done false
3560 1712 Acked Write 4 Done Sent Done false
3561 1713 Acked Read 2 Done Sent Done false
3562 1714 Acked Read 6 Done Sent Done false
3563 1715 Acked Read 1 Done Sent Done false
3564 1716 Acked Write 3 Done Sent Done false
3565 1717 Acked Read 7 Done Sent Done false
3566 1718 Acked Write 9 Done Sent Done false
3567 1719 Acked Read 3 Done Sent Done false
3568 1720 Acked Write 8 Done Sent Done false
3569 1721 Acked Read 4 Done Sent Done false
3570 1722 Acked Write 5 Done Sent Done false
3571 1723 Acked Read 7 Done Sent Done false
3572 1724 Acked Read 9 Done Sent Done false
3573 1725 Acked Write 10 Done Sent Done false
3574 1726 Acked Write 5 Done Sent Done false
3575 1727 Acked Flush 0 Done Sent Done false
3576 1728 Acked Write 4 Done Sent Done false
3577 1729 Acked Read 6 Done Sent Done false
3578 1730 Acked Read 7 Done Sent Done false
3579 1731 Acked Write 10 Done Sent Done false
3580 1732 Acked Write 7 Done Sent Done false
3581 1733 Acked Read 9 Done Sent Done false
3582 1734 Acked Write 9 Done Sent Done false
3583 1735 Acked Write 9 Done Sent Done false
3584 1736 Acked Read 1 Done Sent Done false
3585 1737 Acked Read 8 Done Sent Done false
3586 1738 Acked Flush 0 Done Sent Done false
3587 1739 Acked Write 5 Done Sent Done false
3588 1740 Acked Write 5 Done Sent Done false
3589 1741 Acked Read 7 Done Sent Done false
3590 1742 Acked Read 10 Done Sent Done false
3591 1743 Acked Read 2 Done Sent Done false
3592 1744 Acked Read 2 Done Sent Done false
3593 1745 Acked Write 7 Done Sent Done false
3594 1746 Acked Read 1 Done Sent Done false
3595 1747 Acked Read 3 Done Sent Done false
3596 1748 Acked Read 5 Done Sent Done false
3597 1749 Acked Write 2 Done Sent Done false
3598 1750 Acked Read 3 Done Sent Done false
3599 1751 Acked Read 6 Done Sent Done false
3600 1752 Acked Read 2 Done Sent Done false
3601 1753 Acked Write 7 Done Sent Done false
3602 1754 Acked Flush 0 Done Sent Done false
3603 1755 Acked Write 10 Done Sent Done false
3604 1756 Acked Read 2 Done Sent Done false
3605 1757 Acked Read 6 Done Sent Done false
3606 1758 Acked Read 6 Done Sent Done false
3607 1759 Acked Flush 0 Done Sent Done false
3608 1760 Acked Write 1 Done Sent Done false
3609 1761 Acked Read 6 Done Sent Done false
3610 1762 Acked Read 10 Done Sent Done false
3611 1763 Acked Read 4 Done Sent Done false
3612 1764 Acked Write 2 Done Sent Done false
3613 1765 Acked Read 3 Done Sent Done false
3614 1766 Acked Read 3 Done Sent Done false
3615 1767 Acked Write 6 Done Sent Done false
3616 1768 Acked Read 10 Done Sent Done false
3617 1769 Acked Read 8 Done Sent Done false
3618 1770 Acked Write 8 Done Sent Done false
3619 1771 Acked Write 4 Done Sent Done false
3620 1772 Acked Write 4 Done Sent Done false
3621 1773 Acked Write 9 Done Sent Done false
3622 1774 Acked Flush 0 Done Sent Done false
3623 1775 Acked Read 1 Done Sent Done false
3624 1776 Acked Flush 0 Done Sent Done false
3625 1777 Acked Read 10 Done Sent Done false
3626 1778 Acked Read 2 Done Sent Done false
3627 1779 Acked Read 2 Done Sent Done false
3628 1780 Acked Write 10 Done Sent Done false
3629 1781 Acked Read 2 Done Sent Done false
3630 1782 Acked Read 2 Done Sent Done false
3631 1783 Acked Read 10 Done Sent Done false
3632 1784 Acked Read 4 Done Sent Done false
3633 1785 Acked Read 9 Done Sent Done false
3634 1786 Acked Write 10 Done Sent Done false
3635 1787 Acked Write 7 Done Sent Done false
3636 1788 Acked Read 10 Done Sent Done false
3637 1789 Acked Read 6 Done Sent Done false
3638 1790 Acked Write 8 Done Sent Done false
3639 1791 Acked Read 10 Done Sent Done false
3640 1792 Acked Write 2 Done Sent Done false
3641 1793 Acked Flush 0 Done Sent Done false
3642 1794 Acked Read 4 Done Sent Done false
3643 1795 Acked Write 8 Done Sent Done false
3644 1796 Acked Read 6 Done Sent Done false
3645 1797 Acked Write 4 Done Sent Done false
3646 1798 Acked Read 7 Done Sent Done false
3647 1799 Acked Write 3 Done Sent Done false
3648 1800 Acked Write 5 Done Sent Done false
3649 1801 Acked Read 6 Done Sent Done false
3650 1802 Acked Read 1 Done Sent Done false
3651 1803 Acked Read 3 Done Sent Done false
3652 1804 Acked Read 4 Done Sent Done false
3653 1805 Acked Read 8 Done Sent Done false
3654 1806 Acked Read 1 Done Sent Done false
3655 1807 Acked Read 10 Done Sent Done false
3656 1808 Acked Write 8 Done Sent Done false
3657 1809 Acked Read 2 Done Sent Done false
3658 1810 Acked Write 4 Done Sent Done false
3659 1811 Acked Flush 0 Done Sent Done false
3660 1812 Acked Read 8 Done Sent Done false
3661 1813 Acked Write 2 Done Sent Done false
3662 1814 Acked Read 8 Done Sent Done false
3663 1815 Acked Read 1 Done Sent Done false
3664 1816 Acked Read 2 Done Sent Done false
3665 1817 Acked Read 2 Done Sent Done false
3666 1818 Acked Flush 0 Done Sent Done false
3667 1819 Acked Write 3 Done Sent Done false
3668 1820 Acked Read 3 Done Sent Done false
3669 1821 Acked Write 7 Done Sent Done false
3670 1822 Acked Write 5 Done Sent Done false
3671 1823 Acked Write 3 Done Sent Done false
3672 1824 Acked Read 5 Done Sent Done false
3673 1825 Acked Read 8 Done Sent Done false
3674 1826 Acked Flush 0 Done Sent Done false
3675 1827 Acked Read 6 Done Sent Done false
3676 1828 Acked Read 1 Done Sent Done false
3677 1829 Acked Flush 0 Done Sent Done false
3678 1830 Acked Read 3 Done Sent Done false
3679 1831 Acked Write 2 Done Sent Done false
3680 1832 Acked Flush 0 Done Sent Done false
3681 1833 Acked Write 9 Done Sent Done false
3682 1834 Acked Read 4 Done Sent Done false
3683 1835 Acked Write 8 Done Sent Done false
3684 1836 Acked Read 10 Done Sent Done false
3685 1837 Acked Write 3 Done Sent Done false
3686 1838 Acked Flush 0 Done Sent Done false
3687 1839 Acked Read 4 Done Sent Done false
3688 1840 Acked Write 6 Done Sent Done false
3689 1841 Acked Write 6 Done Sent Done false
3690 1842 Acked Read 7 Done Sent Done false
3691 1843 Acked Read 7 Done Sent Done false
3692 1844 Acked Write 4 Done Sent Done false
3693 1845 Acked Write 7 Done Sent Done false
3694 1846 Acked Read 4 Done Sent Done false
3695 1847 Acked Write 2 Done Sent Done false
3696 1848 Acked Read 4 Done Sent Done false
3697 1849 Acked Write 10 Done Sent Done false
3698 1850 Acked Flush 0 Done Sent Done false
3699 1851 Acked Write 5 Done Sent Done false
3700 1852 Acked Read 3 Done Sent Done false
3701 1853 Acked Read 3 Done Sent Done false
3702 1854 Acked Write 4 Done Sent Done false
3703 1855 Acked Read 10 Done Sent Done false
3704 1856 Acked Flush 0 Done Sent Done false
3705 1857 Acked Write 10 Done Sent Done false
3706 1858 Acked Read 9 Done Sent Done false
3707 1859 Acked Read 7 Done Sent Done false
3708 1860 Acked Write 4 Done Sent Done false
3709 1861 Acked Read 9 Done Sent Done false
3710 1862 Acked Read 9 Done Sent Done false
3711 1863 Acked Read 4 Done Sent Done false
3712 1864 Acked Read 6 Done Sent Done false
3713 1865 Acked Write 1 Done Sent Done false
3714 1866 Acked Write 10 Done Sent Done false
3715 1867 Acked Flush 0 Done Sent Done false
3716 1868 Acked Write 7 Done Sent Done false
3717 1869 Acked Read 10 Done Sent Done false
3718 1870 Acked Read 5 Done Sent Done false
3719 1871 Acked Write 4 Done Sent Done false
3720 1872 Acked Write 9 Done Sent Done false
3721 1873 Acked Read 10 Done Sent Done false
3722 1874 Acked Read 1 Done Sent Done false
3723 1875 Acked Write 2 Done Sent Done false
3724 1876 Acked Read 3 Done Sent Done false
3725 1877 Acked Read 4 Done Sent Done false
3726 1878 Acked Read 2 Done Sent Done false
3727 1879 Acked Read 10 Done Sent Done false
3728 1880 Acked Flush 0 Done Sent Done false
3729 1881 Acked Read 4 Done Sent Done false
3730 1882 Acked Read 1 Done Sent Done false
3731 1883 Acked Read 7 Done Sent Done false
3732 1884 Acked Write 3 Done Sent Done false
3733 1885 Acked Read 9 Done Sent Done false
3734 1886 Acked Read 9 Done Sent Done false
3735 1887 Acked Write 10 Done Sent Done false
3736 1888 Acked Read 9 Done Sent Done false
3737 1889 Acked Read 9 Done Sent Done false
3738 1890 Acked Flush 0 Done Sent Done false
3739 1891 Acked Read 6 Done Sent Done false
3740 1892 Acked Write 2 Done Sent Done false
3741 1893 Acked Write 9 Done Sent Done false
3742 1894 Acked Read 3 Done Sent Done false
3743 1895 Acked Read 10 Done Sent Done false
3744 1896 Acked Read 8 Done Sent Done false
3745 1897 Acked Read 5 Done Sent Done false
3746 1898 Acked Read 9 Done Sent Done false
3747 1899 Acked Read 6 Done Sent Done false
3748 1900 Acked Write 6 Done Sent Done false
3749 1901 Acked Read 2 Done Sent Done false
3750 1902 Acked Read 2 Done Sent Done false
3751 1903 Acked Read 6 Done Sent Done false
3752 1904 Acked Read 6 Done Sent Done false
3753 STATES DS:0 DS:1 DS:2 TOTAL
3754 Sent 0 301 0 301
3755 Done 301 0 301 602
3756 Skipped 0 0 0 0
3757 Error 0 0 0 0
3758 Last Flush: Some(JobId(1890)) Some(JobId(1603)) Some(JobId(1890))
3759 Downstairs last five completed: 1603 1602 1601 1600 1599
3760 Upstairs last five completed: 1904 1903 1902 1901 1900
3761 CLIENT: Up:0 ds:301 act:2
37622025-03-17T22:17:00.662ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
37632025-03-17T22:17:01.664ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
37642025-03-17T22:17:02.667ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
37652025-03-17T22:17:03.657ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
37662025-03-17T22:17:03.658ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:46696) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
37672025-03-17T22:17:03.659ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
37682025-03-17T22:17:03.659ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
37692025-03-17T22:17:03.659ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
37702025-03-17T22:17:03.659ZINFOcrucible: send last flush ID to this DS: Some(JobId(1603)) = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
37712025-03-17T22:17:03.659ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
37722025-03-17T22:17:03.659ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(1603)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
37732025-03-17T22:17:03.659ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(1603)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
3774 ----------------------------------------------------------------
3775 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:14
3776 States: Active Active Active
3777 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
3778 1891 Acked Read 6 Done Done Done true
3779 1892 Acked Write 2 Done Done Done true
3780 1893 Acked Write 9 Done Done Done true
3781 1894 Acked Read 3 Done Done Done true
3782 1895 Acked Read 10 Done Done Done true
3783 1896 Acked Read 8 Done Done Done true
3784 1897 Acked Read 5 Done Done Done true
3785 1898 Acked Read 9 Done Done Done true
3786 1899 Acked Read 6 Done Done Done true
3787 1900 Acked Write 6 Done Done Done true
3788 1901 Acked Read 2 Done Done Done true
3789 1902 Acked Read 2 Done Done Done true
3790 1903 Acked Read 6 Done Done Done true
3791 1904 Acked Read 6 Done Done Done true
3792 STATES DS:0 DS:1 DS:2 TOTAL
3793 Sent 0 0 0 0
3794 Done 14 14 14 42
3795 Skipped 0 0 0 0
3796 Error 0 0 0 0
3797 Last Flush: Some(JobId(1890)) Some(JobId(1890)) Some(JobId(1890))
3798 Downstairs last five completed: 1890 1889 1888 1887 1886
3799 Upstairs last five completed: 1904 1903 1902 1901 1900
3800 CLIENT: Up:0 ds:14 act:3
3801 ----------------------------------------------------------------
3802 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
3803 Downstairs last five completed: 1905 1904 1903 1902 1901
3804 Upstairs last five completed: 1905 1904 1903 1902 1901
3805 CLIENT: Up:0 ds:0 act:3
3806 Replay: All jobs finished
3807 Main task has work to do, go find it
3808 got dsc Stop(0)
3809 stop 0
3810 Mar 17 22:17:07.919 INFO request completed, latency_us: [8810] Got stop action so:false kr:false
3811 329, response_code: 204, uri: /stop/cid/0, method: POST, req_id: bd503fb3-f032-4365-b245-dfe6136305dc, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
38122025-03-17T22:17:07.920ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
38132025-03-17T22:17:07.920ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
38142025-03-17T22:17:07.920ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
3815 Mar 17 22:17:07.920 INFO request completed, latency_us: 169, response_code: 200, uri: /state/cid/0, method: GET, req_id: a11e89c8-5cfb-4374-8898-68c8adfeedc3, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
38162025-03-17T22:17:07.920ZINFOcrucible: client task is exiting = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
3817 [8810] Exited with: ExitStatus(unix_wait_status(9))
3818 [8810][0] reports Exit
3819 Mar 17 22:17:11.922 INFO request completed, latency_us: 221, response_code: 200, uri: /state/cid/0, method: GET, req_id: dc723c56-cfbb-4de5-be2f-cb959fb4e773, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
3820 001/300 Read block 715 len 16384
3821 002/300 Write block 4464 len 32768 data: 2 2 2 2 3 3 4 4
3822 003/300 Read block 3212 len 32768
3823 004/300 Read block 4301 len 40960
3824 005/300 Read block 167 len 36864
3825 006/300 Read block 3121 len 24576
3826 007/300 Read block 3865 len 36864
3827 008/300 Write block 1086 len 4096 data: 3
3828 009/300 Read block 3699 len 16384
3829 010/300 Read block 3804 len 40960
3830 011/300 Read block 2026 len 20480
3831 012/300 Write block 4667 len 4096 data: 2
3832 013/300 Read block 2493 len 32768
3833 014/300 Read block 3386 len 28672
3834 015/300 Write block 3386 len 16384 data: 2 3 3 2
3835 016/300 Read block 4387 len 40960
3836 017/300 Write block 3460 len 36864 data: 3 3 2 2 2 2 2 2 2
3837 018/300 Write block 372 len 36864 data: 2 2 2 2 2 2 2 2 2
3838 019/300 Read block 3374 len 32768
3839 020/300 Write block 1989 len 16384 data: 2 2 2 2
3840 021/300 Write block 1389 len 20480 data: 2 2 2 2 3
3841 022/300 Read block 2879 len 20480
3842 023/300 Read block 4116 len 32768
3843 024/300 Write block 683 len 12288 data: 4 4 3
3844 025/300 Read block 187 len 40960
3845 026/300 Read block 1590 len 24576
3846 027/300 Write block 3453 len 8192 data: 2 2
3847 028/300 Read block 1941 len 24576
3848 029/300 Read block 4279 len 32768
3849 030/300 Read block 3599 len 24576
3850 031/300 Read block 4616 len 12288
3851 032/300 Write block 4943 len 4096 data: 2
3852 033/300 Write block 1075 len 20480 data: 3 3 3 3 3
3853 034/300 Write block 4083 len 32768 data: 2 2 2 2 2 2 2 2
3854 035/300 Read block 4572 len 32768
3855 036/300 Read block 1200 len 24576
3856 037/300 Flush
3857 038/300 Write block 27 len 8192 data: 3 3
3858 039/300 Read block 418 len 24576
3859 040/300 Write block 2146 len 40960 data: 3 3 3 3 3 2 2 2 2 2
3860 041/300 Write block 1871 len 4096 data: 3
3861 042/300 Read block 123 len 32768
3862 043/300 Read block 886 len 24576
3863 044/300 Write block 1958 len 32768 data: 3 3 3 3 3 3 3 3
3864 045/300 Read block 2240 len 12288
3865 046/300 Write block 4928 len 12288 data: 2 2 2
3866 047/300 Read block 2830 len 28672
3867 048/300 Write block 1747 len 32768 data: 3 3 3 3 3 3 3 3
3868 049/300 Read block 3108 len 36864
3869 050/300 Write block 269 len 20480 data: 2 3 3 3 3
3870 051/300 Read block 583 len 40960
3871 052/300 Read block 2974 len 28672
3872 053/300 Read block 4336 len 16384
3873 054/300 Write block 4376 len 8192 data: 2 2
3874 055/300 Read block 1299 len 12288
3875 056/300 Write block 4900 len 32768 data: 3 3 3 4 3 3 3 3
3876 057/300 Write block 2476 len 16384 data: 4 3 3 3
3877 058/300 Read block 1283 len 36864
3878 059/300 Write block 232 len 8192 data: 3 4
3879 060/300 Write block 10 len 12288 data: 2 2 3
3880 061/300 Read block 3344 len 12288
3881 062/300 Read block 1984 len 32768
3882 063/300 Read block 4976 len 20480
3883 064/300 Read block 1316 len 16384
3884 065/300 Read block 4654 len 12288
3885 066/300 Read block 2854 len 24576
3886 067/300 Flush
3887 068/300 Read block 1899 len 28672
3888 069/300 Read block 172 len 16384
3889 070/300 Write block 598 len 36864 data: 2 2 2 2 2 2 2 2 2
3890 071/300 Flush
3891 072/300 Read block 4605 len 24576
3892 073/300 Write block 2767 len 20480 data: 3 4 4 4 4
3893 074/300 Read block 2150 len 24576
3894 075/300 Write block 2206 len 40960 data: 2 2 2 2 2 2 2 2 3 4
3895 076/300 Flush
3896 077/300 Write block 4442 len 16384 data: 2 2 2 2
3897 078/300 Write block 784 len 8192 data: 2 2
3898 079/300 Flush
3899 080/300 Write block 2044 len 4096 data: 2
3900 081/300 Read block 2067 len 20480
3901 082/300 Flush
3902 083/300 Read block 1666 len 4096
3903 084/300 Write block 362 len 16384 data: 3 3 3 2
3904 085/300 Read block 4930 len 4096
3905 086/300 Read block 896 len 36864
3906 087/300 Write block 1040 len 12288 data: 3 3 3
3907 088/300 Write block 1366 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3908 089/300 Flush
3909 090/300 Write block 2671 len 32768 data: 2 3 3 3 3 3 2 2
3910 091/300 Write block 464 len 24576 data: 2 2 2 2 2 2
3911 092/300 Read block 4965 len 20480
3912 093/300 Read block 3425 len 4096
3913 094/300 Read block 144 len 4096
3914 095/300 Read block 4656 len 20480
3915 096/300 Write block 4732 len 4096 data: 3
3916 097/300 Write block 2904 len 28672 data: 2 2 2 2 2 3 2
3917 098/300 Write block 1704 len 16384 data: 3 3 3 3
3918 099/300 Read block 3489 len 28672
3919 100/300 Write block 829 len 20480 data: 3 2 2 2 2
3920 101/300 Read block 1297 len 4096
3921 102/300 Flush
3922 103/300 Read block 3486 len 20480
3923 104/300 Read block 156 len 20480
3924 105/300 Read block 320 len 4096
3925 106/300 Read block 3632 len 12288
3926 107/300 Write block 3970 len 36864 data: 2 2 2 2 2 2 2 2 2
3927 108/300 Read block 2105 len 8192
3928 109/300 Write block 3552 len 4096 data: 2
3929 110/300 Write block 4056 len 36864 data: 3 4 4 4 3 3 3 4 4
3930 111/300 Write block 1826 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3931 112/300 Read block 4060 len 4096
3932 113/300 Write block 3699 len 24576 data: 2 2 2 2 2 2
3933 114/300 Read block 4161 len 40960
3934 115/300 Flush
3935 116/300 Read block 2320 len 24576
3936 117/300 Write block 4217 len 16384 data: 2 2 2 2
3937 118/300 Read block 1345 len 8192
3938 119/300 Write block 1029 len 32768 data: 2 2 2 2 3 3 3 3
3939 120/300 Read block 1626 len 36864
3940 121/300 Write block 1826 len 4096 data: 3
3941 122/300 Read block 3767 len 24576
3942 123/300 Read block 2055 len 8192
3943 124/300 Write block 2201 len 40960 data: 3 3 3 3 3 3 3 3 3 3
3944 125/300 Write block 2036 len 20480 data: 2 2 2 2 2
3945 126/300 Read block 222 len 12288
3946 127/300 Read block 2814 len 28672
3947 128/300 Read block 91 len 16384
3948 129/300 Read block 4747 len 32768
3949 130/300 Write block 2586 len 36864 data: 2 2 2 2 2 2 2 2 3
3950 131/300 Write block 3880 len 36864 data: 4 4 4 4 3 3 3 3 2
3951 132/300 Read block 815 len 28672
3952 133/300 Read block 3962 len 16384
3953 134/300 Read block 4310 len 28672
3954 135/300 Write block 2335 len 20480 data: 3 3 3 3 3
3955 136/300 Read block 1533 len 24576
3956 137/300 Read block 467 len 24576
3957 138/300 Write block 42 len 20480 data: 2 2 2 2 2
3958 139/300 Read block 3037 len 8192
3959 140/300 Read block 1196 len 36864
3960 141/300 Write block 554 len 36864 data: 2 2 3 4 4 4 4 4 4
3961 142/300 Read block 375 len 24576
3962 143/300 Read block 1877 len 36864
3963 144/300 Read block 978 len 12288
3964 145/300 Write block 3914 len 20480 data: 2 2 2 2 2
3965 146/300 Write block 2956 len 24576 data: 2 2 2 2 2 2
3966 147/300 Write block 2200 len 20480 data: 3 4 4 4 4
3967 148/300 Read block 2066 len 16384
3968 149/300 Write block 4182 len 12288 data: 2 2 2
3969 150/300 Write block 2808 len 4096 data: 2
3970 151/300 Write block 75 len 8192 data: 3 4
3971 152/300 Write block 2219 len 32768 data: 5 5 4 4 3 3 2 2
3972 153/300 Read block 2638 len 8192
3973 154/300 Write block 1502 len 32768 data: 2 2 2 3 2 2 2 2
3974 155/300 Write block 4513 len 28672 data: 2 2 2 2 2 2 2
3975 156/300 Read block 969 len 24576
3976 157/300 Read block 2081 len 4096
3977 158/300 Write block 1027 len 24576 data: 2 2 3 3 3 3
3978 159/300 Write block 2117 len 28672 data: 3 3 3 2 2 2 2
3979 160/300 Flush
3980 161/300 Read block 2735 len 24576
3981 162/300 Write block 570 len 8192 data: 2 2
3982 163/300 Read block 3984 len 24576
3983 164/300 Write block 655 len 20480 data: 4 4 3 3 3
3984 165/300 Write block 2131 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3985 166/300 Write block 624 len 20480 data: 3 3 3 3 3
3986 167/300 Write block 3286 len 32768 data: 2 3 3 3 3 3 3 3
3987 168/300 Write block 1066 len 4096 data: 2
3988 169/300 Flush
3989 170/300 Read block 1673 len 36864
3990 171/300 Write block 3939 len 36864 data: 2 2 2 2 2 2 2 2 2
3991 172/300 Write block 3045 len 24576 data: 3 3 3 3 2 2
3992 173/300 Flush
3993 174/300 Read block 2997 len 32768
3994 175/300 Flush
3995 176/300 Read block 4738 len 12288
3996 177/300 Flush
3997 178/300 Read block 3307 len 16384
3998 179/300 Read block 3382 len 32768
3999 180/300 Read block 2753 len 36864
4000 181/300 Read block 3071 len 40960
4001 182/300 Write block 59 len 32768 data: 2 2 2 2 2 2 2 2
4002 183/300 Write block 1974 len 20480 data: 2 2 2 2 2
4003 184/300 Write block 2830 len 36864 data: 3 3 2 2 2 2 2 2 3
4004 185/300 Write block 2816 len 36864 data: 2 2 2 2 2 2 2 2 3
4005 186/300 Write block 585 len 20480 data: 2 2 2 2 2
4006 187/300 Read block 3937 len 20480
4007 188/300 Read block 2175 len 4096
4008 189/300 Write block 3671 len 8192 data: 2 2
4009 190/300 Write block 3202 len 32768 data: 3 3 3 3 3 3 2 2
4010 191/300 Read block 2928 len 4096
4011 192/300 Read block 1481 len 32768
4012 193/300 Read block 2688 len 40960
4013 194/300 Read block 4348 len 40960
4014 195/300 Read block 3161 len 24576
4015 196/300 Flush
4016 197/300 Read block 4497 len 32768
4017 198/300 Read block 1871 len 4096
4018 199/300 Write block 757 len 4096 data: 2
4019 200/300 Read block 1228 len 32768
4020 201/300 Read block 3034 len 4096
4021 202/300 Flush
4022 203/300 Write block 3658 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4023 204/300 Read block 3402 len 24576
4024 205/300 Read block 899 len 24576
4025 206/300 Read block 2033 len 24576
4026 207/300 Write block 3327 len 20480 data: 3 3 2 2 2
4027 208/300 Read block 1395 len 8192
4028 209/300 Read block 1245 len 28672
4029 210/300 Read block 4447 len 20480
4030 211/300 Write block 2110 len 20480 data: 3 3 3 3 3
4031 212/300 Flush
4032 213/300 Read block 2008 len 28672
4033 214/300 Read block 2660 len 8192
4034 215/300 Read block 3749 len 20480
4035 216/300 Write block 3215 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4036 217/300 Write block 301 len 28672 data: 2 2 2 3 3 3 3
4037 218/300 Read block 2537 len 32768
4038 219/300 Write block 2613 len 12288 data: 2 2 2
4039 220/300 Write block 4604 len 28672 data: 4 4 5 5 5 4 3
4040 221/300 Write block 222 len 4096 data: 2
4041 222/300 Read block 2371 len 8192
4042 223/300 Read block 2922 len 40960
4043 224/300 Read block 764 len 12288
4044 225/300 Read block 2161 len 8192
4045 226/300 Read block 3710 len 36864
4046 227/300 Write block 3785 len 28672 data: 2 2 2 2 2 2 2
4047 228/300 Write block 2206 len 40960 data: 4 4 4 4 4 3 3 3 4 5
4048 229/300 Write block 1010 len 12288 data: 2 2 2
4049 230/300 Write block 1785 len 8192 data: 2 2
4050 231/300 Read block 3781 len 12288
4051 232/300 Read block 3621 len 4096
4052 233/300 Write block 703 len 24576 data: 3 2 2 2 2 2
4053 234/300 Write block 1127 len 8192 data: 2 2
4054 235/300 Read block 540 len 20480
4055 236/300 Write block 4099 len 16384 data: 3 3 3 2
4056 237/300 Flush
4057 238/300 Read block 4800 len 4096
4058 239/300 Write block 1660 len 12288 data: 2 2 2
4059 240/300 Read block 4385 len 40960
4060 241/300 Read block 2256 len 12288
4061 242/300 Write block 1667 len 32768 data: 3 3 4 4 3 3 3 3
4062 243/300 Flush
4063 244/300 Write block 3816 len 24576 data: 2 2 2 2 3 3
4064 245/300 Write block 3458 len 32768 data: 3 3 4 4 3 3 3 3
4065 246/300 Read block 651 len 40960
4066 247/300 Read block 1916 len 12288
4067 248/300 Write block 184 len 36864 data: 4 5 4 4 4 4 2 2 2
4068 249/300 Write block 2926 len 40960 data: 2 2 2 2 4 4 4 4 4 3
4069 250/300 Write block 2380 len 24576 data: 4 4 4 4 4 4
4070 251/300 Read block 4453 len 8192
4071 252/300 Read block 4734 len 40960
4072 253/300 Read block 2535 len 24576
4073 254/300 Read block 2942 len 36864
4074 255/300 Read block 739 len 4096
4075 256/300 Flush
4076 257/300 Read block 1247 len 20480
4077 258/300 Read block 4341 len 8192
4078 259/300 Read block 1261 len 28672
4079 260/300 Read block 3599 len 40960
4080 261/300 Read block 3915 len 8192
4081 262/300 Read block 224 len 12288
4082 263/300 Write block 2880 len 8192 data: 4 3
4083 264/300 Read block 2305 len 24576
4084 265/300 Write block 2921 len 8192 data: 2 2
4085 266/300 Read block 4684 len 32768
4086 267/300 Read block 4032 len 20480
4087 268/300 Read block 436 len 12288
4088 269/300 Write block 2463 len 12288 data: 2 2 2
4089 270/300 Write block 1945 len 36864 data: 2 2 2 2 2 2 2 2 2
4090 271/300 Write block 629 len 24576 data: 3 3 2 2 2 2
4091 272/300 Flush
4092 273/300 Write block 2068 len 32768 data: 2 2 2 2 2 2 2 2
4093 274/300 Write block 1568 len 36864 data: 3 3 3 2 2 2 2 2 2
4094 275/300 Write block 2628 len 24576 data: 3 3 3 2 2 2
4095 276/300 Write block 3859 len 28672 data: 3 3 3 3 2 2 2
4096 277/300 Write block 1196 len 36864 data: 2 2 2 2 2 2 2 2 2
4097 278/300 Write block 4212 len 36864 data: 3 3 3 2 2 3 3 3 3
4098 279/300 Write block 1621 len 12288 data: 2 2 2
4099 280/300 Write block 4531 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4100 281/300 Read block 3976 len 32768
4101 282/300 Read block 2492 len 36864
4102 283/300 Read block 3513 len 36864
4103 284/300 Write block 4325 len 12288 data: 3 3 3
4104 285/300 Write block 4235 len 36864 data: 2 2 2 2 2 2 2 2 2
4105 286/300 Write block 3641 len 4096 data: 2
4106 287/300 Read block 1404 len 32768
4107 288/300 Write block 2926 len 36864 data: 3 3 3 3 5 5 5 5 5
4108 289/300 Read block 3786 len 20480
4109 290/300 Write block 493 len 20480 data: 2 2 2 2 3
4110 291/300 Write block 3892 len 8192 data: 2 2
4111 292/300 Write block 3026 len 4096 data: 3
4112 293/300 Flush
4113 294/300 Read block 1549 len 40960
4114 295/300 Read block 678 len 36864
4115 296/300 Flush
4116 297/300 Write block 26 len 4096 data: 3
4117 298/300 Write block 351 len 24576 data: 2 2 2 2 2 2
4118 299/300 Read block 1907 len 32768
4119 300/300 Write block 2256 len 20480 data: 2 2 2 2 2
4120 301/300 Write block 4807 len 24576 data: 4 4 4 4 3 3
4121 Main task has work to do, go find it
4122 got dsc Start(0)
4123 start 0
4124 Mar 17 22:17:12.171 INFO request completed, latency_us: 214, response_code: 204, uri: /start/cid/0, method: POST, req_id: fce48508-196d-4a36-a700-29810c1543d2, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
4125 Replay: started 0, returned:Ok(())
4126 ----------------------------------------------------------------
4127 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4128 States: Offline Active Active
4129 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4130 1906 Acked Read 4 Sent Done Done false
4131 1907 Acked Write 8 Sent Done Done false
4132 1908 Acked Read 8 Sent Done Done false
4133 1909 Acked Read 10 Sent Done Done false
4134 1910 Acked Read 9 Sent Done Done false
4135 1911 Acked Read 6 Sent Done Done false
4136 1912 Acked Read 9 Sent Done Done false
4137 1913 Acked Write 1 Sent Done Done false
4138 1914 Acked Read 4 Sent Done Done false
4139 1915 Acked Read 10 Sent Done Done false
4140 1916 Acked Read 5 Sent Done Done false
4141 1917 Acked Write 1 Sent Done Done false
4142 1918 Acked Read 8 Sent Done Done false
4143 1919 Acked Read 7 Sent Done Done false
4144 1920 Acked Write 4 Sent Done Done false
4145 1921 Acked Read 10 Sent Done Done false
4146 1922 Acked Write 9 Sent Done Done false
4147 1923 Acked Write 9 Sent Done Done false
4148 1924 Acked Read 8 Sent Done Done false
4149 1925 Acked Write 4 Sent Done Done false
4150 1926 Acked Write 5 Sent Done Done false
4151 1927 Acked Read 5 Sent Done Done false
4152 1928 Acked Read 8 Sent Done Done false
4153 1929 Acked Write 3 Sent Done Done false
4154 1930 Acked Read 10 Sent Done Done false
4155 1931 Acked Read 6 Sent Done Done false
4156 1932 Acked Write 2 Sent Done Done false
4157 1933 Acked Read 6 Sent Done Done false
4158 1934 Acked Read 8 Sent Done Done false
4159 1935 Acked Read 6 Sent Done Done false
4160 1936 Acked Read 3 Sent Done Done false
4161 1937 Acked Write 1 Sent Done Done false
4162 1938 Acked Write 5 Sent Done Done false
4163 1939 Acked Write 8 Sent Done Done false
4164 1940 Acked Read 8 Sent Done Done false
4165 1941 Acked Read 6 Sent Done Done false
4166 1942 Acked Flush 0 Sent Done Done false
4167 1943 Acked Write 2 Sent Done Done false
4168 1944 Acked Read 6 Sent Done Done false
4169 1945 Acked Write 10 Sent Done Done false
4170 1946 Acked Write 1 Sent Done Done false
4171 1947 Acked Read 8 Sent Done Done false
4172 1948 Acked Read 6 Sent Done Done false
4173 1949 Acked Write 8 Sent Done Done false
4174 1950 Acked Read 3 Sent Done Done false
4175 1951 Acked Write 3 Sent Done Done false
4176 1952 Acked Read 7 Sent Done Done false
4177 1953 Acked Write 8 Sent Done Done false
4178 1954 Acked Read 9 Sent Done Done false
4179 1955 Acked Write 5 Sent Done Done false
4180 1956 Acked Read 10 Sent Done Done false
4181 1957 Acked Read 7 Sent Done Done false
4182 1958 Acked Read 4 Sent Done Done false
4183 1959 Acked Write 2 Sent Done Done false
4184 1960 Acked Read 3 Sent Done Done false
4185 1961 Acked Write 8 Sent Done Done false
4186 1962 Acked Write 4 Sent Done Done false
4187 1963 Acked Read 9 Sent Done Done false
4188 1964 Acked Write 2 Sent Done Done false
4189 1965 Acked Write 3 Sent Done Done false
4190 1966 Acked Read 3 Sent Done Done false
4191 1967 Acked Read 8 Sent Done Done false
4192 1968 Acked Read 5 Sent Done Done false
4193 1969 Acked Read 4 Sent Done Done false
4194 1970 Acked Read 3 Sent Done Done false
4195 1971 Acked Read 6 Sent Done Done false
4196 1972 Acked Flush 0 Sent Done Done false
4197 1973 Acked Read 7 Sent Done Done false
4198 1974 Acked Read 4 Sent Done Done false
4199 1975 Acked Write 9 Sent Done Done false
4200 1976 Acked Flush 0 Sent Done Done false
4201 1977 Acked Read 6 Sent Done Done false
4202 1978 Acked Write 5 Sent Done Done false
4203 1979 Acked Read 6 Sent Done Done false
4204 1980 Acked Write 10 Sent Done Done false
4205 1981 Acked Flush 0 Sent Done Done false
4206 1982 Acked Write 4 Sent Done Done false
4207 1983 Acked Write 2 Sent Done Done false
4208 1984 Acked Flush 0 Sent Done Done false
4209 1985 Acked Write 1 Sent Done Done false
4210 1986 Acked Read 5 Sent Done Done false
4211 1987 Acked Flush 0 Sent Done Done false
4212 1988 Acked Read 1 Sent Done Done false
4213 1989 Acked Write 4 Sent Done Done false
4214 1990 Acked Read 1 Sent Done Done false
4215 1991 Acked Read 9 Sent Done Done false
4216 1992 Acked Write 3 Sent Done Done false
4217 1993 Acked Write 10 Sent Done Done false
4218 1994 Acked Flush 0 Sent Done Done false
4219 1995 Acked Write 8 Sent Done Done false
4220 1996 Acked Write 6 Sent Done Done false
4221 1997 Acked Read 5 Sent Done Done false
4222 1998 Acked Read 1 Sent Done Done false
4223 1999 Acked Read 1 Sent Done Done false
4224 2000 Acked Read 5 Sent Done Done false
4225 2001 Acked Write 1 Sent Done Done false
4226 2002 Acked Write 7 Sent Done Done false
4227 2003 Acked Write 4 Sent Done Done false
4228 2004 Acked Read 7 Sent Done Done false
4229 2005 Acked Write 5 Sent Done Done false
4230 2006 Acked Read 1 Sent Done Done false
4231 2007 Acked Flush 0 Sent Done Done false
4232 2008 Acked Read 5 Sent Done Done false
4233 2009 Acked Read 5 Sent Done Done false
4234 2010 Acked Read 1 Sent Done Done false
4235 2011 Acked Read 3 Sent Done Done false
4236 2012 Acked Write 9 Sent Done Done false
4237 2013 Acked Read 2 Sent Done Done false
4238 2014 Acked Write 1 Sent Done Done false
4239 2015 Acked Write 9 Sent Done Done false
4240 2016 Acked Write 10 Sent Done Done false
4241 2017 Acked Read 1 Sent Done Done false
4242 2018 Acked Write 6 Sent Done Done false
4243 2019 Acked Read 10 Sent Done Done false
4244 2020 Acked Flush 0 Sent Done Done false
4245 2021 Acked Read 6 Sent Done Done false
4246 2022 Acked Write 4 Sent Done Done false
4247 2023 Acked Read 2 Sent Done Done false
4248 2024 Acked Write 8 Sent Done Done false
4249 2025 Acked Read 9 Sent Done Done false
4250 2026 Acked Write 1 Sent Done Done false
4251 2027 Acked Read 6 Sent Done Done false
4252 2028 Acked Read 2 Sent Done Done false
4253 2029 Acked Write 10 Sent Done Done false
4254 2030 Acked Write 5 Sent Done Done false
4255 2031 Acked Read 3 Sent Done Done false
4256 2032 Acked Read 7 Sent Done Done false
4257 2033 Acked Read 4 Sent Done Done false
4258 2034 Acked Read 8 Sent Done Done false
4259 2035 Acked Write 9 Sent Done Done false
4260 2036 Acked Write 9 Sent Done Done false
4261 2037 Acked Read 7 Sent Done Done false
4262 2038 Acked Read 4 Sent Done Done false
4263 2039 Acked Read 7 Sent Done Done false
4264 2040 Acked Write 5 Sent Done Done false
4265 2041 Acked Read 6 Sent Done Done false
4266 2042 Acked Read 6 Sent Done Done false
4267 2043 Acked Write 5 Sent Done Done false
4268 2044 Acked Read 2 Sent Done Done false
4269 2045 Acked Read 9 Sent Done Done false
4270 2046 Acked Write 9 Sent Done Done false
4271 2047 Acked Read 6 Sent Done Done false
4272 2048 Acked Read 9 Sent Done Done false
4273 2049 Acked Read 3 Sent Done Done false
4274 2050 Acked Write 5 Sent Done Done false
4275 2051 Acked Write 6 Sent Done Done false
4276 2052 Acked Write 5 Sent Done Done false
4277 2053 Acked Read 4 Sent Done Done false
4278 2054 Acked Write 3 Sent Done Done false
4279 2055 Acked Write 1 Sent Done Done false
4280 2056 Acked Write 2 Sent Done Done false
4281 2057 Acked Write 8 Sent Done Done false
4282 2058 Acked Read 2 Sent Done Done false
4283 2059 Acked Write 8 Sent Done Done false
4284 2060 Acked Write 7 Sent Done Done false
4285 2061 Acked Read 6 Sent Done Done false
4286 2062 Acked Read 1 Sent Done Done false
4287 2063 Acked Write 6 Sent Done Done false
4288 2064 Acked Write 7 Sent Done Done false
4289 2065 Acked Flush 0 Sent Done Done false
4290 2066 Acked Read 6 Sent Done Done false
4291 2067 Acked Write 2 Sent Done Done false
4292 2068 Acked Read 6 Sent Done Done false
4293 2069 Acked Write 5 Sent Done Done false
4294 2070 Acked Write 10 Sent Done Done false
4295 2071 Acked Write 5 Sent Done Done false
4296 2072 Acked Write 8 Sent Done Done false
4297 2073 Acked Write 1 Sent Done Done false
4298 2074 Acked Flush 0 Sent Done Done false
4299 2075 Acked Read 9 Sent Done Done false
4300 2076 Acked Write 9 Sent Done Done false
4301 2077 Acked Write 6 Sent Done Done false
4302 2078 Acked Flush 0 Sent Done Done false
4303 2079 Acked Read 8 Sent Done Done false
4304 2080 Acked Flush 0 Sent Done Done false
4305 2081 Acked Read 3 Sent Done Done false
4306 2082 Acked Flush 0 Sent Done Done false
4307 2083 Acked Read 4 Sent Done Done false
4308 2084 Acked Read 8 Sent Done Done false
4309 2085 Acked Read 9 Sent Done Done false
4310 2086 Acked Read 10 Sent Done Done false
4311 2087 Acked Write 8 Sent Done Done false
4312 2088 Acked Write 5 Sent Done Done false
4313 2089 Acked Write 9 Sent Done Done false
4314 2090 Acked Write 9 Sent Done Done false
4315 2091 Acked Write 5 Sent Done Done false
4316 2092 Acked Read 5 Sent Done Done false
4317 2093 Acked Read 1 Sent Done Done false
4318 2094 Acked Write 2 Sent Done Done false
4319 2095 Acked Write 8 Sent Done Done false
4320 2096 Acked Read 1 Sent Done Done false
4321 2097 Acked Read 8 Sent Done Done false
4322 2098 Acked Read 10 Sent Done Done false
4323 2099 Acked Read 10 Sent Done Done false
4324 2100 Acked Read 6 Sent Done Done false
4325 2101 Acked Flush 0 Sent Done Done false
4326 2102 Acked Read 8 Sent Done Done false
4327 2103 Acked Read 1 Sent Done Done false
4328 2104 Acked Write 1 Sent Done Done false
4329 2105 Acked Read 8 Sent Done Done false
4330 2106 Acked Read 1 Sent Done Done false
4331 2107 Acked Flush 0 Sent Done Done false
4332 2108 Acked Write 10 Sent Done Done false
4333 2109 Acked Read 6 Sent Done Done false
4334 2110 Acked Read 6 Sent Done Done false
4335 2111 Acked Read 6 Sent Done Done false
4336 2112 Acked Write 5 Sent Done Done false
4337 2113 Acked Read 2 Sent Done Done false
4338 2114 Acked Read 7 Sent Done Done false
4339 2115 Acked Read 5 Sent Done Done false
4340 2116 Acked Write 5 Sent Done Done false
4341 2117 Acked Flush 0 Sent Done Done false
4342 2118 Acked Read 7 Sent Done Done false
4343 2119 Acked Read 2 Sent Done Done false
4344 2120 Acked Read 5 Sent Done Done false
4345 2121 Acked Write 10 Sent Done Done false
4346 2122 Acked Write 7 Sent Done Done false
4347 2123 Acked Read 8 Sent Done Done false
4348 2124 Acked Write 3 Sent Done Done false
4349 2125 Acked Write 7 Sent Done Done false
4350 2126 Acked Write 1 Sent Done Done false
4351 2127 Acked Read 2 Sent Done Done false
4352 2128 Acked Read 10 Sent Done Done false
4353 2129 Acked Read 3 Sent Done Done false
4354 2130 Acked Read 2 Sent Done Done false
4355 2131 Acked Read 9 Sent Done Done false
4356 2132 Acked Write 7 Sent Done Done false
4357 2133 Acked Write 10 Sent Done Done false
4358 2134 Acked Write 3 Sent Done Done false
4359 2135 Acked Write 2 Sent Done Done false
4360 2136 Acked Read 3 Sent Done Done false
4361 2137 Acked Read 1 Sent Done Done false
4362 2138 Acked Write 6 Sent Done Done false
4363 2139 Acked Write 2 Sent Done Done false
4364 2140 Acked Read 5 Sent Done Done false
4365 2141 Acked Write 4 Sent Done Done false
4366 2142 Acked Flush 0 Sent Done Done false
4367 2143 Acked Read 1 Sent Done Done false
4368 2144 Acked Write 3 Sent Done Done false
4369 2145 Acked Read 10 Sent Done Done false
4370 2146 Acked Read 3 Sent Done Done false
4371 2147 Acked Write 8 Sent Done Done false
4372 2148 Acked Flush 0 Sent Done Done false
4373 2149 Acked Write 6 Sent Done Done false
4374 2150 Acked Write 8 Sent Done Done false
4375 2151 Acked Read 10 Sent Done Done false
4376 2152 Acked Read 3 Sent Done Done false
4377 2153 Acked Write 9 Sent Done Done false
4378 2154 Acked Write 10 Sent Done Done false
4379 2155 Acked Write 6 Sent Done Done false
4380 2156 Acked Read 2 Sent Done Done false
4381 2157 Acked Read 10 Sent Done Done false
4382 2158 Acked Read 6 Sent Done Done false
4383 2159 Acked Read 9 Sent Done Done false
4384 2160 Acked Read 1 Sent Done Done false
4385 2161 Acked Flush 0 Sent Done Done false
4386 2162 Acked Read 5 Sent Done Done false
4387 2163 Acked Read 2 Sent Done Done false
4388 2164 Acked Read 7 Sent Done Done false
4389 2165 Acked Read 10 Sent Done Done false
4390 2166 Acked Read 2 Sent Done Done false
4391 2167 Acked Read 3 Sent Done Done false
4392 2168 Acked Write 2 Sent Done Done false
4393 2169 Acked Read 6 Sent Done Done false
4394 2170 Acked Write 2 Sent Done Done false
4395 2171 Acked Read 8 Sent Done Done false
4396 2172 Acked Read 5 Sent Done Done false
4397 2173 Acked Read 3 Sent Done Done false
4398 2174 Acked Write 3 Sent Done Done false
4399 2175 Acked Write 9 Sent Done Done false
4400 2176 Acked Write 6 Sent Done Done false
4401 2177 Acked Flush 0 Sent Done Done false
4402 2178 Acked Write 8 Sent Done Done false
4403 2179 Acked Write 9 Sent Done Done false
4404 2180 Acked Write 6 Sent Done Done false
4405 2181 Acked Write 7 Sent Done Done false
4406 2182 Acked Write 9 Sent Done Done false
4407 2183 Acked Write 9 Sent Done Done false
4408 2184 Acked Write 3 Sent Done Done false
4409 2185 Acked Write 10 Sent Done Done false
4410 2186 Acked Read 8 Sent Done Done false
4411 2187 Acked Read 9 Sent Done Done false
4412 2188 Acked Read 9 Sent Done Done false
4413 2189 Acked Write 3 Sent Done Done false
4414 2190 Acked Write 9 Sent Done Done false
4415 2191 Acked Write 1 Sent Done Done false
4416 2192 Acked Read 8 Sent Done Done false
4417 2193 Acked Write 9 Sent Done Done false
4418 2194 Acked Read 5 Sent Done Done false
4419 2195 Acked Write 5 Sent Done Done false
4420 2196 Acked Write 2 Sent Done Done false
4421 2197 Acked Write 1 Sent Done Done false
4422 2198 Acked Flush 0 Sent Done Done false
4423 2199 Acked Read 10 Sent Done Done false
4424 2200 Acked Read 9 Sent Done Done false
4425 2201 Acked Flush 0 Sent Done Done false
4426 2202 Acked Write 1 Sent Done Done false
4427 2203 Acked Write 6 Sent Done Done false
4428 2204 Acked Read 8 Sent Done Done false
4429 2205 Acked Write 5 Sent Done Done false
4430 2206 Acked Write 6 Sent Done Done false
4431 STATES DS:0 DS:1 DS:2 TOTAL
4432 Sent 301 0 0 301
4433 Done 0 301 301 602
4434 Skipped 0 0 0 0
4435 Error 0 0 0 0
4436 Last Flush: Some(JobId(1905)) Some(JobId(2201)) Some(JobId(2201))
4437 Downstairs last five completed: 1905 1904 1903 1902 1901
4438 Upstairs last five completed: 2206 2205 2204 2203 2202
4439 CLIENT: Up:0 ds:301 act:2
44402025-03-17T22:17:12.685ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
44412025-03-17T22:17:14.687ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
44422025-03-17T22:17:15.688ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
4443 [8810] got start action
4444 [8810] I am going to restart
4445 Starting downstairs at port 8810
4446 Make output file at "/tmp/dsc/downstairs-8810.txt"
4447 [8810][0] reports Starting
4448 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1210)
4449 [8810][0] reports Running
4450 ----------------------------------------------------------------
4451 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4452 States: Offline Active Active
4453 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
4454 1906 Acked Read 4 Sent Done Done false
4455 1907 Acked Write 8 Sent Done Done false
4456 1908 Acked Read 8 Sent Done Done false
4457 1909 Acked Read 10 Sent Done Done false
4458 1910 Acked Read 9 Sent Done Done false
4459 1911 Acked Read 6 Sent Done Done false
4460 1912 Acked Read 9 Sent Done Done false
4461 1913 Acked Write 1 Sent Done Done false
4462 1914 Acked Read 4 Sent Done Done false
4463 1915 Acked Read 10 Sent Done Done false
4464 1916 Acked Read 5 Sent Done Done false
4465 1917 Acked Write 1 Sent Done Done false
4466 1918 Acked Read 8 Sent Done Done false
4467 1919 Acked Read 7 Sent Done Done false
4468 1920 Acked Write 4 Sent Done Done false
4469 1921 Acked Read 10 Sent Done Done false
4470 1922 Acked Write 9 Sent Done Done false
4471 1923 Acked Write 9 Sent Done Done false
4472 1924 Acked Read 8 Sent Done Done false
4473 1925 Acked Write 4 Sent Done Done false
4474 1926 Acked Write 5 Sent Done Done false
4475 1927 Acked Read 5 Sent Done Done false
4476 1928 Acked Read 8 Sent Done Done false
4477 1929 Acked Write 3 Sent Done Done false
4478 1930 Acked Read 10 Sent Done Done false
4479 1931 Acked Read 6 Sent Done Done false
4480 1932 Acked Write 2 Sent Done Done false
4481 1933 Acked Read 6 Sent Done Done false
4482 1934 Acked Read 8 Sent Done Done false
4483 1935 Acked Read 6 Sent Done Done false
4484 1936 Acked Read 3 Sent Done Done false
4485 1937 Acked Write 1 Sent Done Done false
4486 1938 Acked Write 5 Sent Done Done false
4487 1939 Acked Write 8 Sent Done Done false
4488 1940 Acked Read 8 Sent Done Done false
4489 1941 Acked Read 6 Sent Done Done false
4490 1942 Acked Flush 0 Sent Done Done false
4491 1943 Acked Write 2 Sent Done Done false
4492 1944 Acked Read 6 Sent Done Done false
4493 1945 Acked Write 10 Sent Done Done false
4494 1946 Acked Write 1 Sent Done Done false
4495 1947 Acked Read 8 Sent Done Done false
4496 1948 Acked Read 6 Sent Done Done false
4497 1949 Acked Write 8 Sent Done Done false
4498 1950 Acked Read 3 Sent Done Done false
4499 1951 Acked Write 3 Sent Done Done false
4500 1952 Acked Read 7 Sent Done Done false
4501 1953 Acked Write 8 Sent Done Done false
4502 1954 Acked Read 9 Sent Done Done false
4503 1955 Acked Write 5 Sent Done Done false
4504 1956 Acked Read 10 Sent Done Done false
4505 1957 Acked Read 7 Sent Done Done false
4506 1958 Acked Read 4 Sent Done Done false
4507 1959 Acked Write 2 Sent Done Done false
4508 1960 Acked Read 3 Sent Done Done false
4509 1961 Acked Write 8 Sent Done Done false
4510 1962 Acked Write 4 Sent Done Done false
4511 1963 Acked Read 9 Sent Done Done false
4512 1964 Acked Write 2 Sent Done Done false
4513 1965 Acked Write 3 Sent Done Done false
4514 1966 Acked Read 3 Sent Done Done false
4515 1967 Acked Read 8 Sent Done Done false
4516 1968 Acked Read 5 Sent Done Done false
4517 1969 Acked Read 4 Sent Done Done false
4518 1970 Acked Read 3 Sent Done Done false
4519 1971 Acked Read 6 Sent Done Done false
4520 1972 Acked Flush 0 Sent Done Done false
4521 1973 Acked Read 7 Sent Done Done false
4522 1974 Acked Read 4 Sent Done Done false
4523 1975 Acked Write 9 Sent Done Done false
4524 1976 Acked Flush 0 Sent Done Done false
4525 1977 Acked Read 6 Sent Done Done false
4526 1978 Acked Write 5 Sent Done Done false
4527 1979 Acked Read 6 Sent Done Done false
4528 1980 Acked Write 10 Sent Done Done false
4529 1981 Acked Flush 0 Sent Done Done false
4530 1982 Acked Write 4 Sent Done Done false
4531 1983 Acked Write 2 Sent Done Done false
4532 1984 Acked Flush 0 Sent Done Done false
4533 1985 Acked Write 1 Sent Done Done false
4534 1986 Acked Read 5 Sent Done Done false
4535 1987 Acked Flush 0 Sent Done Done false
4536 1988 Acked Read 1 Sent Done Done false
4537 1989 Acked Write 4 Sent Done Done false
4538 1990 Acked Read 1 Sent Done Done false
4539 1991 Acked Read 9 Sent Done Done false
4540 1992 Acked Write 3 Sent Done Done false
4541 1993 Acked Write 10 Sent Done Done false
4542 1994 Acked Flush 0 Sent Done Done false
4543 1995 Acked Write 8 Sent Done Done false
4544 1996 Acked Write 6 Sent Done Done false
4545 1997 Acked Read 5 Sent Done Done false
4546 1998 Acked Read 1 Sent Done Done false
4547 1999 Acked Read 1 Sent Done Done false
4548 2000 Acked Read 5 Sent Done Done false
4549 2001 Acked Write 1 Sent Done Done false
4550 2002 Acked Write 7 Sent Done Done false
4551 2003 Acked Write 4 Sent Done Done false
4552 2004 Acked Read 7 Sent Done Done false
4553 2005 Acked Write 5 Sent Done Done false
4554 2006 Acked Read 1 Sent Done Done false
4555 2007 Acked Flush 0 Sent Done Done false
4556 2008 Acked Read 5 Sent Done Done false
4557 2009 Acked Read 5 Sent Done Done false
4558 2010 Acked Read 1 Sent Done Done false
4559 2011 Acked Read 3 Sent Done Done false
4560 2012 Acked Write 9 Sent Done Done false
4561 2013 Acked Read 2 Sent Done Done false
4562 2014 Acked Write 1 Sent Done Done false
4563 2015 Acked Write 9 Sent Done Done false
4564 2016 Acked Write 10 Sent Done Done false
4565 2017 Acked Read 1 Sent Done Done false
4566 2018 Acked Write 6 Sent Done Done false
4567 2019 Acked Read 10 Sent Done Done false
4568 2020 Acked Flush 0 Sent Done Done false
4569 2021 Acked Read 6 Sent Done Done false
4570 2022 Acked Write 4 Sent Done Done false
4571 2023 Acked Read 2 Sent Done Done false
4572 2024 Acked Write 8 Sent Done Done false
4573 2025 Acked Read 9 Sent Done Done false
4574 2026 Acked Write 1 Sent Done Done false
4575 2027 Acked Read 6 Sent Done Done false
4576 2028 Acked Read 2 Sent Done Done false
4577 2029 Acked Write 10 Sent Done Done false
4578 2030 Acked Write 5 Sent Done Done false
4579 2031 Acked Read 3 Sent Done Done false
4580 2032 Acked Read 7 Sent Done Done false
4581 2033 Acked Read 4 Sent Done Done false
4582 2034 Acked Read 8 Sent Done Done false
4583 2035 Acked Write 9 Sent Done Done false
4584 2036 Acked Write 9 Sent Done Done false
4585 2037 Acked Read 7 Sent Done Done false
4586 2038 Acked Read 4 Sent Done Done false
4587 2039 Acked Read 7 Sent Done Done false
4588 2040 Acked Write 5 Sent Done Done false
4589 2041 Acked Read 6 Sent Done Done false
4590 2042 Acked Read 6 Sent Done Done false
4591 2043 Acked Write 5 Sent Done Done false
4592 2044 Acked Read 2 Sent Done Done false
4593 2045 Acked Read 9 Sent Done Done false
4594 2046 Acked Write 9 Sent Done Done false
4595 2047 Acked Read 6 Sent Done Done false
4596 2048 Acked Read 9 Sent Done Done false
4597 2049 Acked Read 3 Sent Done Done false
4598 2050 Acked Write 5 Sent Done Done false
4599 2051 Acked Write 6 Sent Done Done false
4600 2052 Acked Write 5 Sent Done Done false
4601 2053 Acked Read 4 Sent Done Done false
4602 2054 Acked Write 3 Sent Done Done false
4603 2055 Acked Write 1 Sent Done Done false
4604 2056 Acked Write 2 Sent Done Done false
4605 2057 Acked Write 8 Sent Done Done false
4606 2058 Acked Read 2 Sent Done Done false
4607 2059 Acked Write 8 Sent Done Done false
4608 2060 Acked Write 7 Sent Done Done false
4609 2061 Acked Read 6 Sent Done Done false
4610 2062 Acked Read 1 Sent Done Done false
4611 2063 Acked Write 6 Sent Done Done false
4612 2064 Acked Write 7 Sent Done Done false
4613 2065 Acked Flush 0 Sent Done Done false
4614 2066 Acked Read 6 Sent Done Done false
4615 2067 Acked Write 2 Sent Done Done false
4616 2068 Acked Read 6 Sent Done Done false
4617 2069 Acked Write 5 Sent Done Done false
4618 2070 Acked Write 10 Sent Done Done false
4619 2071 Acked Write 5 Sent Done Done false
4620 2072 Acked Write 8 Sent Done Done false
4621 2073 Acked Write 1 Sent Done Done false
4622 2074 Acked Flush 0 Sent Done Done false
4623 2075 Acked Read 9 Sent Done Done false
4624 2076 Acked Write 9 Sent Done Done false
4625 2077 Acked Write 6 Sent Done Done false
4626 2078 Acked Flush 0 Sent Done Done false
4627 2079 Acked Read 8 Sent Done Done false
4628 2080 Acked Flush 0 Sent Done Done false
4629 2081 Acked Read 3 Sent Done Done false
4630 2082 Acked Flush 0 Sent Done Done false
4631 2083 Acked Read 4 Sent Done Done false
4632 2084 Acked Read 8 Sent Done Done false
4633 2085 Acked Read 9 Sent Done Done false
4634 2086 Acked Read 10 Sent Done Done false
4635 2087 Acked Write 8 Sent Done Done false
4636 2088 Acked Write 5 Sent Done Done false
4637 2089 Acked Write 9 Sent Done Done false
4638 2090 Acked Write 9 Sent Done Done false
4639 2091 Acked Write 5 Sent Done Done false
4640 2092 Acked Read 5 Sent Done Done false
4641 2093 Acked Read 1 Sent Done Done false
4642 2094 Acked Write 2 Sent Done Done false
4643 2095 Acked Write 8 Sent Done Done false
4644 2096 Acked Read 1 Sent Done Done false
4645 2097 Acked Read 8 Sent Done Done false
4646 2098 Acked Read 10 Sent Done Done false
4647 2099 Acked Read 10 Sent Done Done false
4648 2100 Acked Read 6 Sent Done Done false
4649 2101 Acked Flush 0 Sent Done Done false
4650 2102 Acked Read 8 Sent Done Done false
4651 2103 Acked Read 1 Sent Done Done false
4652 2104 Acked Write 1 Sent Done Done false
4653 2105 Acked Read 8 Sent Done Done false
4654 2106 Acked Read 1 Sent Done Done false
4655 2107 Acked Flush 0 Sent Done Done false
4656 2108 Acked Write 10 Sent Done Done false
4657 2109 Acked Read 6 Sent Done Done false
4658 2110 Acked Read 6 Sent Done Done false
4659 2111 Acked Read 6 Sent Done Done false
4660 2112 Acked Write 5 Sent Done Done false
4661 2113 Acked Read 2 Sent Done Done false
4662 2114 Acked Read 7 Sent Done Done false
4663 2115 Acked Read 5 Sent Done Done false
4664 2116 Acked Write 5 Sent Done Done false
4665 2117 Acked Flush 0 Sent Done Done false
4666 2118 Acked Read 7 Sent Done Done false
4667 2119 Acked Read 2 Sent Done Done false
4668 2120 Acked Read 5 Sent Done Done false
4669 2121 Acked Write 10 Sent Done Done false
4670 2122 Acked Write 7 Sent Done Done false
4671 2123 Acked Read 8 Sent Done Done false
4672 2124 Acked Write 3 Sent Done Done false
4673 2125 Acked Write 7 Sent Done Done false
4674 2126 Acked Write 1 Sent Done Done false
4675 2127 Acked Read 2 Sent Done Done false
4676 2128 Acked Read 10 Sent Done Done false
4677 2129 Acked Read 3 Sent Done Done false
4678 2130 Acked Read 2 Sent Done Done false
4679 2131 Acked Read 9 Sent Done Done false
4680 2132 Acked Write 7 Sent Done Done false
4681 2133 Acked Write 10 Sent Done Done false
4682 2134 Acked Write 3 Sent Done Done false
4683 2135 Acked Write 2 Sent Done Done false
4684 2136 Acked Read 3 Sent Done Done false
4685 2137 Acked Read 1 Sent Done Done false
4686 2138 Acked Write 6 Sent Done Done false
4687 2139 Acked Write 2 Sent Done Done false
4688 2140 Acked Read 5 Sent Done Done false
4689 2141 Acked Write 4 Sent Done Done false
4690 2142 Acked Flush 0 Sent Done Done false
4691 2143 Acked Read 1 Sent Done Done false
4692 2144 Acked Write 3 Sent Done Done false
4693 2145 Acked Read 10 Sent Done Done false
4694 2146 Acked Read 3 Sent Done Done false
4695 2147 Acked Write 8 Sent Done Done false
4696 2148 Acked Flush 0 Sent Done Done false
4697 2149 Acked Write 6 Sent Done Done false
4698 2150 Acked Write 8 Sent Done Done false
4699 2151 Acked Read 10 Sent Done Done false
4700 2152 Acked Read 3 Sent Done Done false
4701 2153 Acked Write 9 Sent Done Done false
4702 2154 Acked Write 10 Sent Done Done false
4703 2155 Acked Write 6 Sent Done Done false
4704 2156 Acked Read 2 Sent Done Done false
4705 2157 Acked Read 10 Sent Done Done false
4706 2158 Acked Read 6 Sent Done Done false
4707 2159 Acked Read 9 Sent Done Done false
4708 2160 Acked Read 1 Sent Done Done false
4709 2161 Acked Flush 0 Sent Done Done false
4710 2162 Acked Read 5 Sent Done Done false
4711 2163 Acked Read 2 Sent Done Done false
4712 2164 Acked Read 7 Sent Done Done false
4713 2165 Acked Read 10 Sent Done Done false
4714 2166 Acked Read 2 Sent Done Done false
4715 2167 Acked Read 3 Sent Done Done false
4716 2168 Acked Write 2 Sent Done Done false
4717 2169 Acked Read 6 Sent Done Done false
4718 2170 Acked Write 2 Sent Done Done false
4719 2171 Acked Read 8 Sent Done Done false
4720 2172 Acked Read 5 Sent Done Done false
4721 2173 Acked Read 3 Sent Done Done false
4722 2174 Acked Write 3 Sent Done Done false
4723 2175 Acked Write 9 Sent Done Done false
4724 2176 Acked Write 6 Sent Done Done false
4725 2177 Acked Flush 0 Sent Done Done false
4726 2178 Acked Write 8 Sent Done Done false
4727 2179 Acked Write 9 Sent Done Done false
4728 2180 Acked Write 6 Sent Done Done false
4729 2181 Acked Write 7 Sent Done Done false
4730 2182 Acked Write 9 Sent Done Done false
4731 2183 Acked Write 9 Sent Done Done false
4732 2184 Acked Write 3 Sent Done Done false
4733 2185 Acked Write 10 Sent Done Done false
4734 2186 Acked Read 8 Sent Done Done false
4735 2187 Acked Read 9 Sent Done Done false
4736 2188 Acked Read 9 Sent Done Done false
4737 2189 Acked Write 3 Sent Done Done false
4738 2190 Acked Write 9 Sent Done Done false
4739 2191 Acked Write 1 Sent Done Done false
4740 2192 Acked Read 8 Sent Done Done false
4741 2193 Acked Write 9 Sent Done Done false
4742 2194 Acked Read 5 Sent Done Done false
4743 2195 Acked Write 5 Sent Done Done false
4744 2196 Acked Write 2 Sent Done Done false
4745 2197 Acked Write 1 Sent Done Done false
4746 2198 Acked Flush 0 Sent Done Done false
4747 2199 Acked Read 10 Sent Done Done false
4748 2200 Acked Read 9 Sent Done Done false
4749 2201 Acked Flush 0 Sent Done Done false
4750 2202 Acked Write 1 Sent Done Done false
4751 2203 Acked Write 6 Sent Done Done false
4752 2204 Acked Read 8 Sent Done Done false
4753 2205 Acked Write 5 Sent Done Done false
4754 2206 Acked Write 6 Sent Done Done false
4755 STATES DS:0 DS:1 DS:2 TOTAL
4756 Sent 301 0 0 301
4757 Done 0 301 301 602
4758 Skipped 0 0 0 0
4759 Error 0 0 0 0
4760 Last Flush: Some(JobId(1905)) Some(JobId(2201)) Some(JobId(2201))
4761 Downstairs last five completed: 1905 1904 1903 1902 1901
4762 Upstairs last five completed: 2206 2205 2204 2203 2202
4763 CLIENT: Up:0 ds:301 act:2
47642025-03-17T22:17:16.689ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
47652025-03-17T22:17:17.691ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
47662025-03-17T22:17:17.922ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
47672025-03-17T22:17:17.922ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:44522) = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
47682025-03-17T22:17:17.923ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
47692025-03-17T22:17:17.924ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
47702025-03-17T22:17:17.924ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
47712025-03-17T22:17:17.924ZINFOcrucible: send last flush ID to this DS: Some(JobId(1905)) = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
47722025-03-17T22:17:17.924ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
47732025-03-17T22:17:17.924ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(1905)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
47742025-03-17T22:17:17.924ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(1905)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
4775 ----------------------------------------------------------------
4776 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
4777 Downstairs last five completed: 2207 2206 2205 2204 2203
4778 Upstairs last five completed: 2207 2206 2205 2204 2203
4779 CLIENT: Up:0 ds:0 act:3
4780 Replay: All jobs finished
4781 Main task has work to do, go find it
4782 got dsc Stop(2)
4783 stop 2
4784 Mar 17 22:17:20.176 INFO request completed, latency_us: 371, response_code: 204, uri: /stop/cid/2, method: POST, req_id: fc74c579-8391-4841-8913-76a5f6c25f90, remote_addr: 127.0.0.1:60423, local_addr:[8830] Got stop action so:false kr:false
4785 127.0.0.1:9998
4786 Mar 17 22:17:20.177 INFO request completed, latency_us: 162, response_code: 200, uri: /state/cid/2, method: GET, req_id: af38f59c-4b58-41f5-af31-543b109695d8, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
47872025-03-17T22:17:20.177ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
47882025-03-17T22:17:20.177ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
47892025-03-17T22:17:20.177ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
47902025-03-17T22:17:20.177ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
4791 [8830] Exited with: ExitStatus(unix_wait_status(9))
4792 [8830][2] reports Exit
4793 Mar 17 22:17:24.179 INFO request completed, latency_us: 265, response_code: 200, uri: /state/cid/2, method: GET, req_id: 1be3b3f6-9ed8-4c58-acb1-3f2c5e63b31f, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
4794 001/300 Write block 1176 len 12288 data: 2 2 2
4795 002/300 Read block 4460 len 16384
4796 003/300 Read block 3977 len 12288
4797 004/300 Flush
4798 005/300 Write block 292 len 16384 data: 2 2 2 2
4799 006/300 Read block 3245 len 36864
4800 007/300 Write block 3649 len 32768 data: 3 3 2 2 2 2 2 2
4801 008/300 Read block 2120 len 4096
4802 009/300 Write block 2615 len 36864 data: 3 2 3 3 3 3 3 2 3
4803 010/300 Read block 1573 len 8192
4804 011/300 Read block 1119 len 40960
4805 012/300 Read block 1833 len 8192
4806 013/300 Read block 2097 len 20480
4807 014/300 Write block 1417 len 12288 data: 2 2 2
4808 015/300 Read block 25 len 16384
4809 016/300 Flush
4810 017/300 Flush
4811 018/300 Read block 3675 len 8192
4812 019/300 Read block 2664 len 4096
4813 020/300 Write block 1013 len 36864 data: 2 2 2 2 2 2 2 2 2
4814 021/300 Write block 4651 len 24576 data: 2 2 2 2 2 2
4815 022/300 Flush
4816 023/300 Read block 3462 len 8192
4817 024/300 Read block 1032 len 36864
4818 025/300 Read block 3323 len 32768
4819 026/300 Write block 3490 len 12288 data: 2 2 3
4820 027/300 Write block 915 len 4096 data: 2
4821 028/300 Write block 607 len 20480 data: 2 2 2 2 2
4822 029/300 Read block 2193 len 32768
4823 030/300 Write block 3985 len 28672 data: 2 2 3 2 2 2 2
4824 031/300 Read block 4937 len 20480
4825 032/300 Read block 802 len 8192
4826 033/300 Write block 4963 len 4096 data: 2
4827 034/300 Read block 604 len 24576
4828 035/300 Read block 2166 len 36864
4829 036/300 Write block 4404 len 28672 data: 2 2 3 3 3 3 2
4830 037/300 Write block 2868 len 24576 data: 3 3 4 5 5 5
4831 038/300 Write block 3230 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4832 039/300 Write block 4146 len 24576 data: 2 2 2 2 2 2
4833 040/300 Read block 104 len 4096
4834 041/300 Read block 1233 len 36864
4835 042/300 Read block 2228 len 4096
4836 043/300 Write block 3603 len 20480 data: 2 2 3 3 2
4837 044/300 Write block 1219 len 40960 data: 3 3 3 3 2 2 2 2 2 2
4838 045/300 Read block 2476 len 40960
4839 046/300 Write block 2165 len 8192 data: 2 2
4840 047/300 Read block 3880 len 20480
4841 048/300 Flush
4842 049/300 Read block 1705 len 12288
4843 050/300 Read block 2681 len 4096
4844 051/300 Write block 632 len 4096 data: 3
4845 052/300 Write block 933 len 36864 data: 4 3 3 3 3 3 2 3 3
4846 053/300 Write block 1504 len 40960 data: 3 4 3 3 3 3 2 2 2 2
4847 054/300 Read block 2640 len 4096
4848 055/300 Flush
4849 056/300 Read block 569 len 36864
4850 057/300 Write block 557 len 16384 data: 5 5 5 5
4851 058/300 Write block 851 len 32768 data: 3 2 2 2 2 2 2 2
4852 059/300 Write block 4056 len 24576 data: 4 5 5 5 4 4
4853 060/300 Read block 126 len 32768
4854 061/300 Read block 4147 len 16384
4855 062/300 Write block 2703 len 20480 data: 2 2 2 2 2
4856 063/300 Read block 2191 len 8192
4857 064/300 Read block 2827 len 8192
4858 065/300 Write block 165 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4859 066/300 Write block 4469 len 20480 data: 4 5 5 4 4
4860 067/300 Write block 2565 len 28672 data: 2 2 2 2 2 2 2
4861 068/300 Read block 4871 len 4096
4862 069/300 Read block 4056 len 40960
4863 070/300 Write block 2820 len 32768 data: 3 3 3 3 4 4 4 4
4864 071/300 Read block 1922 len 12288
4865 072/300 Write block 4651 len 4096 data: 3
4866 073/300 Read block 1902 len 28672
4867 074/300 Read block 1436 len 8192
4868 075/300 Read block 3635 len 28672
4869 076/300 Write block 4463 len 40960 data: 2 3 3 3 3 4 5 6 6 5
4870 077/300 Read block 4973 len 24576
4871 078/300 Read block 3161 len 32768
4872 079/300 Write block 3114 len 36864 data: 2 2 2 2 3 3 2 2 2
4873 080/300 Read block 3549 len 40960
4874 081/300 Read block 3343 len 20480
4875 082/300 Read block 2077 len 12288
4876 083/300 Read block 87 len 12288
4877 084/300 Write block 190 len 28672 data: 3 3 3 2 2 2 2
4878 085/300 Write block 4158 len 36864 data: 2 2 3 3 3 3 3 2 2
4879 086/300 Read block 1044 len 32768
4880 087/300 Write block 1489 len 4096 data: 2
4881 088/300 Flush
4882 089/300 Write block 2796 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4883 090/300 Read block 1737 len 24576
4884 091/300 Read block 4736 len 16384
4885 092/300 Read block 2960 len 36864
4886 093/300 Read block 4780 len 28672
4887 094/300 Write block 4666 len 36864 data: 2 3 2 2 2 2 2 2 2
4888 095/300 Write block 2691 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4889 096/300 Read block 4277 len 36864
4890 097/300 Flush
4891 098/300 Write block 3585 len 36864 data: 2 2 2 2 2 2 2 2 2
4892 099/300 Read block 547 len 16384
4893 100/300 Read block 721 len 12288
4894 101/300 Read block 1847 len 32768
4895 102/300 Read block 3309 len 4096
4896 103/300 Read block 3366 len 40960
4897 104/300 Write block 313 len 28672 data: 2 2 2 2 2 2 2
4898 105/300 Write block 4046 len 8192 data: 2 3
4899 106/300 Read block 4564 len 36864
4900 107/300 Read block 2423 len 28672
4901 108/300 Write block 1807 len 24576 data: 2 2 2 2 2 2
4902 109/300 Read block 2156 len 36864
4903 110/300 Read block 3524 len 16384
4904 111/300 Write block 89 len 12288 data: 3 3 3
4905 112/300 Read block 3882 len 20480
4906 113/300 Flush
4907 114/300 Read block 1409 len 8192
4908 115/300 Read block 4610 len 4096
4909 116/300 Read block 4205 len 24576
4910 117/300 Read block 4973 len 24576
4911 118/300 Read block 2688 len 4096
4912 119/300 Read block 1418 len 40960
4913 120/300 Write block 1595 len 40960 data: 3 3 4 4 5 3 3 3 3 3
4914 121/300 Write block 720 len 8192 data: 3 3
4915 122/300 Read block 4376 len 8192
4916 123/300 Read block 2808 len 40960
4917 124/300 Read block 1270 len 24576
4918 125/300 Read block 4860 len 12288
4919 126/300 Write block 1500 len 24576 data: 2 2 3 3 4 5
4920 127/300 Read block 3801 len 16384
4921 128/300 Read block 1320 len 32768
4922 129/300 Write block 1845 len 36864 data: 2 2 2 2 2 2 2 2 2
4923 130/300 Write block 1272 len 40960 data: 2 2 2 3 3 3 3 2 2 2
4924 131/300 Read block 2154 len 24576
4925 132/300 Read block 165 len 8192
4926 133/300 Write block 4426 len 20480 data: 2 2 2 2 2
4927 134/300 Read block 1205 len 28672
4928 135/300 Read block 1830 len 20480
4929 136/300 Read block 4040 len 36864
4930 137/300 Read block 1815 len 28672
4931 138/300 Write block 1398 len 36864 data: 2 2 2 2 2 2 2 2 2
4932 139/300 Write block 4461 len 8192 data: 2 2
4933 140/300 Write block 3474 len 12288 data: 2 2 2
4934 141/300 Flush
4935 142/300 Write block 1869 len 12288 data: 3 3 4
4936 143/300 Write block 1794 len 24576 data: 2 2 2 2 3 3
4937 144/300 Write block 1756 len 8192 data: 2 2
4938 145/300 Read block 3151 len 8192
4939 146/300 Read block 1163 len 12288
4940 147/300 Read block 2988 len 40960
4941 148/300 Read block 3151 len 32768
4942 149/300 Read block 4401 len 8192
4943 150/300 Flush
4944 151/300 Flush
4945 152/300 Flush
4946 153/300 Read block 3082 len 20480
4947 154/300 Read block 2131 len 36864
4948 155/300 Read block 3171 len 8192
4949 156/300 Read block 310 len 8192
4950 157/300 Read block 1373 len 20480
4951 158/300 Read block 2544 len 20480
4952 159/300 Read block 4305 len 16384
4953 160/300 Read block 2215 len 40960
4954 161/300 Flush
4955 162/300 Read block 4093 len 24576
4956 163/300 Read block 4108 len 24576
4957 164/300 Read block 4572 len 20480
4958 165/300 Read block 3402 len 40960
4959 166/300 Read block 1235 len 16384
4960 167/300 Flush
4961 168/300 Read block 3686 len 40960
4962 169/300 Read block 1076 len 28672
4963 170/300 Read block 35 len 20480
4964 171/300 Write block 559 len 32768 data: 6 6 5 5 4 4 4 3
4965 172/300 Write block 3403 len 28672 data: 2 2 2 2 2 2 2
4966 173/300 Read block 2529 len 36864
4967 174/300 Write block 3159 len 40960 data: 2 2 2 2 2 2 2 3 3 3
4968 175/300 Read block 3458 len 8192
4969 176/300 Read block 2151 len 28672
4970 177/300 Write block 3584 len 40960 data: 2 3 3 3 3 3 3 3 3 3
4971 178/300 Read block 2137 len 36864
4972 179/300 Write block 157 len 36864 data: 2 2 2 2 2 2 2 2 3
4973 180/300 Flush
4974 181/300 Write block 3524 len 16384 data: 2 2 2 2
4975 182/300 Write block 1679 len 32768 data: 2 3 3 3 3 3 3 2
4976 183/300 Read block 4554 len 32768
4977 184/300 Write block 1212 len 32768 data: 3 2 2 3 3 3 3 4
4978 185/300 Write block 3062 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4979 186/300 Write block 1693 len 20480 data: 3 3 3 3 3
4980 187/300 Write block 587 len 24576 data: 3 3 3 3 2 2
4981 188/300 Read block 1498 len 20480
4982 189/300 Read block 3316 len 16384
4983 190/300 Read block 2652 len 12288
4984 191/300 Read block 4622 len 16384
4985 192/300 Write block 357 len 24576 data: 2 2 3 3 3 4
4986 193/300 Write block 2244 len 24576 data: 2 2 3 3 3 3
4987 194/300 Write block 1200 len 36864 data: 3 3 3 3 3 2 2 2 2
4988 195/300 Write block 2596 len 16384 data: 3 3 3 4
4989 196/300 Write block 2059 len 36864 data: 5 4 4 4 4 3 2 2 2
4990 197/300 Write block 809 len 4096 data: 2
4991 198/300 Read block 1737 len 20480
4992 199/300 Flush
4993 200/300 Read block 1847 len 32768
4994 201/300 Read block 4040 len 40960
4995 202/300 Read block 4185 len 36864
4996 203/300 Read block 4197 len 20480
4997 204/300 Write block 1543 len 8192 data: 2 2
4998 205/300 Read block 3373 len 32768
4999 206/300 Write block 4383 len 20480 data: 2 2 2 2 2
5000 207/300 Read block 1498 len 4096
5001 208/300 Read block 2419 len 40960
5002 209/300 Read block 3252 len 28672
5003 210/300 Write block 2358 len 36864 data: 4 4 2 2 2 2 2 2 3
5004 211/300 Write block 4271 len 12288 data: 2 2 2
5005 212/300 Read block 2321 len 4096
5006 213/300 Read block 8 len 12288
5007 214/300 Read block 2969 len 24576
5008 215/300 Read block 2776 len 16384
5009 216/300 Flush
5010 217/300 Write block 3590 len 32768 data: 4 4 4 4 2 2 2 2
5011 218/300 Read block 1730 len 40960
5012 219/300 Write block 2880 len 8192 data: 5 4
5013 220/300 Read block 4257 len 8192
5014 221/300 Write block 2207 len 4096 data: 5
5015 222/300 Write block 2191 len 4096 data: 2
5016 223/300 Write block 4545 len 4096 data: 2
5017 224/300 Read block 2520 len 40960
5018 225/300 Write block 4610 len 16384 data: 4 3 2 2
5019 226/300 Read block 442 len 4096
5020 227/300 Read block 2537 len 32768
5021 228/300 Read block 820 len 20480
5022 229/300 Read block 2406 len 36864
5023 230/300 Write block 3895 len 28672 data: 2 3 3 3 3 3 3
5024 231/300 Flush
5025 232/300 Read block 3630 len 4096
5026 233/300 Write block 2543 len 32768 data: 2 2 2 2 2 2 2 2
5027 234/300 Write block 3464 len 4096 data: 4
5028 235/300 Write block 1131 len 4096 data: 2
5029 236/300 Read block 4063 len 32768
5030 237/300 Write block 4549 len 32768 data: 2 2 2 2 2 2 2 2
5031 238/300 Read block 3763 len 24576
5032 239/300 Flush
5033 240/300 Write block 2664 len 8192 data: 2 2
5034 241/300 Read block 1795 len 24576
5035 242/300 Read block 2465 len 32768
5036 243/300 Write block 373 len 16384 data: 3 3 3 3
5037 244/300 Read block 4896 len 36864
5038 245/300 Read block 3527 len 8192
5039 246/300 Read block 1511 len 40960
5040 247/300 Write block 4266 len 4096 data: 2
5041 248/300 Read block 1515 len 24576
5042 249/300 Write block 3039 len 4096 data: 3
5043 250/300 Read block 4124 len 28672
5044 251/300 Flush
5045 252/300 Read block 4289 len 8192
5046 253/300 Read block 3438 len 8192
5047 254/300 Write block 1940 len 20480 data: 2 2 2 2 2
5048 255/300 Read block 727 len 24576
5049 256/300 Read block 1613 len 36864
5050 257/300 Write block 2930 len 4096 data: 6
5051 258/300 Flush
5052 259/300 Read block 651 len 32768
5053 260/300 Read block 3108 len 16384
5054 261/300 Read block 16 len 28672
5055 262/300 Read block 4703 len 16384
5056 263/300 Read block 2710 len 16384
5057 264/300 Write block 2478 len 32768 data: 4 4 2 2 2 2 2 2
5058 265/300 Read block 2904 len 24576
5059 266/300 Write block 3443 len 28672 data: 2 3 3 3 2 2 2
5060 267/300 Write block 4013 len 28672 data: 2 2 2 2 2 2 2
5061 268/300 Flush
5062 269/300 Write block 3799 len 32768 data: 3 3 3 3 2 2 2 3
5063 270/300 Write block 4079 len 36864 data: 2 2 2 2 3 3 3 3 3
5064 271/300 Read block 2441 len 24576
5065 272/300 Write block 3533 len 20480 data: 2 2 2 2 2
5066 273/300 Read block 781 len 32768
5067 274/300 Read block 2959 len 8192
5068 275/300 Write block 1465 len 24576 data: 4 4 4 3 3 3
5069 276/300 Read block 1718 len 16384
5070 277/300 Write block 1687 len 24576 data: 2 2 2 2 2 2
5071 278/300 Write block 4461 len 24576 data: 3 3 3 4 4 4
5072 279/300 Read block 4717 len 28672
5073 280/300 Write block 1528 len 36864 data: 4 4 4 4 4 5 5 4 4
5074 281/300 Write block 333 len 16384 data: 4 4 4 4
5075 282/300 Read block 3239 len 4096
5076 283/300 Read block 4977 len 8192
5077 284/300 Read block 796 len 4096
5078 285/300 Flush
5079 286/300 Read block 4568 len 24576
5080 287/300 Read block 2959 len 28672
5081 288/300 Write block 4631 len 36864 data: 2 2 2 2 2 2 2 2 2
5082 289/300 Write block 1356 len 40960 data: 3 3 3 3 3 3 3 3 2 2
5083 290/300 Flush
5084 291/300 Read block 3061 len 24576
5085 292/300 Read block 2316 len 36864
5086 293/300 Flush
5087 294/300 Write block 2803 len 28672 data: 3 3 3 2 2 3 2
5088 295/300 Read block 4884 len 36864
5089 296/300 Write block 2189 len 28672 data: 2 2 3 2 2 2 2
5090 297/300 Write block 1410 len 28672 data: 2 2 2 2 2 2 2
5091 298/300 Flush
5092 299/300 Read block 848 len 12288
5093 300/300 Read block 222 len 16384
5094 301/300 Flush
5095 Main task has work to do, go find it
5096 got dsc Start(2)
5097 start 2
5098 Mar 17 22:17:24.434 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/2, method: POST, req_id: 04cc5e8e-76ab-491d-bce9-13253fb8f7bf, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
5099 Replay: started 2, returned:Ok(())
5100 ----------------------------------------------------------------
5101 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5102 States: Active Active Offline
5103 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5104 2208 Acked Write 3 Done Done Sent false
5105 2209 Acked Read 4 Done Done Sent false
5106 2210 Acked Read 3 Done Done Sent false
5107 2211 Acked Flush 0 Done Done Sent false
5108 2212 Acked Write 4 Done Done Sent false
5109 2213 Acked Read 9 Done Done Sent false
5110 2214 Acked Write 8 Done Done Sent false
5111 2215 Acked Read 1 Done Done Sent false
5112 2216 Acked Write 9 Done Done Sent false
5113 2217 Acked Read 2 Done Done Sent false
5114 2218 Acked Read 10 Done Done Sent false
5115 2219 Acked Read 2 Done Done Sent false
5116 2220 Acked Read 5 Done Done Sent false
5117 2221 Acked Write 3 Done Done Sent false
5118 2222 Acked Read 4 Done Done Sent false
5119 2223 Acked Flush 0 Done Done Sent false
5120 2224 Acked Flush 0 Done Done Sent false
5121 2225 Acked Read 2 Done Done Sent false
5122 2226 Acked Read 1 Done Done Sent false
5123 2227 Acked Write 9 Done Done Sent false
5124 2228 Acked Write 6 Done Done Sent false
5125 2229 Acked Flush 0 Done Done Sent false
5126 2230 Acked Read 2 Done Done Sent false
5127 2231 Acked Read 9 Done Done Sent false
5128 2232 Acked Read 8 Done Done Sent false
5129 2233 Acked Write 3 Done Done Sent false
5130 2234 Acked Write 1 Done Done Sent false
5131 2235 Acked Write 5 Done Done Sent false
5132 2236 Acked Read 8 Done Done Sent false
5133 2237 Acked Write 7 Done Done Sent false
5134 2238 Acked Read 5 Done Done Sent false
5135 2239 Acked Read 2 Done Done Sent false
5136 2240 Acked Write 1 Done Done Sent false
5137 2241 Acked Read 6 Done Done Sent false
5138 2242 Acked Read 9 Done Done Sent false
5139 2243 Acked Write 7 Done Done Sent false
5140 2244 Acked Write 6 Done Done Sent false
5141 2245 Acked Write 10 Done Done Sent false
5142 2246 Acked Write 6 Done Done Sent false
5143 2247 Acked Read 1 Done Done Sent false
5144 2248 Acked Read 9 Done Done Sent false
5145 2249 Acked Read 1 Done Done Sent false
5146 2250 Acked Write 5 Done Done Sent false
5147 2251 Acked Write 10 Done Done Sent false
5148 2252 Acked Read 10 Done Done Sent false
5149 2253 Acked Write 2 Done Done Sent false
5150 2254 Acked Read 5 Done Done Sent false
5151 2255 Acked Flush 0 Done Done Sent false
5152 2256 Acked Read 3 Done Done Sent false
5153 2257 Acked Read 1 Done Done Sent false
5154 2258 Acked Write 1 Done Done Sent false
5155 2259 Acked Write 9 Done Done Sent false
5156 2260 Acked Write 10 Done Done Sent false
5157 2261 Acked Read 1 Done Done Sent false
5158 2262 Acked Flush 0 Done Done Sent false
5159 2263 Acked Read 9 Done Done Sent false
5160 2264 Acked Write 4 Done Done Sent false
5161 2265 Acked Write 8 Done Done Sent false
5162 2266 Acked Write 6 Done Done Sent false
5163 2267 Acked Read 8 Done Done Sent false
5164 2268 Acked Read 4 Done Done Sent false
5165 2269 Acked Write 5 Done Done Sent false
5166 2270 Acked Read 2 Done Done Sent false
5167 2271 Acked Read 2 Done Done Sent false
5168 2272 Acked Write 10 Done Done Sent false
5169 2273 Acked Write 5 Done Done Sent false
5170 2274 Acked Write 7 Done Done Sent false
5171 2275 Acked Read 1 Done Done Sent false
5172 2276 Acked Read 10 Done Done Sent false
5173 2277 Acked Write 8 Done Done Sent false
5174 2278 Acked Read 3 Done Done Sent false
5175 2279 Acked Write 1 Done Done Sent false
5176 2280 Acked Read 7 Done Done Sent false
5177 2281 Acked Read 2 Done Done Sent false
5178 2282 Acked Read 7 Done Done Sent false
5179 2283 Acked Write 10 Done Done Sent false
5180 2284 Acked Read 6 Done Done Sent false
5181 2285 Acked Read 8 Done Done Sent false
5182 2286 Acked Write 9 Done Done Sent false
5183 2287 Acked Read 10 Done Done Sent false
5184 2288 Acked Read 5 Done Done Sent false
5185 2289 Acked Read 3 Done Done Sent false
5186 2290 Acked Read 3 Done Done Sent false
5187 2291 Acked Write 7 Done Done Sent false
5188 2292 Acked Write 9 Done Done Sent false
5189 2293 Acked Read 8 Done Done Sent false
5190 2294 Acked Write 1 Done Done Sent false
5191 2295 Acked Flush 0 Done Done Sent false
5192 2296 Acked Write 10 Done Done Sent false
5193 2297 Acked Read 6 Done Done Sent false
5194 2298 Acked Read 4 Done Done Sent false
5195 2299 Acked Read 9 Done Done Sent false
5196 2300 Acked Read 7 Done Done Sent false
5197 2301 Acked Write 9 Done Done Sent false
5198 2302 Acked Write 10 Done Done Sent false
5199 2303 Acked Read 9 Done Done Sent false
5200 2304 Acked Flush 0 Done Done Sent false
5201 2305 Acked Write 9 Done Done Sent false
5202 2306 Acked Read 4 Done Done Sent false
5203 2307 Acked Read 3 Done Done Sent false
5204 2308 Acked Read 8 Done Done Sent false
5205 2309 Acked Read 1 Done Done Sent false
5206 2310 Acked Read 10 Done Done Sent false
5207 2311 Acked Write 7 Done Done Sent false
5208 2312 Acked Write 2 Done Done Sent false
5209 2313 Acked Read 9 Done Done Sent false
5210 2314 Acked Read 7 Done Done Sent false
5211 2315 Acked Write 6 Done Done Sent false
5212 2316 Acked Read 9 Done Done Sent false
5213 2317 Acked Read 4 Done Done Sent false
5214 2318 Acked Write 3 Done Done Sent false
5215 2319 Acked Read 5 Done Done Sent false
5216 2320 Acked Flush 0 Done Done Sent false
5217 2321 Acked Read 2 Done Done Sent false
5218 2322 Acked Read 1 Done Done Sent false
5219 2323 Acked Read 6 Done Done Sent false
5220 2324 Acked Read 6 Done Done Sent false
5221 2325 Acked Read 1 Done Done Sent false
5222 2326 Acked Read 10 Done Done Sent false
5223 2327 Acked Write 10 Done Done Sent false
5224 2328 Acked Write 2 Done Done Sent false
5225 2329 Acked Read 2 Done Done Sent false
5226 2330 Acked Read 10 Done Done Sent false
5227 2331 Acked Read 6 Done Done Sent false
5228 2332 Acked Read 3 Done Done Sent false
5229 2333 Acked Write 6 Done Done Sent false
5230 2334 Acked Read 4 Done Done Sent false
5231 2335 Acked Read 8 Done Done Sent false
5232 2336 Acked Write 9 Done Done Sent false
5233 2337 Acked Write 10 Done Done Sent false
5234 2338 Acked Read 6 Done Done Sent false
5235 2339 Acked Read 2 Done Done Sent false
5236 2340 Acked Write 5 Done Done Sent false
5237 2341 Acked Read 7 Done Done Sent false
5238 2342 Acked Read 5 Done Done Sent false
5239 2343 Acked Read 9 Done Done Sent false
5240 2344 Acked Read 7 Done Done Sent false
5241 2345 Acked Write 9 Done Done Sent false
5242 2346 Acked Write 2 Done Done Sent false
5243 2347 Acked Write 3 Done Done Sent false
5244 2348 Acked Flush 0 Done Done Sent false
5245 2349 Acked Write 3 Done Done Sent false
5246 2350 Acked Write 6 Done Done Sent false
5247 2351 Acked Write 2 Done Done Sent false
5248 2352 Acked Read 2 Done Done Sent false
5249 2353 Acked Read 3 Done Done Sent false
5250 2354 Acked Read 10 Done Done Sent false
5251 2355 Acked Read 8 Done Done Sent false
5252 2356 Acked Read 2 Done Done Sent false
5253 2357 Acked Flush 0 Done Done Sent false
5254 2358 Acked Flush 0 Done Done Sent false
5255 2359 Acked Flush 0 Done Done Sent false
5256 2360 Acked Read 5 Done Done Sent false
5257 2361 Acked Read 9 Done Done Sent false
5258 2362 Acked Read 2 Done Done Sent false
5259 2363 Acked Read 2 Done Done Sent false
5260 2364 Acked Read 5 Done Done Sent false
5261 2365 Acked Read 5 Done Done Sent false
5262 2366 Acked Read 4 Done Done Sent false
5263 2367 Acked Read 10 Done Done Sent false
5264 2368 Acked Flush 0 Done Done Sent false
5265 2369 Acked Read 6 Done Done Sent false
5266 2370 Acked Read 6 Done Done Sent false
5267 2371 Acked Read 5 Done Done Sent false
5268 2372 Acked Read 10 Done Done Sent false
5269 2373 Acked Read 4 Done Done Sent false
5270 2374 Acked Flush 0 Done Done Sent false
5271 2375 Acked Read 10 Done Done Sent false
5272 2376 Acked Read 7 Done Done Sent false
5273 2377 Acked Read 5 Done Done Sent false
5274 2378 Acked Write 8 Done Done Sent false
5275 2379 Acked Write 7 Done Done Sent false
5276 2380 Acked Read 9 Done Done Sent false
5277 2381 Acked Write 10 Done Done Sent false
5278 2382 Acked Read 2 Done Done Sent false
5279 2383 Acked Read 7 Done Done Sent false
5280 2384 Acked Write 10 Done Done Sent false
5281 2385 Acked Read 9 Done Done Sent false
5282 2386 Acked Write 9 Done Done Sent false
5283 2387 Acked Flush 0 Done Done Sent false
5284 2388 Acked Write 4 Done Done Sent false
5285 2389 Acked Write 8 Done Done Sent false
5286 2390 Acked Read 8 Done Done Sent false
5287 2391 Acked Write 8 Done Done Sent false
5288 2392 Acked Write 10 Done Done Sent false
5289 2393 Acked Write 5 Done Done Sent false
5290 2394 Acked Write 6 Done Done Sent false
5291 2395 Acked Read 5 Done Done Sent false
5292 2396 Acked Read 4 Done Done Sent false
5293 2397 Acked Read 3 Done Done Sent false
5294 2398 Acked Read 4 Done Done Sent false
5295 2399 Acked Write 6 Done Done Sent false
5296 2400 Acked Write 6 Done Done Sent false
5297 2401 Acked Write 9 Done Done Sent false
5298 2402 Acked Write 4 Done Done Sent false
5299 2403 Acked Write 9 Done Done Sent false
5300 2404 Acked Write 1 Done Done Sent false
5301 2405 Acked Read 5 Done Done Sent false
5302 2406 Acked Flush 0 Done Done Sent false
5303 2407 Acked Read 8 Done Done Sent false
5304 2408 Acked Read 10 Done Done Sent false
5305 2409 Acked Read 9 Done Done Sent false
5306 2410 Acked Read 5 Done Done Sent false
5307 2411 Acked Write 2 Done Done Sent false
5308 2412 Acked Read 8 Done Done Sent false
5309 2413 Acked Write 5 Done Done Sent false
5310 2414 Acked Read 1 Done Done Sent false
5311 2415 Acked Read 10 Done Done Sent false
5312 2416 Acked Read 7 Done Done Sent false
5313 2417 Acked Write 9 Done Done Sent false
5314 2418 Acked Write 3 Done Done Sent false
5315 2419 Acked Read 1 Done Done Sent false
5316 2420 Acked Read 3 Done Done Sent false
5317 2421 Acked Read 6 Done Done Sent false
5318 2422 Acked Read 4 Done Done Sent false
5319 2423 Acked Flush 0 Done Done Sent false
5320 2424 Acked Write 8 Done Done Sent false
5321 2425 Acked Read 10 Done Done Sent false
5322 2426 Acked Write 2 Done Done Sent false
5323 2427 Acked Read 2 Done Done Sent false
5324 2428 Acked Write 1 Done Done Sent false
5325 2429 Acked Write 1 Done Done Sent false
5326 2430 Acked Write 1 Done Done Sent false
5327 2431 Acked Read 10 Done Done Sent false
5328 2432 Acked Write 4 Done Done Sent false
5329 2433 Acked Read 1 Done Done Sent false
5330 2434 Acked Read 8 Done Done Sent false
5331 2435 Acked Read 5 Done Done Sent false
5332 2436 Acked Read 9 Done Done Sent false
5333 2437 Acked Write 7 Done Done Sent false
5334 2438 Acked Flush 0 Done Done Sent false
5335 2439 Acked Read 1 Done Done Sent false
5336 2440 Acked Write 8 Done Done Sent false
5337 2441 Acked Write 1 Done Done Sent false
5338 2442 Acked Write 1 Done Done Sent false
5339 2443 Acked Read 8 Done Done Sent false
5340 2444 Acked Write 8 Done Done Sent false
5341 2445 Acked Read 6 Done Done Sent false
5342 2446 Acked Flush 0 Done Done Sent false
5343 2447 Acked Write 2 Done Done Sent false
5344 2448 Acked Read 6 Done Done Sent false
5345 2449 Acked Read 8 Done Done Sent false
5346 2450 Acked Write 4 Done Done Sent false
5347 2451 Acked Read 9 Done Done Sent false
5348 2452 Acked Read 2 Done Done Sent false
5349 2453 Acked Read 10 Done Done Sent false
5350 2454 Acked Write 1 Done Done Sent false
5351 2455 Acked Read 6 Done Done Sent false
5352 2456 Acked Write 1 Done Done Sent false
5353 2457 Acked Read 7 Done Done Sent false
5354 2458 Acked Flush 0 Done Done Sent false
5355 2459 Acked Read 2 Done Done Sent false
5356 2460 Acked Read 2 Done Done Sent false
5357 2461 Acked Write 5 Done Done Sent false
5358 2462 Acked Read 6 Done Done Sent false
5359 2463 Acked Read 9 Done Done Sent false
5360 2464 Acked Write 1 Done Done Sent false
5361 2465 Acked Flush 0 Done Done Sent false
5362 2466 Acked Read 8 Done Done Sent false
5363 2467 Acked Read 4 Done Done Sent false
5364 2468 Acked Read 7 Done Done Sent false
5365 2469 Acked Read 4 Done Done Sent false
5366 2470 Acked Read 4 Done Done Sent false
5367 2471 Acked Write 8 Done Done Sent false
5368 2472 Acked Read 6 Done Done Sent false
5369 2473 Acked Write 7 Done Done Sent false
5370 2474 Acked Write 7 Done Done Sent false
5371 2475 Acked Flush 0 Done Done Sent false
5372 2476 Acked Write 8 Done Done Sent false
5373 2477 Acked Write 9 Done Done Sent false
5374 2478 Acked Read 6 Done Done Sent false
5375 2479 Acked Write 5 Done Done Sent false
5376 2480 Acked Read 8 Done Done Sent false
5377 2481 Acked Read 2 Done Done Sent false
5378 2482 Acked Write 6 Done Done Sent false
5379 2483 Acked Read 4 Done Done Sent false
5380 2484 Acked Write 6 Done Done Sent false
5381 2485 Acked Write 6 Done Done Sent false
5382 2486 Acked Read 7 Done Done Sent false
5383 2487 Acked Write 9 Done Done Sent false
5384 2488 Acked Write 4 Done Done Sent false
5385 2489 Acked Read 1 Done Done Sent false
5386 2490 Acked Read 2 Done Done Sent false
5387 2491 Acked Read 1 Done Done Sent false
5388 2492 Acked Flush 0 Done Done Sent false
5389 2493 Acked Read 6 Done Done Sent false
5390 2494 Acked Read 7 Done Done Sent false
5391 2495 Acked Write 9 Done Done Sent false
5392 2496 Acked Write 10 Done Done Sent false
5393 2497 Acked Flush 0 Done Done Sent false
5394 2498 Acked Read 6 Done Done Sent false
5395 2499 Acked Read 9 Done Done Sent false
5396 2500 Acked Flush 0 Done Done Sent false
5397 2501 Acked Write 7 Done Done Sent false
5398 2502 Acked Read 9 Done Done Sent false
5399 2503 Acked Write 7 Done Done Sent false
5400 2504 Acked Write 7 Done Done Sent false
5401 2505 Acked Flush 0 Done Done Sent false
5402 2506 Acked Read 3 Done Done Sent false
5403 2507 Acked Read 4 Done Done Sent false
5404 2508 Acked Flush 0 Done Done Sent false
5405 STATES DS:0 DS:1 DS:2 TOTAL
5406 Sent 0 0 301 301
5407 Done 301 301 0 602
5408 Skipped 0 0 0 0
5409 Error 0 0 0 0
5410 Last Flush: Some(JobId(2508)) Some(JobId(2508)) Some(JobId(2207))
5411 Downstairs last five completed: 2207 2206 2205 2204 2203
5412 Upstairs last five completed: 2508 2507 2506 2505 2504
5413 CLIENT: Up:0 ds:301 act:2
54142025-03-17T22:17:24.936ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
54152025-03-17T22:17:25.704ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
5416 [8830] got start action
5417 [8830] I am going to restart
5418 Starting downstairs at port 8830
5419 Make output file at "/tmp/dsc/downstairs-8830.txt"
5420 [8830][2] reports Starting
5421 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1211)
5422 [8830][2] reports Running
54232025-03-17T22:17:26.706ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
54242025-03-17T22:17:27.709ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
5425 ----------------------------------------------------------------
5426 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5427 States: Active Active Offline
5428 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
5429 2208 Acked Write 3 Done Done Sent false
5430 2209 Acked Read 4 Done Done Sent false
5431 2210 Acked Read 3 Done Done Sent false
5432 2211 Acked Flush 0 Done Done Sent false
5433 2212 Acked Write 4 Done Done Sent false
5434 2213 Acked Read 9 Done Done Sent false
5435 2214 Acked Write 8 Done Done Sent false
5436 2215 Acked Read 1 Done Done Sent false
5437 2216 Acked Write 9 Done Done Sent false
5438 2217 Acked Read 2 Done Done Sent false
5439 2218 Acked Read 10 Done Done Sent false
5440 2219 Acked Read 2 Done Done Sent false
5441 2220 Acked Read 5 Done Done Sent false
5442 2221 Acked Write 3 Done Done Sent false
5443 2222 Acked Read 4 Done Done Sent false
5444 2223 Acked Flush 0 Done Done Sent false
5445 2224 Acked Flush 0 Done Done Sent false
5446 2225 Acked Read 2 Done Done Sent false
5447 2226 Acked Read 1 Done Done Sent false
5448 2227 Acked Write 9 Done Done Sent false
5449 2228 Acked Write 6 Done Done Sent false
5450 2229 Acked Flush 0 Done Done Sent false
5451 2230 Acked Read 2 Done Done Sent false
5452 2231 Acked Read 9 Done Done Sent false
5453 2232 Acked Read 8 Done Done Sent false
5454 2233 Acked Write 3 Done Done Sent false
5455 2234 Acked Write 1 Done Done Sent false
5456 2235 Acked Write 5 Done Done Sent false
5457 2236 Acked Read 8 Done Done Sent false
5458 2237 Acked Write 7 Done Done Sent false
5459 2238 Acked Read 5 Done Done Sent false
5460 2239 Acked Read 2 Done Done Sent false
5461 2240 Acked Write 1 Done Done Sent false
5462 2241 Acked Read 6 Done Done Sent false
5463 2242 Acked Read 9 Done Done Sent false
5464 2243 Acked Write 7 Done Done Sent false
5465 2244 Acked Write 6 Done Done Sent false
5466 2245 Acked Write 10 Done Done Sent false
5467 2246 Acked Write 6 Done Done Sent false
5468 2247 Acked Read 1 Done Done Sent false
5469 2248 Acked Read 9 Done Done Sent false
5470 2249 Acked Read 1 Done Done Sent false
5471 2250 Acked Write 5 Done Done Sent false
5472 2251 Acked Write 10 Done Done Sent false
5473 2252 Acked Read 10 Done Done Sent false
5474 2253 Acked Write 2 Done Done Sent false
5475 2254 Acked Read 5 Done Done Sent false
5476 2255 Acked Flush 0 Done Done Sent false
5477 2256 Acked Read 3 Done Done Sent false
5478 2257 Acked Read 1 Done Done Sent false
5479 2258 Acked Write 1 Done Done Sent false
5480 2259 Acked Write 9 Done Done Sent false
5481 2260 Acked Write 10 Done Done Sent false
5482 2261 Acked Read 1 Done Done Sent false
5483 2262 Acked Flush 0 Done Done Sent false
5484 2263 Acked Read 9 Done Done Sent false
5485 2264 Acked Write 4 Done Done Sent false
5486 2265 Acked Write 8 Done Done Sent false
5487 2266 Acked Write 6 Done Done Sent false
5488 2267 Acked Read 8 Done Done Sent false
5489 2268 Acked Read 4 Done Done Sent false
5490 2269 Acked Write 5 Done Done Sent false
5491 2270 Acked Read 2 Done Done Sent false
5492 2271 Acked Read 2 Done Done Sent false
5493 2272 Acked Write 10 Done Done Sent false
5494 2273 Acked Write 5 Done Done Sent false
5495 2274 Acked Write 7 Done Done Sent false
5496 2275 Acked Read 1 Done Done Sent false
5497 2276 Acked Read 10 Done Done Sent false
5498 2277 Acked Write 8 Done Done Sent false
5499 2278 Acked Read 3 Done Done Sent false
5500 2279 Acked Write 1 Done Done Sent false
5501 2280 Acked Read 7 Done Done Sent false
5502 2281 Acked Read 2 Done Done Sent false
5503 2282 Acked Read 7 Done Done Sent false
5504 2283 Acked Write 10 Done Done Sent false
5505 2284 Acked Read 6 Done Done Sent false
5506 2285 Acked Read 8 Done Done Sent false
5507 2286 Acked Write 9 Done Done Sent false
5508 2287 Acked Read 10 Done Done Sent false
5509 2288 Acked Read 5 Done Done Sent false
5510 2289 Acked Read 3 Done Done Sent false
5511 2290 Acked Read 3 Done Done Sent false
5512 2291 Acked Write 7 Done Done Sent false
5513 2292 Acked Write 9 Done Done Sent false
5514 2293 Acked Read 8 Done Done Sent false
5515 2294 Acked Write 1 Done Done Sent false
5516 2295 Acked Flush 0 Done Done Sent false
5517 2296 Acked Write 10 Done Done Sent false
5518 2297 Acked Read 6 Done Done Sent false
5519 2298 Acked Read 4 Done Done Sent false
5520 2299 Acked Read 9 Done Done Sent false
5521 2300 Acked Read 7 Done Done Sent false
5522 2301 Acked Write 9 Done Done Sent false
5523 2302 Acked Write 10 Done Done Sent false
5524 2303 Acked Read 9 Done Done Sent false
5525 2304 Acked Flush 0 Done Done Sent false
5526 2305 Acked Write 9 Done Done Sent false
5527 2306 Acked Read 4 Done Done Sent false
5528 2307 Acked Read 3 Done Done Sent false
5529 2308 Acked Read 8 Done Done Sent false
5530 2309 Acked Read 1 Done Done Sent false
5531 2310 Acked Read 10 Done Done Sent false
5532 2311 Acked Write 7 Done Done Sent false
5533 2312 Acked Write 2 Done Done Sent false
5534 2313 Acked Read 9 Done Done Sent false
5535 2314 Acked Read 7 Done Done Sent false
5536 2315 Acked Write 6 Done Done Sent false
5537 2316 Acked Read 9 Done Done Sent false
5538 2317 Acked Read 4 Done Done Sent false
5539 2318 Acked Write 3 Done Done Sent false
5540 2319 Acked Read 5 Done Done Sent false
5541 2320 Acked Flush 0 Done Done Sent false
5542 2321 Acked Read 2 Done Done Sent false
5543 2322 Acked Read 1 Done Done Sent false
5544 2323 Acked Read 6 Done Done Sent false
5545 2324 Acked Read 6 Done Done Sent false
5546 2325 Acked Read 1 Done Done Sent false
5547 2326 Acked Read 10 Done Done Sent false
5548 2327 Acked Write 10 Done Done Sent false
5549 2328 Acked Write 2 Done Done Sent false
5550 2329 Acked Read 2 Done Done Sent false
5551 2330 Acked Read 10 Done Done Sent false
5552 2331 Acked Read 6 Done Done Sent false
5553 2332 Acked Read 3 Done Done Sent false
5554 2333 Acked Write 6 Done Done Sent false
5555 2334 Acked Read 4 Done Done Sent false
5556 2335 Acked Read 8 Done Done Sent false
5557 2336 Acked Write 9 Done Done Sent false
5558 2337 Acked Write 10 Done Done Sent false
5559 2338 Acked Read 6 Done Done Sent false
5560 2339 Acked Read 2 Done Done Sent false
5561 2340 Acked Write 5 Done Done Sent false
5562 2341 Acked Read 7 Done Done Sent false
5563 2342 Acked Read 5 Done Done Sent false
5564 2343 Acked Read 9 Done Done Sent false
5565 2344 Acked Read 7 Done Done Sent false
5566 2345 Acked Write 9 Done Done Sent false
5567 2346 Acked Write 2 Done Done Sent false
5568 2347 Acked Write 3 Done Done Sent false
5569 2348 Acked Flush 0 Done Done Sent false
5570 2349 Acked Write 3 Done Done Sent false
5571 2350 Acked Write 6 Done Done Sent false
5572 2351 Acked Write 2 Done Done Sent false
5573 2352 Acked Read 2 Done Done Sent false
5574 2353 Acked Read 3 Done Done Sent false
5575 2354 Acked Read 10 Done Done Sent false
5576 2355 Acked Read 8 Done Done Sent false
5577 2356 Acked Read 2 Done Done Sent false
5578 2357 Acked Flush 0 Done Done Sent false
5579 2358 Acked Flush 0 Done Done Sent false
5580 2359 Acked Flush 0 Done Done Sent false
5581 2360 Acked Read 5 Done Done Sent false
5582 2361 Acked Read 9 Done Done Sent false
5583 2362 Acked Read 2 Done Done Sent false
5584 2363 Acked Read 2 Done Done Sent false
5585 2364 Acked Read 5 Done Done Sent false
5586 2365 Acked Read 5 Done Done Sent false
5587 2366 Acked Read 4 Done Done Sent false
5588 2367 Acked Read 10 Done Done Sent false
5589 2368 Acked Flush 0 Done Done Sent false
5590 2369 Acked Read 6 Done Done Sent false
5591 2370 Acked Read 6 Done Done Sent false
5592 2371 Acked Read 5 Done Done Sent false
5593 2372 Acked Read 10 Done Done Sent false
5594 2373 Acked Read 4 Done Done Sent false
5595 2374 Acked Flush 0 Done Done Sent false
5596 2375 Acked Read 10 Done Done Sent false
5597 2376 Acked Read 7 Done Done Sent false
5598 2377 Acked Read 5 Done Done Sent false
5599 2378 Acked Write 8 Done Done Sent false
5600 2379 Acked Write 7 Done Done Sent false
5601 2380 Acked Read 9 Done Done Sent false
5602 2381 Acked Write 10 Done Done Sent false
5603 2382 Acked Read 2 Done Done Sent false
5604 2383 Acked Read 7 Done Done Sent false
5605 2384 Acked Write 10 Done Done Sent false
5606 2385 Acked Read 9 Done Done Sent false
5607 2386 Acked Write 9 Done Done Sent false
5608 2387 Acked Flush 0 Done Done Sent false
5609 2388 Acked Write 4 Done Done Sent false
5610 2389 Acked Write 8 Done Done Sent false
5611 2390 Acked Read 8 Done Done Sent false
5612 2391 Acked Write 8 Done Done Sent false
5613 2392 Acked Write 10 Done Done Sent false
5614 2393 Acked Write 5 Done Done Sent false
5615 2394 Acked Write 6 Done Done Sent false
5616 2395 Acked Read 5 Done Done Sent false
5617 2396 Acked Read 4 Done Done Sent false
5618 2397 Acked Read 3 Done Done Sent false
5619 2398 Acked Read 4 Done Done Sent false
5620 2399 Acked Write 6 Done Done Sent false
5621 2400 Acked Write 6 Done Done Sent false
5622 2401 Acked Write 9 Done Done Sent false
5623 2402 Acked Write 4 Done Done Sent false
5624 2403 Acked Write 9 Done Done Sent false
5625 2404 Acked Write 1 Done Done Sent false
5626 2405 Acked Read 5 Done Done Sent false
5627 2406 Acked Flush 0 Done Done Sent false
5628 2407 Acked Read 8 Done Done Sent false
5629 2408 Acked Read 10 Done Done Sent false
5630 2409 Acked Read 9 Done Done Sent false
5631 2410 Acked Read 5 Done Done Sent false
5632 2411 Acked Write 2 Done Done Sent false
5633 2412 Acked Read 8 Done Done Sent false
5634 2413 Acked Write 5 Done Done Sent false
5635 2414 Acked Read 1 Done Done Sent false
5636 2415 Acked Read 10 Done Done Sent false
5637 2416 Acked Read 7 Done Done Sent false
5638 2417 Acked Write 9 Done Done Sent false
5639 2418 Acked Write 3 Done Done Sent false
5640 2419 Acked Read 1 Done Done Sent false
5641 2420 Acked Read 3 Done Done Sent false
5642 2421 Acked Read 6 Done Done Sent false
5643 2422 Acked Read 4 Done Done Sent false
5644 2423 Acked Flush 0 Done Done Sent false
5645 2424 Acked Write 8 Done Done Sent false
5646 2425 Acked Read 10 Done Done Sent false
5647 2426 Acked Write 2 Done Done Sent false
5648 2427 Acked Read 2 Done Done Sent false
5649 2428 Acked Write 1 Done Done Sent false
5650 2429 Acked Write 1 Done Done Sent false
5651 2430 Acked Write 1 Done Done Sent false
5652 2431 Acked Read 10 Done Done Sent false
5653 2432 Acked Write 4 Done Done Sent false
5654 2433 Acked Read 1 Done Done Sent false
5655 2434 Acked Read 8 Done Done Sent false
5656 2435 Acked Read 5 Done Done Sent false
5657 2436 Acked Read 9 Done Done Sent false
5658 2437 Acked Write 7 Done Done Sent false
5659 2438 Acked Flush 0 Done Done Sent false
5660 2439 Acked Read 1 Done Done Sent false
5661 2440 Acked Write 8 Done Done Sent false
5662 2441 Acked Write 1 Done Done Sent false
5663 2442 Acked Write 1 Done Done Sent false
5664 2443 Acked Read 8 Done Done Sent false
5665 2444 Acked Write 8 Done Done Sent false
5666 2445 Acked Read 6 Done Done Sent false
5667 2446 Acked Flush 0 Done Done Sent false
5668 2447 Acked Write 2 Done Done Sent false
5669 2448 Acked Read 6 Done Done Sent false
5670 2449 Acked Read 8 Done Done Sent false
5671 2450 Acked Write 4 Done Done Sent false
5672 2451 Acked Read 9 Done Done Sent false
5673 2452 Acked Read 2 Done Done Sent false
5674 2453 Acked Read 10 Done Done Sent false
5675 2454 Acked Write 1 Done Done Sent false
5676 2455 Acked Read 6 Done Done Sent false
5677 2456 Acked Write 1 Done Done Sent false
5678 2457 Acked Read 7 Done Done Sent false
5679 2458 Acked Flush 0 Done Done Sent false
5680 2459 Acked Read 2 Done Done Sent false
5681 2460 Acked Read 2 Done Done Sent false
5682 2461 Acked Write 5 Done Done Sent false
5683 2462 Acked Read 6 Done Done Sent false
5684 2463 Acked Read 9 Done Done Sent false
5685 2464 Acked Write 1 Done Done Sent false
5686 2465 Acked Flush 0 Done Done Sent false
5687 2466 Acked Read 8 Done Done Sent false
5688 2467 Acked Read 4 Done Done Sent false
5689 2468 Acked Read 7 Done Done Sent false
5690 2469 Acked Read 4 Done Done Sent false
5691 2470 Acked Read 4 Done Done Sent false
5692 2471 Acked Write 8 Done Done Sent false
5693 2472 Acked Read 6 Done Done Sent false
5694 2473 Acked Write 7 Done Done Sent false
5695 2474 Acked Write 7 Done Done Sent false
5696 2475 Acked Flush 0 Done Done Sent false
5697 2476 Acked Write 8 Done Done Sent false
5698 2477 Acked Write 9 Done Done Sent false
5699 2478 Acked Read 6 Done Done Sent false
5700 2479 Acked Write 5 Done Done Sent false
5701 2480 Acked Read 8 Done Done Sent false
5702 2481 Acked Read 2 Done Done Sent false
5703 2482 Acked Write 6 Done Done Sent false
5704 2483 Acked Read 4 Done Done Sent false
5705 2484 Acked Write 6 Done Done Sent false
5706 2485 Acked Write 6 Done Done Sent false
5707 2486 Acked Read 7 Done Done Sent false
5708 2487 Acked Write 9 Done Done Sent false
5709 2488 Acked Write 4 Done Done Sent false
5710 2489 Acked Read 1 Done Done Sent false
5711 2490 Acked Read 2 Done Done Sent false
5712 2491 Acked Read 1 Done Done Sent false
5713 2492 Acked Flush 0 Done Done Sent false
5714 2493 Acked Read 6 Done Done Sent false
5715 2494 Acked Read 7 Done Done Sent false
5716 2495 Acked Write 9 Done Done Sent false
5717 2496 Acked Write 10 Done Done Sent false
5718 2497 Acked Flush 0 Done Done Sent false
5719 2498 Acked Read 6 Done Done Sent false
5720 2499 Acked Read 9 Done Done Sent false
5721 2500 Acked Flush 0 Done Done Sent false
5722 2501 Acked Write 7 Done Done Sent false
5723 2502 Acked Read 9 Done Done Sent false
5724 2503 Acked Write 7 Done Done Sent false
5725 2504 Acked Write 7 Done Done Sent false
5726 2505 Acked Flush 0 Done Done Sent false
5727 2506 Acked Read 3 Done Done Sent false
5728 2507 Acked Read 4 Done Done Sent false
5729 2508 Acked Flush 0 Done Done Sent false
5730 STATES DS:0 DS:1 DS:2 TOTAL
5731 Sent 0 0 301 301
5732 Done 301 301 0 602
5733 Skipped 0 0 0 0
5734 Error 0 0 0 0
5735 Last Flush: Some(JobId(2508)) Some(JobId(2508)) Some(JobId(2207))
5736 Downstairs last five completed: 2207 2206 2205 2204 2203
5737 Upstairs last five completed: 2508 2507 2506 2505 2504
5738 CLIENT: Up:0 ds:301 act:2
57392025-03-17T22:17:29.164ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
57402025-03-17T22:17:29.712ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
57412025-03-17T22:17:30.179ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
57422025-03-17T22:17:30.179ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:58451) = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
57432025-03-17T22:17:30.180ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
57442025-03-17T22:17:30.180ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
57452025-03-17T22:17:30.180ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
57462025-03-17T22:17:30.180ZINFOcrucible: send last flush ID to this DS: Some(JobId(2207)) = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
57472025-03-17T22:17:30.180ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
57482025-03-17T22:17:30.180ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(2207)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
57492025-03-17T22:17:30.180ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(2207)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
5750 ----------------------------------------------------------------
5751 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
5752 Downstairs last five completed: 2508 2507 2506 2505 2504
5753 Upstairs last five completed: 2508 2507 2506 2505 2504
5754 CLIENT: Up:0 ds:0 act:3
5755 Replay: All jobs finished
5756 Main task has work to do, go find it
5757 got dsc Stop(1)
5758 stop 1
5759 Mar 17 22:17:32.439 INFO request completed, latency_us: 360, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 4bf1fd8c-394c-4e63-ac99-7f02be6f976c, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
5760 [8820] Got stop action so:false kr:false
57612025-03-17T22:17:32.440ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
57622025-03-17T22:17:32.440ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
5763 Mar 17 22:17:32.440 INFO request completed, latency_us: 132, response_code: 200, uri: /state/cid/1, {"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:17:32.440710957Z","hostname":"w-01JPK2S810SFAA9787XEG8GFX5","pid":1206,"session_id":"0f8310d7-099e-4ead-8484-91ddc162d307"}
5764 method: GET, req_id: 715e180d-e559-41fe-89b0-d035e74ac9ed, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
57652025-03-17T22:17:32.440ZINFOcrucible: client task is exiting = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
5766 [8820] Exited with: ExitStatus(unix_wait_status(9))
5767 [8820][1] reports Exit
5768 Mar 17 22:17:36.442 INFO request completed, latency_us: 207, response_code: 200, uri: /state/cid/1, method: GET, req_id: a65b8aaf-fcb4-4274-8a49-cf4bd31549a5, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
5769 001/300 Read block 175 len 24576
5770 002/300 Read block 2531 len 24576
5771 003/300 Write block 4730 len 28672 data: 4 3 4 3 3 3 3
5772 004/300 Write block 2973 len 4096 data: 2
5773 005/300 Write block 2437 len 24576 data: 2 2 2 3 3 3
5774 006/300 Write block 2607 len 36864 data: 3 3 3 3 3 3 3 3 4
5775 007/300 Read block 4376 len 24576
5776 008/300 Read block 3426 len 4096
5777 009/300 Read block 1269 len 20480
5778 010/300 Read block 2967 len 16384
5779 011/300 Write block 4593 len 8192 data: 2 2
5780 012/300 Read block 473 len 28672
5781 013/300 Write block 3840 len 16384 data: 2 2 2 2
5782 014/300 Write block 555 len 32768 data: 3 4 6 6 7 7 6 6
5783 015/300 Write block 601 len 40960 data: 3 3 3 3 3 3 3 3 3 3
5784 016/300 Write block 2115 len 40960 data: 3 3 4 4 4 3 3 3 3 2
5785 017/300 Write block 3828 len 12288 data: 3 3 3
5786 018/300 Write block 1772 len 28672 data: 2 2 2 2 2 2 2
5787 019/300 Write block 3559 len 32768 data: 2 2 2 2 2 2 2 2
5788 020/300 Write block 2243 len 32768 data: 2 3 3 4 4 4 4 3
5789 021/300 Write block 2320 len 28672 data: 2 2 2 3 3 3 3
5790 022/300 Write block 3330 len 28672 data: 3 3 2 2 2 2 2
5791 023/300 Read block 532 len 24576
5792 024/300 Read block 911 len 16384
5793 025/300 Write block 1095 len 32768 data: 2 2 2 2 2 2 2 3
5794 026/300 Write block 3749 len 8192 data: 2 2
5795 027/300 Write block 1456 len 16384 data: 2 2 2 3
5796 028/300 Write block 603 len 28672 data: 4 4 4 4 4 4 4
5797 029/300 Write block 4922 len 36864 data: 2 2 2 2 2 2 3 3 3
5798 030/300 Read block 3684 len 12288
5799 031/300 Flush
5800 032/300 Write block 4198 len 8192 data: 2 2
5801 033/300 Write block 2437 len 28672 data: 3 3 3 4 4 4 3
5802 034/300 Write block 3928 len 32768 data: 2 2 2 2 2 2 2 2
5803 035/300 Write block 4227 len 36864 data: 2 2 2 2 2 2 2 2 3
5804 036/300 Read block 4785 len 36864
5805 037/300 Read block 1937 len 20480
5806 038/300 Write block 4367 len 36864 data: 2 2 2 2 2 2 2 2 2
5807 039/300 Write block 3820 len 32768 data: 4 4 2 2 2 2 2 2
5808 040/300 Read block 2260 len 4096
5809 041/300 Read block 1320 len 16384
5810 042/300 Write block 2667 len 8192 data: 3 3
5811 043/300 Read block 3458 len 12288
5812 044/300 Write block 2686 len 12288 data: 2 2 2
5813 045/300 Flush
5814 046/300 Flush
5815 047/300 Write block 2200 len 16384 data: 4 5 5 5
5816 048/300 Read block 3393 len 24576
5817 049/300 Write block 115 len 40960 data: 2 2 2 2 2 2 2 2 2 3
5818 050/300 Read block 1632 len 12288
5819 051/300 Read block 2429 len 4096
5820 052/300 Flush
5821 053/300 Write block 3490 len 12288 data: 3 3 4
5822 054/300 Read block 4801 len 20480
5823 055/300 Write block 2245 len 16384 data: 4 5 5 5
5824 056/300 Write block 1842 len 32768 data: 2 2 2 3 3 3 3 3
5825 057/300 Flush
5826 058/300 Flush
5827 059/300 Write block 2434 len 28672 data: 2 2 2 4 4 4 5
5828 060/300 Read block 2105 len 28672
5829 061/300 Read block 2142 len 32768
5830 062/300 Write block 4980 len 4096 data: 2
5831 063/300 Write block 2587 len 28672 data: 3 3 3 3 3 3 3
5832 064/300 Write block 2937 len 32768 data: 3 3 2 2 2 2 2 2
5833 065/300 Read block 2422 len 36864
5834 066/300 Read block 3165 len 36864
5835 067/300 Write block 1669 len 40960 data: 5 5 4 4 4 4 3 3 3 2
5836 068/300 Flush
5837 069/300 Write block 3998 len 28672 data: 2 2 2 2 2 2 2
5838 070/300 Read block 2976 len 16384
5839 071/300 Write block 480 len 24576 data: 2 2 2 2 2 2
5840 072/300 Read block 2510 len 4096
5841 073/300 Write block 1815 len 8192 data: 3 3
5842 074/300 Write block 2845 len 16384 data: 4 4 3 2
5843 075/300 Read block 3933 len 4096
5844 076/300 Read block 4530 len 40960
5845 077/300 Read block 3437 len 24576
5846 078/300 Read block 1609 len 24576
5847 079/300 Write block 4416 len 24576 data: 2 2 2 2 2 2
5848 080/300 Write block 696 len 12288 data: 2 2 2
5849 081/300 Write block 4134 len 8192 data: 2 2
5850 082/300 Write block 4924 len 20480 data: 3 3 3 3 4
5851 083/300 Write block 2242 len 4096 data: 2
5852 084/300 Read block 4479 len 16384
5853 085/300 Read block 3220 len 4096
5854 086/300 Write block 1376 len 12288 data: 2 2 2
5855 087/300 Write block 3283 len 32768 data: 3 2 2 3 4 4 4 4
5856 088/300 Write block 535 len 40960 data: 2 2 2 2 2 2 3 3 3 3
5857 089/300 Write block 560 len 8192 data: 8 7
5858 090/300 Read block 4710 len 4096
5859 091/300 Write block 3173 len 32768 data: 2 2 2 2 2 2 2 2
5860 092/300 Write block 3422 len 36864 data: 2 2 2 2 2 2 2 2 2
5861 093/300 Read block 2153 len 40960
5862 094/300 Read block 4865 len 20480
5863 095/300 Read block 122 len 16384
5864 096/300 Write block 3747 len 4096 data: 2
5865 097/300 Read block 137 len 36864
5866 098/300 Write block 4751 len 24576 data: 2 2 2 2 2 3
5867 099/300 Write block 3231 len 20480 data: 3 3 3 3 3
5868 100/300 Read block 4246 len 24576
5869 101/300 Read block 4326 len 20480
5870 102/300 Flush
5871 103/300 Read block 671 len 20480
5872 104/300 Flush
5873 105/300 Write block 3139 len 12288 data: 4 4 3
5874 106/300 Read block 3708 len 4096
5875 107/300 Read block 615 len 16384
5876 108/300 Read block 290 len 12288
5877 109/300 Write block 210 len 36864 data: 2 2 2 2 2 2 2 2 2
5878 110/300 Read block 328 len 36864
5879 111/300 Read block 2947 len 32768
5880 112/300 Write block 1321 len 4096 data: 2
5881 113/300 Read block 2361 len 28672
5882 114/300 Read block 3308 len 24576
5883 115/300 Read block 2345 len 36864
5884 116/300 Read block 4602 len 8192
5885 117/300 Write block 1992 len 4096 data: 3
5886 118/300 Write block 2106 len 28672 data: 3 3 2 2 4 4 4
5887 119/300 Read block 3504 len 40960
5888 120/300 Read block 3714 len 20480
5889 121/300 Write block 1737 len 12288 data: 2 2 2
5890 122/300 Read block 3089 len 4096
5891 123/300 Write block 959 len 24576 data: 2 2 2 2 2 2
5892 124/300 Write block 686 len 20480 data: 3 3 2 2 2
5893 125/300 Write block 4876 len 36864 data: 2 2 2 2 2 3 3 3 3
5894 126/300 Read block 4047 len 36864
5895 127/300 Flush
5896 128/300 Write block 719 len 20480 data: 3 4 4 3 2
5897 129/300 Flush
5898 130/300 Write block 2951 len 40960 data: 3 3 3 3 2 3 3 3 3 3
5899 131/300 Write block 854 len 4096 data: 3
5900 132/300 Read block 2445 len 12288
5901 133/300 Flush
5902 134/300 Read block 285 len 12288
5903 135/300 Write block 3015 len 36864 data: 2 2 2 3 3 3 3 3 3
5904 136/300 Read block 4537 len 20480
5905 137/300 Read block 2736 len 8192
5906 138/300 Read block 81 len 8192
5907 139/300 Read block 2158 len 28672
5908 140/300 Read block 2680 len 40960
5909 141/300 Write block 2059 len 12288 data: 6 5 5
5910 142/300 Read block 1994 len 24576
5911 143/300 Write block 193 len 32768 data: 3 3 3 3 2 2 2 2
5912 144/300 Read block 1005 len 4096
5913 145/300 Read block 835 len 36864
5914 146/300 Read block 1109 len 24576
5915 147/300 Write block 4257 len 28672 data: 2 3 3 3 3 2 2
5916 148/300 Read block 2229 len 32768
5917 149/300 Read block 3347 len 20480
5918 150/300 Read block 1211 len 20480
5919 151/300 Flush
5920 152/300 Read block 2800 len 36864
5921 153/300 Flush
5922 154/300 Write block 4779 len 16384 data: 3 3 3 3
5923 155/300 Write block 2048 len 28672 data: 2 2 2 2 2 2 3
5924 156/300 Write block 2293 len 32768 data: 2 2 3 3 3 3 3 3
5925 157/300 Flush
5926 158/300 Write block 3233 len 40960 data: 4 4 4 3 3 3 3 2 2 2
5927 159/300 Read block 2313 len 36864
5928 160/300 Write block 1503 len 4096 data: 4
5929 161/300 Read block 3367 len 20480
5930 162/300 Flush
5931 163/300 Read block 4737 len 40960
5932 164/300 Read block 1181 len 12288
5933 165/300 Write block 880 len 40960 data: 2 2 2 2 2 2 2 3 3 3
5934 166/300 Read block 74 len 32768
5935 167/300 Read block 1647 len 32768
5936 168/300 Write block 703 len 40960 data: 4 3 3 3 3 3 2 2 2 2
5937 169/300 Read block 3846 len 4096
5938 170/300 Write block 2973 len 24576 data: 3 2 2 3 3 3
5939 171/300 Read block 625 len 8192
5940 172/300 Read block 4900 len 28672
5941 173/300 Read block 406 len 28672
5942 174/300 Read block 1176 len 36864
5943 175/300 Read block 3441 len 8192
5944 176/300 Write block 2656 len 16384 data: 2 2 2 2
5945 177/300 Read block 2076 len 28672
5946 178/300 Write block 3466 len 4096 data: 3
5947 179/300 Read block 973 len 32768
5948 180/300 Read block 4255 len 24576
5949 181/300 Flush
5950 182/300 Read block 815 len 24576
5951 183/300 Write block 670 len 8192 data: 2 2
5952 184/300 Read block 1847 len 4096
5953 185/300 Read block 154 len 8192
5954 186/300 Read block 2712 len 36864
5955 187/300 Write block 4873 len 28672 data: 2 2 2 3 3 3 3
5956 188/300 Write block 4630 len 8192 data: 2 3
5957 189/300 Write block 47 len 24576 data: 2 2 2 2 2 2
5958 190/300 Write block 2045 len 28672 data: 2 2 2 3 3 3 3
5959 191/300 Write block 1801 len 20480 data: 3 2 2 2 2
5960 192/300 Read block 1629 len 36864
5961 193/300 Flush
5962 194/300 Read block 97 len 16384
5963 195/300 Write block 3044 len 36864 data: 3 4 4 4 4 3 3 2 2
5964 196/300 Read block 2122 len 4096
5965 197/300 Write block 504 len 28672 data: 5 5 5 4 3 3 2
5966 198/300 Flush
5967 199/300 Write block 1185 len 4096 data: 2
5968 200/300 Write block 1481 len 40960 data: 3 3 3 3 3 2 2 2 3 2
5969 201/300 Read block 4591 len 28672
5970 202/300 Read block 1888 len 4096
5971 203/300 Write block 187 len 8192 data: 5 5
5972 204/300 Flush
5973 205/300 Write block 153 len 36864 data: 2 2 2 2 3 3 3 3 3
5974 206/300 Write block 71 len 36864 data: 2 2 2 3 4 5 4 4 3
5975 207/300 Read block 425 len 24576
5976 208/300 Read block 1618 len 12288
5977 209/300 Write block 2939 len 20480 data: 3 3 3 3 3
5978 210/300 Flush
5979 211/300 Read block 3297 len 36864
5980 212/300 Write block 1648 len 4096 data: 3
5981 213/300 Read block 3535 len 24576
5982 214/300 Write block 1144 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5983 215/300 Flush
5984 216/300 Write block 3428 len 40960 data: 3 3 3 2 2 2 2 2 2 2
5985 217/300 Read block 1232 len 8192
5986 218/300 Write block 160 len 40960 data: 4 4 3 3 3 4 3 3 3 3
5987 219/300 Flush
5988 220/300 Write block 1961 len 40960 data: 4 4 4 4 4 2 2 2 2 2
5989 221/300 Read block 745 len 8192
5990 222/300 Flush
5991 223/300 Write block 1752 len 4096 data: 4
5992 224/300 Write block 1254 len 24576 data: 2 2 2 2 2 2
5993 225/300 Read block 970 len 28672
5994 226/300 Flush
5995 227/300 Read block 2593 len 36864
5996 228/300 Read block 2517 len 32768
5997 229/300 Write block 4642 len 24576 data: 2 2 2 2 2 2
5998 230/300 Write block 1176 len 8192 data: 3 3
5999 231/300 Flush
6000 232/300 Write block 2791 len 12288 data: 2 2 2
6001 233/300 Read block 1493 len 32768
6002 234/300 Write block 335 len 16384 data: 5 5 4 3
6003 235/300 Read block 844 len 24576
6004 236/300 Read block 4565 len 24576
6005 237/300 Write block 1132 len 20480 data: 2 2 2 2 2
6006 238/300 Read block 4567 len 28672
6007 239/300 Write block 3662 len 12288 data: 3 3 3
6008 240/300 Read block 468 len 40960
6009 241/300 Read block 3306 len 36864
6010 242/300 Read block 2375 len 8192
6011 243/300 Read block 1266 len 20480
6012 244/300 Flush
6013 245/300 Read block 571 len 16384
6014 246/300 Read block 61 len 16384
6015 247/300 Write block 4931 len 8192 data: 2 2
6016 248/300 Write block 3948 len 24576 data: 2 2 2 2 2 2
6017 249/300 Write block 3593 len 32768 data: 5 3 3 3 3 2 2 2
6018 250/300 Read block 2446 len 20480
6019 251/300 Read block 2210 len 12288
6020 252/300 Flush
6021 253/300 Read block 4592 len 16384
6022 254/300 Read block 4350 len 4096
6023 255/300 Write block 2573 len 4096 data: 2
6024 256/300 Flush
6025 257/300 Read block 2609 len 4096
6026 258/300 Read block 3908 len 32768
6027 259/300 Read block 1550 len 36864
6028 260/300 Flush
6029 261/300 Flush
6030 262/300 Read block 2212 len 36864
6031 263/300 Write block 43 len 20480 data: 3 3 3 3 3
6032 264/300 Read block 2017 len 40960
6033 265/300 Read block 4043 len 28672
6034 266/300 Read block 2663 len 24576
6035 267/300 Write block 628 len 28672 data: 4 4 4 3 4 3 3
6036 268/300 Write block 1419 len 16384 data: 3 2 2 2
6037 269/300 Read block 1341 len 40960
6038 270/300 Flush
6039 271/300 Write block 4715 len 40960 data: 3 3 3 3 3 3 3 3 4 4
6040 272/300 Read block 2302 len 40960
6041 273/300 Read block 1272 len 12288
6042 274/300 Write block 1797 len 8192 data: 3 4
6043 275/300 Write block 3036 len 16384 data: 3 3 3 4
6044 276/300 Write block 1351 len 12288 data: 2 2 2
6045 277/300 Read block 3264 len 4096
6046 278/300 Read block 132 len 40960
6047 279/300 Read block 869 len 28672
6048 280/300 Flush
6049 281/300 Write block 3715 len 4096 data: 3
6050 282/300 Read block 4030 len 40960
6051 283/300 Write block 4568 len 12288 data: 3 3 2
6052 284/300 Write block 2421 len 28672 data: 2 2 2 2 2 2 2
6053 285/300 Read block 1334 len 36864
6054 286/300 Read block 653 len 8192
6055 287/300 Write block 1560 len 12288 data: 3 3 3
6056 288/300 Read block 4962 len 36864
6057 289/300 Read block 346 len 4096
6058 290/300 Read block 16 len 24576
6059 291/300 Flush
6060 292/300 Read block 4829 len 36864
6061 293/300 Read block 4646 len 24576
6062 294/300 Flush
6063 295/300 Read block 1482 len 32768
6064 296/300 Read block 4135 len 8192
6065 297/300 Write block 2032 len 24576 data: 2 2 2 2 3 3
6066 298/300 Read block 114 len 32768
6067 299/300 Read block 1386 len 4096
6068 300/300 Read block 3065 len 28672
6069 301/300 Write block 4271 len 32768 data: 3 3 3 2 2 2 2 2
6070 Main task has work to do, go find it
6071 got dsc Start(1)
6072 start 1
6073 Mar 17 22:17:36.683 INFO request completed, latency_us: 290, response_code: 204, uri: /start/cid/1, method: POST, req_id: 42ea32c9-13fe-4ce3-94de-c7d3677cfa59, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
6074 Replay: started 1, returned:Ok(())
6075 ----------------------------------------------------------------
6076 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6077 States: Active Offline Active
6078 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6079 2509 Acked Read 6 Done Sent Done false
6080 2510 Acked Read 6 Done Sent Done false
6081 2511 Acked Write 7 Done Sent Done false
6082 2512 Acked Write 1 Done Sent Done false
6083 2513 Acked Write 6 Done Sent Done false
6084 2514 Acked Write 9 Done Sent Done false
6085 2515 Acked Read 6 Done Sent Done false
6086 2516 Acked Read 1 Done Sent Done false
6087 2517 Acked Read 5 Done Sent Done false
6088 2518 Acked Read 4 Done Sent Done false
6089 2519 Acked Write 2 Done Sent Done false
6090 2520 Acked Read 7 Done Sent Done false
6091 2521 Acked Write 4 Done Sent Done false
6092 2522 Acked Write 8 Done Sent Done false
6093 2523 Acked Write 10 Done Sent Done false
6094 2524 Acked Write 10 Done Sent Done false
6095 2525 Acked Write 3 Done Sent Done false
6096 2526 Acked Write 7 Done Sent Done false
6097 2527 Acked Write 8 Done Sent Done false
6098 2528 Acked Write 8 Done Sent Done false
6099 2529 Acked Write 7 Done Sent Done false
6100 2530 Acked Write 7 Done Sent Done false
6101 2531 Acked Read 6 Done Sent Done false
6102 2532 Acked Read 4 Done Sent Done false
6103 2533 Acked Write 8 Done Sent Done false
6104 2534 Acked Write 2 Done Sent Done false
6105 2535 Acked Write 4 Done Sent Done false
6106 2536 Acked Write 7 Done Sent Done false
6107 2537 Acked Write 9 Done Sent Done false
6108 2538 Acked Read 3 Done Sent Done false
6109 2539 Acked Flush 0 Done Sent Done false
6110 2540 Acked Write 2 Done Sent Done false
6111 2541 Acked Write 7 Done Sent Done false
6112 2542 Acked Write 8 Done Sent Done false
6113 2543 Acked Write 9 Done Sent Done false
6114 2544 Acked Read 9 Done Sent Done false
6115 2545 Acked Read 5 Done Sent Done false
6116 2546 Acked Write 9 Done Sent Done false
6117 2547 Acked Write 8 Done Sent Done false
6118 2548 Acked Read 1 Done Sent Done false
6119 2549 Acked Read 4 Done Sent Done false
6120 2550 Acked Write 2 Done Sent Done false
6121 2551 Acked Read 3 Done Sent Done false
6122 2552 Acked Write 3 Done Sent Done false
6123 2553 Acked Flush 0 Done Sent Done false
6124 2554 Acked Flush 0 Done Sent Done false
6125 2555 Acked Write 4 Done Sent Done false
6126 2556 Acked Read 6 Done Sent Done false
6127 2557 Acked Write 10 Done Sent Done false
6128 2558 Acked Read 3 Done Sent Done false
6129 2559 Acked Read 1 Done Sent Done false
6130 2560 Acked Flush 0 Done Sent Done false
6131 2561 Acked Write 3 Done Sent Done false
6132 2562 Acked Read 5 Done Sent Done false
6133 2563 Acked Write 4 Done Sent Done false
6134 2564 Acked Write 8 Done Sent Done false
6135 2565 Acked Flush 0 Done Sent Done false
6136 2566 Acked Flush 0 Done Sent Done false
6137 2567 Acked Write 7 Done Sent Done false
6138 2568 Acked Read 7 Done Sent Done false
6139 2569 Acked Read 8 Done Sent Done false
6140 2570 Acked Write 1 Done Sent Done false
6141 2571 Acked Write 7 Done Sent Done false
6142 2572 Acked Write 8 Done Sent Done false
6143 2573 Acked Read 9 Done Sent Done false
6144 2574 Acked Read 9 Done Sent Done false
6145 2575 Acked Write 10 Done Sent Done false
6146 2576 Acked Flush 0 Done Sent Done false
6147 2577 Acked Write 7 Done Sent Done false
6148 2578 Acked Read 4 Done Sent Done false
6149 2579 Acked Write 6 Done Sent Done false
6150 2580 Acked Read 1 Done Sent Done false
6151 2581 Acked Write 2 Done Sent Done false
6152 2582 Acked Write 4 Done Sent Done false
6153 2583 Acked Read 1 Done Sent Done false
6154 2584 Acked Read 10 Done Sent Done false
6155 2585 Acked Read 6 Done Sent Done false
6156 2586 Acked Read 6 Done Sent Done false
6157 2587 Acked Write 6 Done Sent Done false
6158 2588 Acked Write 3 Done Sent Done false
6159 2589 Acked Write 2 Done Sent Done false
6160 2590 Acked Write 5 Done Sent Done false
6161 2591 Acked Write 1 Done Sent Done false
6162 2592 Acked Read 4 Done Sent Done false
6163 2593 Acked Read 1 Done Sent Done false
6164 2594 Acked Write 3 Done Sent Done false
6165 2595 Acked Write 8 Done Sent Done false
6166 2596 Acked Write 10 Done Sent Done false
6167 2597 Acked Write 2 Done Sent Done false
6168 2598 Acked Read 1 Done Sent Done false
6169 2599 Acked Write 8 Done Sent Done false
6170 2600 Acked Write 9 Done Sent Done false
6171 2601 Acked Read 10 Done Sent Done false
6172 2602 Acked Read 5 Done Sent Done false
6173 2603 Acked Read 4 Done Sent Done false
6174 2604 Acked Write 1 Done Sent Done false
6175 2605 Acked Read 9 Done Sent Done false
6176 2606 Acked Write 6 Done Sent Done false
6177 2607 Acked Write 5 Done Sent Done false
6178 2608 Acked Read 6 Done Sent Done false
6179 2609 Acked Read 5 Done Sent Done false
6180 2610 Acked Flush 0 Done Sent Done false
6181 2611 Acked Read 5 Done Sent Done false
6182 2612 Acked Flush 0 Done Sent Done false
6183 2613 Acked Write 3 Done Sent Done false
6184 2614 Acked Read 1 Done Sent Done false
6185 2615 Acked Read 4 Done Sent Done false
6186 2616 Acked Read 3 Done Sent Done false
6187 2617 Acked Write 9 Done Sent Done false
6188 2618 Acked Read 9 Done Sent Done false
6189 2619 Acked Read 8 Done Sent Done false
6190 2620 Acked Write 1 Done Sent Done false
6191 2621 Acked Read 7 Done Sent Done false
6192 2622 Acked Read 6 Done Sent Done false
6193 2623 Acked Read 9 Done Sent Done false
6194 2624 Acked Read 2 Done Sent Done false
6195 2625 Acked Write 1 Done Sent Done false
6196 2626 Acked Write 7 Done Sent Done false
6197 2627 Acked Read 10 Done Sent Done false
6198 2628 Acked Read 5 Done Sent Done false
6199 2629 Acked Write 3 Done Sent Done false
6200 2630 Acked Read 1 Done Sent Done false
6201 2631 Acked Write 6 Done Sent Done false
6202 2632 Acked Write 5 Done Sent Done false
6203 2633 Acked Write 9 Done Sent Done false
6204 2634 Acked Read 9 Done Sent Done false
6205 2635 Acked Flush 0 Done Sent Done false
6206 2636 Acked Write 5 Done Sent Done false
6207 2637 Acked Flush 0 Done Sent Done false
6208 2638 Acked Write 10 Done Sent Done false
6209 2639 Acked Write 1 Done Sent Done false
6210 2640 Acked Read 3 Done Sent Done false
6211 2641 Acked Flush 0 Done Sent Done false
6212 2642 Acked Read 3 Done Sent Done false
6213 2643 Acked Write 9 Done Sent Done false
6214 2644 Acked Read 5 Done Sent Done false
6215 2645 Acked Read 2 Done Sent Done false
6216 2646 Acked Read 2 Done Sent Done false
6217 2647 Acked Read 7 Done Sent Done false
6218 2648 Acked Read 10 Done Sent Done false
6219 2649 Acked Write 3 Done Sent Done false
6220 2650 Acked Read 6 Done Sent Done false
6221 2651 Acked Write 8 Done Sent Done false
6222 2652 Acked Read 1 Done Sent Done false
6223 2653 Acked Read 9 Done Sent Done false
6224 2654 Acked Read 6 Done Sent Done false
6225 2655 Acked Write 7 Done Sent Done false
6226 2656 Acked Read 8 Done Sent Done false
6227 2657 Acked Read 5 Done Sent Done false
6228 2658 Acked Read 5 Done Sent Done false
6229 2659 Acked Flush 0 Done Sent Done false
6230 2660 Acked Read 9 Done Sent Done false
6231 2661 Acked Flush 0 Done Sent Done false
6232 2662 Acked Write 4 Done Sent Done false
6233 2663 Acked Write 7 Done Sent Done false
6234 2664 Acked Write 8 Done Sent Done false
6235 2665 Acked Flush 0 Done Sent Done false
6236 2666 Acked Write 10 Done Sent Done false
6237 2667 Acked Read 9 Done Sent Done false
6238 2668 Acked Write 1 Done Sent Done false
6239 2669 Acked Read 5 Done Sent Done false
6240 2670 Acked Flush 0 Done Sent Done false
6241 2671 Acked Read 10 Done Sent Done false
6242 2672 Acked Read 3 Done Sent Done false
6243 2673 Acked Write 10 Done Sent Done false
6244 2674 Acked Read 8 Done Sent Done false
6245 2675 Acked Read 8 Done Sent Done false
6246 2676 Acked Write 10 Done Sent Done false
6247 2677 Acked Read 1 Done Sent Done false
6248 2678 Acked Write 6 Done Sent Done false
6249 2679 Acked Read 2 Done Sent Done false
6250 2680 Acked Read 7 Done Sent Done false
6251 2681 Acked Read 7 Done Sent Done false
6252 2682 Acked Read 9 Done Sent Done false
6253 2683 Acked Read 2 Done Sent Done false
6254 2684 Acked Write 4 Done Sent Done false
6255 2685 Acked Read 7 Done Sent Done false
6256 2686 Acked Write 1 Done Sent Done false
6257 2687 Acked Read 8 Done Sent Done false
6258 2688 Acked Read 6 Done Sent Done false
6259 2689 Acked Flush 0 Done Sent Done false
6260 2690 Acked Read 6 Done Sent Done false
6261 2691 Acked Write 2 Done Sent Done false
6262 2692 Acked Read 1 Done Sent Done false
6263 2693 Acked Read 2 Done Sent Done false
6264 2694 Acked Read 9 Done Sent Done false
6265 2695 Acked Write 7 Done Sent Done false
6266 2696 Acked Write 2 Done Sent Done false
6267 2697 Acked Write 6 Done Sent Done false
6268 2698 Acked Write 7 Done Sent Done false
6269 2699 Acked Write 5 Done Sent Done false
6270 2700 Acked Read 9 Done Sent Done false
6271 2701 Acked Flush 0 Done Sent Done false
6272 2702 Acked Read 4 Done Sent Done false
6273 2703 Acked Write 9 Done Sent Done false
6274 2704 Acked Read 1 Done Sent Done false
6275 2705 Acked Write 7 Done Sent Done false
6276 2706 Acked Flush 0 Done Sent Done false
6277 2707 Acked Write 1 Done Sent Done false
6278 2708 Acked Write 10 Done Sent Done false
6279 2709 Acked Read 7 Done Sent Done false
6280 2710 Acked Read 1 Done Sent Done false
6281 2711 Acked Write 2 Done Sent Done false
6282 2712 Acked Flush 0 Done Sent Done false
6283 2713 Acked Write 9 Done Sent Done false
6284 2714 Acked Write 9 Done Sent Done false
6285 2715 Acked Read 6 Done Sent Done false
6286 2716 Acked Read 3 Done Sent Done false
6287 2717 Acked Write 5 Done Sent Done false
6288 2718 Acked Flush 0 Done Sent Done false
6289 2719 Acked Read 9 Done Sent Done false
6290 2720 Acked Write 1 Done Sent Done false
6291 2721 Acked Read 6 Done Sent Done false
6292 2722 Acked Write 10 Done Sent Done false
6293 2723 Acked Flush 0 Done Sent Done false
6294 2724 Acked Write 10 Done Sent Done false
6295 2725 Acked Read 2 Done Sent Done false
6296 2726 Acked Write 10 Done Sent Done false
6297 2727 Acked Flush 0 Done Sent Done false
6298 2728 Acked Write 10 Done Sent Done false
6299 2729 Acked Read 2 Done Sent Done false
6300 2730 Acked Flush 0 Done Sent Done false
6301 2731 Acked Write 1 Done Sent Done false
6302 2732 Acked Write 6 Done Sent Done false
6303 2733 Acked Read 7 Done Sent Done false
6304 2734 Acked Flush 0 Done Sent Done false
6305 2735 Acked Read 9 Done Sent Done false
6306 2736 Acked Read 8 Done Sent Done false
6307 2737 Acked Write 6 Done Sent Done false
6308 2738 Acked Write 2 Done Sent Done false
6309 2739 Acked Flush 0 Done Sent Done false
6310 2740 Acked Write 3 Done Sent Done false
6311 2741 Acked Read 8 Done Sent Done false
6312 2742 Acked Write 4 Done Sent Done false
6313 2743 Acked Read 6 Done Sent Done false
6314 2744 Acked Read 6 Done Sent Done false
6315 2745 Acked Write 5 Done Sent Done false
6316 2746 Acked Read 7 Done Sent Done false
6317 2747 Acked Write 3 Done Sent Done false
6318 2748 Acked Read 10 Done Sent Done false
6319 2749 Acked Read 9 Done Sent Done false
6320 2750 Acked Read 2 Done Sent Done false
6321 2751 Acked Read 5 Done Sent Done false
6322 2752 Acked Flush 0 Done Sent Done false
6323 2753 Acked Read 4 Done Sent Done false
6324 2754 Acked Read 4 Done Sent Done false
6325 2755 Acked Write 2 Done Sent Done false
6326 2756 Acked Write 6 Done Sent Done false
6327 2757 Acked Write 8 Done Sent Done false
6328 2758 Acked Read 5 Done Sent Done false
6329 2759 Acked Read 3 Done Sent Done false
6330 2760 Acked Flush 0 Done Sent Done false
6331 2761 Acked Read 4 Done Sent Done false
6332 2762 Acked Read 1 Done Sent Done false
6333 2763 Acked Write 1 Done Sent Done false
6334 2764 Acked Flush 0 Done Sent Done false
6335 2765 Acked Read 1 Done Sent Done false
6336 2766 Acked Read 8 Done Sent Done false
6337 2767 Acked Read 9 Done Sent Done false
6338 2768 Acked Flush 0 Done Sent Done false
6339 2769 Acked Flush 0 Done Sent Done false
6340 2770 Acked Read 9 Done Sent Done false
6341 2771 Acked Write 5 Done Sent Done false
6342 2772 Acked Read 10 Done Sent Done false
6343 2773 Acked Read 7 Done Sent Done false
6344 2774 Acked Read 6 Done Sent Done false
6345 2775 Acked Write 7 Done Sent Done false
6346 2776 Acked Write 4 Done Sent Done false
6347 2777 Acked Read 10 Done Sent Done false
6348 2778 Acked Flush 0 Done Sent Done false
6349 2779 Acked Write 10 Done Sent Done false
6350 2780 Acked Read 10 Done Sent Done false
6351 2781 Acked Read 3 Done Sent Done false
6352 2782 Acked Write 2 Done Sent Done false
6353 2783 Acked Write 4 Done Sent Done false
6354 2784 Acked Write 3 Done Sent Done false
6355 2785 Acked Read 1 Done Sent Done false
6356 2786 Acked Read 10 Done Sent Done false
6357 2787 Acked Read 7 Done Sent Done false
6358 2788 Acked Flush 0 Done Sent Done false
6359 2789 Acked Write 1 Done Sent Done false
6360 2790 Acked Read 10 Done Sent Done false
6361 2791 Acked Write 3 Done Sent Done false
6362 2792 Acked Write 7 Done Sent Done false
6363 2793 Acked Read 9 Done Sent Done false
6364 2794 Acked Read 2 Done Sent Done false
6365 2795 Acked Write 3 Done Sent Done false
6366 2796 Acked Read 9 Done Sent Done false
6367 2797 Acked Read 1 Done Sent Done false
6368 2798 Acked Read 6 Done Sent Done false
6369 2799 Acked Flush 0 Done Sent Done false
6370 2800 Acked Read 9 Done Sent Done false
6371 2801 Acked Read 6 Done Sent Done false
6372 2802 Acked Flush 0 Done Sent Done false
6373 2803 Acked Read 8 Done Sent Done false
6374 2804 Acked Read 2 Done Sent Done false
6375 2805 Acked Write 6 Done Sent Done false
6376 2806 Acked Read 8 Done Sent Done false
6377 2807 Acked Read 1 Done Sent Done false
6378 2808 Acked Read 7 Done Sent Done false
6379 2809 Acked Write 8 Done Sent Done false
6380 STATES DS:0 DS:1 DS:2 TOTAL
6381 Sent 0 301 0 301
6382 Done 301 0 301 602
6383 Skipped 0 0 0 0
6384 Error 0 0 0 0
6385 Last Flush: Some(JobId(2802)) Some(JobId(2508)) Some(JobId(2802))
6386 Downstairs last five completed: 2508 2507 2506 2505 2504
6387 Upstairs last five completed: 2809 2808 2807 2806 2805
6388 CLIENT: Up:0 ds:301 act:2
63892025-03-17T22:17:37.185ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
63902025-03-17T22:17:37.723ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
6391 [8820] got start action
6392 [8820] I am going to restart
6393 Starting downstairs at port 8820
6394 Make output file at "/tmp/dsc/downstairs-8820.txt"
6395 [8820][1] reports Starting
6396 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1212)
6397 [8820][1] reports Running
63982025-03-17T22:17:38.725ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
63992025-03-17T22:17:39.726ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
6400 ----------------------------------------------------------------
6401 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6402 States: Active Offline Active
6403 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
6404 2509 Acked Read 6 Done Sent Done false
6405 2510 Acked Read 6 Done Sent Done false
6406 2511 Acked Write 7 Done Sent Done false
6407 2512 Acked Write 1 Done Sent Done false
6408 2513 Acked Write 6 Done Sent Done false
6409 2514 Acked Write 9 Done Sent Done false
6410 2515 Acked Read 6 Done Sent Done false
6411 2516 Acked Read 1 Done Sent Done false
6412 2517 Acked Read 5 Done Sent Done false
6413 2518 Acked Read 4 Done Sent Done false
6414 2519 Acked Write 2 Done Sent Done false
6415 2520 Acked Read 7 Done Sent Done false
6416 2521 Acked Write 4 Done Sent Done false
6417 2522 Acked Write 8 Done Sent Done false
6418 2523 Acked Write 10 Done Sent Done false
6419 2524 Acked Write 10 Done Sent Done false
6420 2525 Acked Write 3 Done Sent Done false
6421 2526 Acked Write 7 Done Sent Done false
6422 2527 Acked Write 8 Done Sent Done false
6423 2528 Acked Write 8 Done Sent Done false
6424 2529 Acked Write 7 Done Sent Done false
6425 2530 Acked Write 7 Done Sent Done false
6426 2531 Acked Read 6 Done Sent Done false
6427 2532 Acked Read 4 Done Sent Done false
6428 2533 Acked Write 8 Done Sent Done false
6429 2534 Acked Write 2 Done Sent Done false
6430 2535 Acked Write 4 Done Sent Done false
6431 2536 Acked Write 7 Done Sent Done false
6432 2537 Acked Write 9 Done Sent Done false
6433 2538 Acked Read 3 Done Sent Done false
6434 2539 Acked Flush 0 Done Sent Done false
6435 2540 Acked Write 2 Done Sent Done false
6436 2541 Acked Write 7 Done Sent Done false
6437 2542 Acked Write 8 Done Sent Done false
6438 2543 Acked Write 9 Done Sent Done false
6439 2544 Acked Read 9 Done Sent Done false
6440 2545 Acked Read 5 Done Sent Done false
6441 2546 Acked Write 9 Done Sent Done false
6442 2547 Acked Write 8 Done Sent Done false
6443 2548 Acked Read 1 Done Sent Done false
6444 2549 Acked Read 4 Done Sent Done false
6445 2550 Acked Write 2 Done Sent Done false
6446 2551 Acked Read 3 Done Sent Done false
6447 2552 Acked Write 3 Done Sent Done false
6448 2553 Acked Flush 0 Done Sent Done false
6449 2554 Acked Flush 0 Done Sent Done false
6450 2555 Acked Write 4 Done Sent Done false
6451 2556 Acked Read 6 Done Sent Done false
6452 2557 Acked Write 10 Done Sent Done false
6453 2558 Acked Read 3 Done Sent Done false
6454 2559 Acked Read 1 Done Sent Done false
6455 2560 Acked Flush 0 Done Sent Done false
6456 2561 Acked Write 3 Done Sent Done false
6457 2562 Acked Read 5 Done Sent Done false
6458 2563 Acked Write 4 Done Sent Done false
6459 2564 Acked Write 8 Done Sent Done false
6460 2565 Acked Flush 0 Done Sent Done false
6461 2566 Acked Flush 0 Done Sent Done false
6462 2567 Acked Write 7 Done Sent Done false
6463 2568 Acked Read 7 Done Sent Done false
6464 2569 Acked Read 8 Done Sent Done false
6465 2570 Acked Write 1 Done Sent Done false
6466 2571 Acked Write 7 Done Sent Done false
6467 2572 Acked Write 8 Done Sent Done false
6468 2573 Acked Read 9 Done Sent Done false
6469 2574 Acked Read 9 Done Sent Done false
6470 2575 Acked Write 10 Done Sent Done false
6471 2576 Acked Flush 0 Done Sent Done false
6472 2577 Acked Write 7 Done Sent Done false
6473 2578 Acked Read 4 Done Sent Done false
6474 2579 Acked Write 6 Done Sent Done false
6475 2580 Acked Read 1 Done Sent Done false
6476 2581 Acked Write 2 Done Sent Done false
6477 2582 Acked Write 4 Done Sent Done false
6478 2583 Acked Read 1 Done Sent Done false
6479 2584 Acked Read 10 Done Sent Done false
6480 2585 Acked Read 6 Done Sent Done false
6481 2586 Acked Read 6 Done Sent Done false
6482 2587 Acked Write 6 Done Sent Done false
6483 2588 Acked Write 3 Done Sent Done false
6484 2589 Acked Write 2 Done Sent Done false
6485 2590 Acked Write 5 Done Sent Done false
6486 2591 Acked Write 1 Done Sent Done false
6487 2592 Acked Read 4 Done Sent Done false
6488 2593 Acked Read 1 Done Sent Done false
6489 2594 Acked Write 3 Done Sent Done false
6490 2595 Acked Write 8 Done Sent Done false
6491 2596 Acked Write 10 Done Sent Done false
6492 2597 Acked Write 2 Done Sent Done false
6493 2598 Acked Read 1 Done Sent Done false
6494 2599 Acked Write 8 Done Sent Done false
6495 2600 Acked Write 9 Done Sent Done false
6496 2601 Acked Read 10 Done Sent Done false
6497 2602 Acked Read 5 Done Sent Done false
6498 2603 Acked Read 4 Done Sent Done false
6499 2604 Acked Write 1 Done Sent Done false
6500 2605 Acked Read 9 Done Sent Done false
6501 2606 Acked Write 6 Done Sent Done false
6502 2607 Acked Write 5 Done Sent Done false
6503 2608 Acked Read 6 Done Sent Done false
6504 2609 Acked Read 5 Done Sent Done false
6505 2610 Acked Flush 0 Done Sent Done false
6506 2611 Acked Read 5 Done Sent Done false
6507 2612 Acked Flush 0 Done Sent Done false
6508 2613 Acked Write 3 Done Sent Done false
6509 2614 Acked Read 1 Done Sent Done false
6510 2615 Acked Read 4 Done Sent Done false
6511 2616 Acked Read 3 Done Sent Done false
6512 2617 Acked Write 9 Done Sent Done false
6513 2618 Acked Read 9 Done Sent Done false
6514 2619 Acked Read 8 Done Sent Done false
6515 2620 Acked Write 1 Done Sent Done false
6516 2621 Acked Read 7 Done Sent Done false
6517 2622 Acked Read 6 Done Sent Done false
6518 2623 Acked Read 9 Done Sent Done false
6519 2624 Acked Read 2 Done Sent Done false
6520 2625 Acked Write 1 Done Sent Done false
6521 2626 Acked Write 7 Done Sent Done false
6522 2627 Acked Read 10 Done Sent Done false
6523 2628 Acked Read 5 Done Sent Done false
6524 2629 Acked Write 3 Done Sent Done false
6525 2630 Acked Read 1 Done Sent Done false
6526 2631 Acked Write 6 Done Sent Done false
6527 2632 Acked Write 5 Done Sent Done false
6528 2633 Acked Write 9 Done Sent Done false
6529 2634 Acked Read 9 Done Sent Done false
6530 2635 Acked Flush 0 Done Sent Done false
6531 2636 Acked Write 5 Done Sent Done false
6532 2637 Acked Flush 0 Done Sent Done false
6533 2638 Acked Write 10 Done Sent Done false
6534 2639 Acked Write 1 Done Sent Done false
6535 2640 Acked Read 3 Done Sent Done false
6536 2641 Acked Flush 0 Done Sent Done false
6537 2642 Acked Read 3 Done Sent Done false
6538 2643 Acked Write 9 Done Sent Done false
6539 2644 Acked Read 5 Done Sent Done false
6540 2645 Acked Read 2 Done Sent Done false
6541 2646 Acked Read 2 Done Sent Done false
6542 2647 Acked Read 7 Done Sent Done false
6543 2648 Acked Read 10 Done Sent Done false
6544 2649 Acked Write 3 Done Sent Done false
6545 2650 Acked Read 6 Done Sent Done false
6546 2651 Acked Write 8 Done Sent Done false
6547 2652 Acked Read 1 Done Sent Done false
6548 2653 Acked Read 9 Done Sent Done false
6549 2654 Acked Read 6 Done Sent Done false
6550 2655 Acked Write 7 Done Sent Done false
6551 2656 Acked Read 8 Done Sent Done false
6552 2657 Acked Read 5 Done Sent Done false
6553 2658 Acked Read 5 Done Sent Done false
6554 2659 Acked Flush 0 Done Sent Done false
6555 2660 Acked Read 9 Done Sent Done false
6556 2661 Acked Flush 0 Done Sent Done false
6557 2662 Acked Write 4 Done Sent Done false
6558 2663 Acked Write 7 Done Sent Done false
6559 2664 Acked Write 8 Done Sent Done false
6560 2665 Acked Flush 0 Done Sent Done false
6561 2666 Acked Write 10 Done Sent Done false
6562 2667 Acked Read 9 Done Sent Done false
6563 2668 Acked Write 1 Done Sent Done false
6564 2669 Acked Read 5 Done Sent Done false
6565 2670 Acked Flush 0 Done Sent Done false
6566 2671 Acked Read 10 Done Sent Done false
6567 2672 Acked Read 3 Done Sent Done false
6568 2673 Acked Write 10 Done Sent Done false
6569 2674 Acked Read 8 Done Sent Done false
6570 2675 Acked Read 8 Done Sent Done false
6571 2676 Acked Write 10 Done Sent Done false
6572 2677 Acked Read 1 Done Sent Done false
6573 2678 Acked Write 6 Done Sent Done false
6574 2679 Acked Read 2 Done Sent Done false
6575 2680 Acked Read 7 Done Sent Done false
6576 2681 Acked Read 7 Done Sent Done false
6577 2682 Acked Read 9 Done Sent Done false
6578 2683 Acked Read 2 Done Sent Done false
6579 2684 Acked Write 4 Done Sent Done false
6580 2685 Acked Read 7 Done Sent Done false
6581 2686 Acked Write 1 Done Sent Done false
6582 2687 Acked Read 8 Done Sent Done false
6583 2688 Acked Read 6 Done Sent Done false
6584 2689 Acked Flush 0 Done Sent Done false
6585 2690 Acked Read 6 Done Sent Done false
6586 2691 Acked Write 2 Done Sent Done false
6587 2692 Acked Read 1 Done Sent Done false
6588 2693 Acked Read 2 Done Sent Done false
6589 2694 Acked Read 9 Done Sent Done false
6590 2695 Acked Write 7 Done Sent Done false
6591 2696 Acked Write 2 Done Sent Done false
6592 2697 Acked Write 6 Done Sent Done false
6593 2698 Acked Write 7 Done Sent Done false
6594 2699 Acked Write 5 Done Sent Done false
6595 2700 Acked Read 9 Done Sent Done false
6596 2701 Acked Flush 0 Done Sent Done false
6597 2702 Acked Read 4 Done Sent Done false
6598 2703 Acked Write 9 Done Sent Done false
6599 2704 Acked Read 1 Done Sent Done false
6600 2705 Acked Write 7 Done Sent Done false
6601 2706 Acked Flush 0 Done Sent Done false
6602 2707 Acked Write 1 Done Sent Done false
6603 2708 Acked Write 10 Done Sent Done false
6604 2709 Acked Read 7 Done Sent Done false
6605 2710 Acked Read 1 Done Sent Done false
6606 2711 Acked Write 2 Done Sent Done false
6607 2712 Acked Flush 0 Done Sent Done false
6608 2713 Acked Write 9 Done Sent Done false
6609 2714 Acked Write 9 Done Sent Done false
6610 2715 Acked Read 6 Done Sent Done false
6611 2716 Acked Read 3 Done Sent Done false
6612 2717 Acked Write 5 Done Sent Done false
6613 2718 Acked Flush 0 Done Sent Done false
6614 2719 Acked Read 9 Done Sent Done false
6615 2720 Acked Write 1 Done Sent Done false
6616 2721 Acked Read 6 Done Sent Done false
6617 2722 Acked Write 10 Done Sent Done false
6618 2723 Acked Flush 0 Done Sent Done false
6619 2724 Acked Write 10 Done Sent Done false
6620 2725 Acked Read 2 Done Sent Done false
6621 2726 Acked Write 10 Done Sent Done false
6622 2727 Acked Flush 0 Done Sent Done false
6623 2728 Acked Write 10 Done Sent Done false
6624 2729 Acked Read 2 Done Sent Done false
6625 2730 Acked Flush 0 Done Sent Done false
6626 2731 Acked Write 1 Done Sent Done false
6627 2732 Acked Write 6 Done Sent Done false
6628 2733 Acked Read 7 Done Sent Done false
6629 2734 Acked Flush 0 Done Sent Done false
6630 2735 Acked Read 9 Done Sent Done false
6631 2736 Acked Read 8 Done Sent Done false
6632 2737 Acked Write 6 Done Sent Done false
6633 2738 Acked Write 2 Done Sent Done false
6634 2739 Acked Flush 0 Done Sent Done false
6635 2740 Acked Write 3 Done Sent Done false
6636 2741 Acked Read 8 Done Sent Done false
6637 2742 Acked Write 4 Done Sent Done false
6638 2743 Acked Read 6 Done Sent Done false
6639 2744 Acked Read 6 Done Sent Done false
6640 2745 Acked Write 5 Done Sent Done false
6641 2746 Acked Read 7 Done Sent Done false
6642 2747 Acked Write 3 Done Sent Done false
6643 2748 Acked Read 10 Done Sent Done false
6644 2749 Acked Read 9 Done Sent Done false
6645 2750 Acked Read 2 Done Sent Done false
6646 2751 Acked Read 5 Done Sent Done false
6647 2752 Acked Flush 0 Done Sent Done false
6648 2753 Acked Read 4 Done Sent Done false
6649 2754 Acked Read 4 Done Sent Done false
6650 2755 Acked Write 2 Done Sent Done false
6651 2756 Acked Write 6 Done Sent Done false
6652 2757 Acked Write 8 Done Sent Done false
6653 2758 Acked Read 5 Done Sent Done false
6654 2759 Acked Read 3 Done Sent Done false
6655 2760 Acked Flush 0 Done Sent Done false
6656 2761 Acked Read 4 Done Sent Done false
6657 2762 Acked Read 1 Done Sent Done false
6658 2763 Acked Write 1 Done Sent Done false
6659 2764 Acked Flush 0 Done Sent Done false
6660 2765 Acked Read 1 Done Sent Done false
6661 2766 Acked Read 8 Done Sent Done false
6662 2767 Acked Read 9 Done Sent Done false
6663 2768 Acked Flush 0 Done Sent Done false
6664 2769 Acked Flush 0 Done Sent Done false
6665 2770 Acked Read 9 Done Sent Done false
6666 2771 Acked Write 5 Done Sent Done false
6667 2772 Acked Read 10 Done Sent Done false
6668 2773 Acked Read 7 Done Sent Done false
6669 2774 Acked Read 6 Done Sent Done false
6670 2775 Acked Write 7 Done Sent Done false
6671 2776 Acked Write 4 Done Sent Done false
6672 2777 Acked Read 10 Done Sent Done false
6673 2778 Acked Flush 0 Done Sent Done false
6674 2779 Acked Write 10 Done Sent Done false
6675 2780 Acked Read 10 Done Sent Done false
6676 2781 Acked Read 3 Done Sent Done false
6677 2782 Acked Write 2 Done Sent Done false
6678 2783 Acked Write 4 Done Sent Done false
6679 2784 Acked Write 3 Done Sent Done false
6680 2785 Acked Read 1 Done Sent Done false
6681 2786 Acked Read 10 Done Sent Done false
6682 2787 Acked Read 7 Done Sent Done false
6683 2788 Acked Flush 0 Done Sent Done false
6684 2789 Acked Write 1 Done Sent Done false
6685 2790 Acked Read 10 Done Sent Done false
6686 2791 Acked Write 3 Done Sent Done false
6687 2792 Acked Write 7 Done Sent Done false
6688 2793 Acked Read 9 Done Sent Done false
6689 2794 Acked Read 2 Done Sent Done false
6690 2795 Acked Write 3 Done Sent Done false
6691 2796 Acked Read 9 Done Sent Done false
6692 2797 Acked Read 1 Done Sent Done false
6693 2798 Acked Read 6 Done Sent Done false
6694 2799 Acked Flush 0 Done Sent Done false
6695 2800 Acked Read 9 Done Sent Done false
6696 2801 Acked Read 6 Done Sent Done false
6697 2802 Acked Flush 0 Done Sent Done false
6698 2803 Acked Read 8 Done Sent Done false
6699 2804 Acked Read 2 Done Sent Done false
6700 2805 Acked Write 6 Done Sent Done false
6701 2806 Acked Read 8 Done Sent Done false
6702 2807 Acked Read 1 Done Sent Done false
6703 2808 Acked Read 7 Done Sent Done false
6704 2809 Acked Write 8 Done Sent Done false
6705 STATES DS:0 DS:1 DS:2 TOTAL
6706 Sent 0 301 0 301
6707 Done 301 0 301 602
6708 Skipped 0 0 0 0
6709 Error 0 0 0 0
6710 Last Flush: Some(JobId(2802)) Some(JobId(2508)) Some(JobId(2802))
6711 Downstairs last five completed: 2508 2507 2506 2505 2504
6712 Upstairs last five completed: 2809 2808 2807 2806 2805
6713 CLIENT: Up:0 ds:301 act:2
67142025-03-17T22:17:41.188ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
67152025-03-17T22:17:41.729ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
67162025-03-17T22:17:42.442ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
67172025-03-17T22:17:42.442ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:43279) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
67182025-03-17T22:17:42.443ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
67192025-03-17T22:17:42.444ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
67202025-03-17T22:17:42.444ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
67212025-03-17T22:17:42.444ZINFOcrucible: send last flush ID to this DS: Some(JobId(2508)) = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
67222025-03-17T22:17:42.444ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
67232025-03-17T22:17:42.444ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(2508)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
67242025-03-17T22:17:42.444ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(2508)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
6725 ----------------------------------------------------------------
6726 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
6727 Downstairs last five completed: 2810 2809 2808 2807 2806
6728 Upstairs last five completed: 2810 2809 2808 2807 2806
6729 CLIENT: Up:0 ds:0 act:3
6730 Replay: All jobs finished
6731 Main task has work to do, go find it
6732 got dsc Stop(0)
6733 stop 0
6734 Mar 17 22:17:44.689 INFO request completed, latency_us: 354, response_code: 204, uri: /stop/cid/0, method: POST, req_id: d83887ec-41ec-4a49-831f-aa761d5fbd8d, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
6735 [8810] Got stop action so:false kr:false
6736 Mar 17 22:17:44.689 INFO request completed, latency_us: 162, response_code: 200, uri: /state/cid/0, method: GET, req_id: ad5ebc1f-9e85-4808-a6ae-a7a80b7b50e7, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
67372025-03-17T22:17:44.690ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
67382025-03-17T22:17:44.690ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
67392025-03-17T22:17:44.690ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
67402025-03-17T22:17:44.690ZINFOcrucible: client task is exiting = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
6741 [8810] Exited with: ExitStatus(unix_wait_status(9))
6742 [8810][0] reports Exit
6743 Mar 17 22:17:48.692 INFO request completed, latency_us: 346, response_code: 200, uri: /state/cid/0, method: GET, req_id: 463015e7-247c-4342-b30d-1d2674126021, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
6744 001/300 Read block 2514 len 36864
6745 002/300 Write block 4706 len 24576 data: 2 2 2 2 2 2
6746 003/300 Read block 2045 len 32768
6747 004/300 Read block 4144 len 28672
6748 005/300 Write block 418 len 4096 data: 3
6749 006/300 Read block 2018 len 8192
6750 007/300 Read block 513 len 4096
6751 008/300 Flush
6752 009/300 Write block 4673 len 16384 data: 3 3 2 2
6753 010/300 Write block 4965 len 8192 data: 2 2
6754 011/300 Write block 1405 len 12288 data: 3 3 2
6755 012/300 Flush
6756 013/300 Read block 1675 len 24576
6757 014/300 Read block 3207 len 36864
6758 015/300 Read block 2140 len 28672
6759 016/300 Read block 4713 len 4096
6760 017/300 Write block 2845 len 8192 data: 5 5
6761 018/300 Flush
6762 019/300 Read block 1074 len 24576
6763 020/300 Write block 2573 len 20480 data: 3 2 2 2 2
6764 021/300 Write block 2435 len 28672 data: 3 3 5 5 5 6 5
6765 022/300 Write block 2278 len 24576 data: 4 4 4 4 4 4
6766 023/300 Read block 3493 len 28672
6767 024/300 Read block 1834 len 32768
6768 025/300 Read block 4345 len 32768
6769 026/300 Read block 4601 len 4096
6770 027/300 Write block 3701 len 20480 data: 3 3 3 3 2
6771 028/300 Read block 2769 len 12288
6772 029/300 Write block 4990 len 36864 data: 2 2 2 2 2 2 2 2 2
6773 030/300 Read block 3590 len 16384
6774 031/300 Write block 1155 len 32768 data: 2 2 2 2 2 2 2 2
6775 032/300 Write block 1188 len 8192 data: 2 2
6776 033/300 Write block 4688 len 16384 data: 3 4 4 4
6777 034/300 Write block 1921 len 4096 data: 2
6778 035/300 Read block 1476 len 24576
6779 036/300 Write block 3129 len 32768 data: 2 2 2 2 3 3 3 3
6780 037/300 Read block 1334 len 36864
6781 038/300 Read block 3059 len 20480
6782 039/300 Flush
6783 040/300 Read block 3893 len 20480
6784 041/300 Read block 4191 len 24576
6785 042/300 Read block 2782 len 12288
6786 043/300 Read block 2502 len 20480
6787 044/300 Read block 10 len 32768
6788 045/300 Read block 2099 len 40960
6789 046/300 Write block 1915 len 16384 data: 2 2 2 2
6790 047/300 Read block 1758 len 24576
6791 048/300 Read block 1095 len 24576
6792 049/300 Read block 2661 len 12288
6793 050/300 Write block 356 len 8192 data: 3 3
6794 051/300 Write block 298 len 16384 data: 2 2 2 3
6795 052/300 Read block 2387 len 36864
6796 053/300 Flush
6797 054/300 Flush
6798 055/300 Read block 485 len 40960
6799 056/300 Write block 3641 len 20480 data: 3 3 4 4 3
6800 057/300 Write block 1823 len 8192 data: 2 3
6801 058/300 Write block 2015 len 12288 data: 2 3 3
6802 059/300 Read block 3572 len 12288
6803 060/300 Write block 1666 len 12288 data: 3 4 4
6804 061/300 Read block 4933 len 4096
6805 062/300 Flush
6806 063/300 Write block 1568 len 24576 data: 4 4 4 3 3 3
6807 064/300 Write block 785 len 16384 data: 3 2 2 2
6808 065/300 Read block 3278 len 20480
6809 066/300 Write block 871 len 36864 data: 2 2 2 2 2 2 2 2 2
6810 067/300 Write block 1175 len 16384 data: 2 4 4 3
6811 068/300 Write block 1068 len 36864 data: 2 2 2 2 3 3 3 4 4
6812 069/300 Read block 3821 len 12288
6813 070/300 Read block 4198 len 36864
6814 071/300 Write block 4085 len 28672 data: 4 4 4 3 3 3 2
6815 072/300 Write block 1156 len 8192 data: 3 3
6816 073/300 Write block 168 len 28672 data: 4 4 3 3 3 3 3
6817 074/300 Write block 4097 len 12288 data: 2 3 4
6818 075/300 Write block 4884 len 12288 data: 4 3 3
6819 076/300 Read block 954 len 16384
6820 077/300 Write block 2604 len 4096 data: 3
6821 078/300 Read block 2260 len 4096
6822 079/300 Write block 813 len 12288 data: 2 2 2
6823 080/300 Write block 1550 len 20480 data: 2 2 2 2 2
6824 081/300 Write block 1384 len 28672 data: 2 2 2 2 2 3 3
6825 082/300 Read block 4075 len 24576
6826 083/300 Write block 160 len 32768 data: 5 5 4 4 4 5 4 4
6827 084/300 Read block 1388 len 36864
6828 085/300 Write block 3422 len 20480 data: 3 3 3 3 3
6829 086/300 Read block 3260 len 28672
6830 087/300 Write block 715 len 4096 data: 4
6831 088/300 Write block 3097 len 4096 data: 3
6832 089/300 Write block 767 len 32768 data: 2 2 2 2 2 2 2 2
6833 090/300 Read block 1294 len 16384
6834 091/300 Read block 1682 len 28672
6835 092/300 Flush
6836 093/300 Write block 339 len 4096 data: 2
6837 094/300 Read block 730 len 8192
6838 095/300 Write block 4547 len 12288 data: 3 2 3
6839 096/300 Write block 1860 len 24576 data: 4 3 2 2 2 2
6840 097/300 Read block 3504 len 28672
6841 098/300 Read block 4755 len 24576
6842 099/300 Read block 1166 len 8192
6843 100/300 Read block 3351 len 20480
6844 101/300 Write block 270 len 32768 data: 4 4 4 4 3 3 3 3
6845 102/300 Read block 1491 len 12288
6846 103/300 Read block 3187 len 8192
6847 104/300 Write block 4019 len 20480 data: 3 2 2 2 2
6848 105/300 Flush
6849 106/300 Read block 249 len 36864
6850 107/300 Write block 594 len 28672 data: 2 2 2 2 3 3 3
6851 108/300 Write block 2884 len 4096 data: 3
6852 109/300 Read block 2423 len 20480
6853 110/300 Read block 2584 len 40960
6854 111/300 Write block 2825 len 24576 data: 5 5 5 3 3 4
6855 112/300 Write block 3047 len 4096 data: 5
6856 113/300 Write block 1341 len 24576 data: 3 3 3 3 2 2
6857 114/300 Read block 3513 len 28672
6858 115/300 Write block 2333 len 16384 data: 2 3 4 4
6859 116/300 Read block 4486 len 16384
6860 117/300 Read block 1588 len 32768
6861 118/300 Write block 3796 len 32768 data: 3 3 3 4 4 4 4 3
6862 119/300 Flush
6863 120/300 Write block 1089 len 28672 data: 3 3 3 3 3 3 3
6864 121/300 Read block 4799 len 12288
6865 122/300 Read block 193 len 36864
6866 123/300 Read block 2599 len 28672
6867 124/300 Flush
6868 125/300 Read block 451 len 4096
6869 126/300 Read block 3781 len 20480
6870 127/300 Write block 2565 len 28672 data: 3 3 3 3 3 3 3
6871 128/300 Write block 3812 len 8192 data: 3 3
6872 129/300 Read block 3840 len 32768
6873 130/300 Write block 4421 len 28672 data: 3 2 2 2 2 3 3
6874 131/300 Write block 922 len 28672 data: 2 2 3 3 3 3 3
6875 132/300 Write block 490 len 36864 data: 2 2 2 3 3 3 3 4 3
6876 133/300 Write block 297 len 20480 data: 2 3 3 3 4
6877 134/300 Write block 2458 len 12288 data: 2 2 2
6878 135/300 Write block 458 len 20480 data: 2 2 2 2 2
6879 136/300 Write block 4904 len 16384 data: 4 4 4 4
6880 137/300 Write block 1452 len 8192 data: 2 2
6881 138/300 Read block 1757 len 4096
6882 139/300 Read block 1660 len 20480
6883 140/300 Write block 3613 len 4096 data: 2
6884 141/300 Write block 1734 len 16384 data: 3 3 2 3
6885 142/300 Read block 4557 len 12288
6886 143/300 Write block 1222 len 28672 data: 4 3 3 3 3 3 3
6887 144/300 Read block 2524 len 32768
6888 145/300 Read block 4113 len 24576
6889 146/300 Read block 4994 len 24576
6890 147/300 Write block 44 len 32768 data: 4 4 4 4 3 3 3 3
6891 148/300 Write block 4436 len 40960 data: 3 3 3 3 3 3 3 3 3 3
6892 149/300 Read block 4807 len 20480
6893 150/300 Read block 1256 len 28672
6894 151/300 Read block 1959 len 24576
6895 152/300 Read block 2347 len 40960
6896 153/300 Write block 2686 len 20480 data: 3 3 3 2 2
6897 154/300 Flush
6898 155/300 Write block 2969 len 28672 data: 3 2 2 2 4 3 3
6899 156/300 Read block 612 len 40960
6900 157/300 Write block 1963 len 4096 data: 5
6901 158/300 Read block 1975 len 8192
6902 159/300 Read block 2802 len 32768
6903 160/300 Read block 863 len 20480
6904 161/300 Write block 2659 len 12288 data: 3 2 2
6905 162/300 Write block 2178 len 40960 data: 3 2 2 3 3 3 3 3 3 3
6906 163/300 Read block 4306 len 32768
6907 164/300 Read block 2438 len 12288
6908 165/300 Write block 4301 len 4096 data: 5
6909 166/300 Read block 64 len 20480
6910 167/300 Read block 523 len 24576
6911 168/300 Read block 1121 len 4096
6912 169/300 Write block 445 len 40960 data: 2 2 2 2 2 2 2 2 2 2
6913 170/300 Read block 1161 len 36864
6914 171/300 Read block 662 len 24576
6915 172/300 Write block 2675 len 16384 data: 4 4 3 3
6916 173/300 Write block 4093 len 12288 data: 2 2 3
6917 174/300 Write block 2012 len 16384 data: 2 2 2 3
6918 175/300 Read block 2625 len 32768
6919 176/300 Write block 2855 len 16384 data: 3 2 2 2
6920 177/300 Read block 1719 len 32768
6921 178/300 Read block 3992 len 8192
6922 179/300 Read block 2861 len 24576
6923 180/300 Write block 1273 len 28672 data: 3 3 4 4 4 4 3
6924 181/300 Flush
6925 182/300 Write block 3897 len 24576 data: 4 4 4 4 4 3
6926 183/300 Write block 3657 len 4096 data: 2
6927 184/300 Read block 58 len 28672
6928 185/300 Read block 4624 len 20480
6929 186/300 Flush
6930 187/300 Flush
6931 188/300 Read block 4907 len 16384
6932 189/300 Flush
6933 190/300 Write block 3001 len 32768 data: 2 2 2 3 3 2 2 2
6934 191/300 Flush
6935 192/300 Write block 1820 len 8192 data: 3 2
6936 193/300 Read block 3988 len 8192
6937 194/300 Read block 1613 len 20480
6938 195/300 Flush
6939 196/300 Write block 3013 len 20480 data: 2 2 3 3 3
6940 197/300 Flush
6941 198/300 Write block 2704 len 8192 data: 3 3
6942 199/300 Write block 2355 len 12288 data: 2 2 2
6943 200/300 Read block 1104 len 24576
6944 201/300 Read block 567 len 36864
6945 202/300 Read block 183 len 28672
6946 203/300 Read block 2465 len 20480
6947 204/300 Flush
6948 205/300 Write block 2957 len 4096 data: 4
6949 206/300 Write block 2502 len 8192 data: 2 2
6950 207/300 Read block 3336 len 36864
6951 208/300 Read block 3291 len 40960
6952 209/300 Read block 2191 len 32768
6953 210/300 Write block 2885 len 8192 data: 4 5
6954 211/300 Read block 2910 len 36864
6955 212/300 Read block 455 len 24576
6956 213/300 Read block 1987 len 28672
6957 214/300 Read block 1230 len 28672
6958 215/300 Write block 2118 len 16384 data: 5 5 4 4
6959 216/300 Read block 1119 len 36864
6960 217/300 Read block 2252 len 12288
6961 218/300 Read block 1290 len 12288
6962 219/300 Read block 2223 len 4096
6963 220/300 Write block 2072 len 12288 data: 3 3 3
6964 221/300 Write block 3968 len 4096 data: 2
6965 222/300 Write block 1926 len 28672 data: 2 2 3 2 2 2 2
6966 223/300 Write block 3669 len 28672 data: 3 2 3 3 2 2 2
6967 224/300 Read block 4875 len 8192
6968 225/300 Read block 4998 len 4096
6969 226/300 Read block 2224 len 20480
6970 227/300 Write block 3009 len 16384 data: 2 2 2 2
6971 228/300 Write block 4614 len 16384 data: 2 2 2 2
6972 229/300 Write block 3089 len 8192 data: 2 2
6973 230/300 Write block 630 len 24576 data: 5 4 5 4 4 2
6974 231/300 Write block 638 len 20480 data: 2 2 2 2 2
6975 232/300 Read block 227 len 24576
6976 233/300 Write block 1410 len 40960 data: 3 3 3 3 3 3 3 3 3 4
6977 234/300 Flush
6978 235/300 Flush
6979 236/300 Write block 4812 len 28672 data: 4 3 2 2 2 2 2
6980 237/300 Flush
6981 238/300 Read block 1107 len 36864
6982 239/300 Read block 408 len 28672
6983 240/300 Write block 1234 len 20480 data: 2 2 2 2 2
6984 241/300 Write block 1738 len 24576 data: 3 3 2 2 2 2
6985 242/300 Read block 632 len 16384
6986 243/300 Write block 1150 len 36864 data: 3 3 3 3 2 3 4 4 3
6987 244/300 Read block 4112 len 32768
6988 245/300 Flush
6989 246/300 Write block 4743 len 28672 data: 2 2 2 2 2 2 2
6990 247/300 Write block 1434 len 32768 data: 3 2 2 2 2 2 2 2
6991 248/300 Read block 4697 len 28672
6992 249/300 Read block 4393 len 36864
6993 250/300 Flush
6994 251/300 Write block 2604 len 32768 data: 4 3 4 4 4 4 4 4
6995 252/300 Write block 4231 len 24576 data: 3 3 3 3 4 3
6996 253/300 Read block 4388 len 20480
6997 254/300 Read block 4883 len 8192
6998 255/300 Flush
6999 256/300 Write block 698 len 16384 data: 3 2 2 2
7000 257/300 Write block 1278 len 28672 data: 5 4 3 3 2 2 2
7001 258/300 Read block 2979 len 40960
7002 259/300 Read block 4530 len 12288
7003 260/300 Read block 833 len 4096
7004 261/300 Read block 1422 len 32768
7005 262/300 Write block 561 len 24576 data: 8 7 5 5 5 4
7006 263/300 Read block 3444 len 4096
7007 264/300 Read block 1588 len 20480
7008 265/300 Read block 4479 len 16384
7009 266/300 Read block 4595 len 12288
7010 267/300 Write block 3916 len 24576 data: 3 3 3 2 2 2
7011 268/300 Write block 2607 len 4096 data: 5
7012 269/300 Read block 3871 len 36864
7013 270/300 Read block 4251 len 4096
7014 271/300 Write block 1150 len 16384 data: 4 4 4 4
7015 272/300 Write block 2052 len 20480 data: 3 3 4 3 3
7016 273/300 Write block 4870 len 32768 data: 2 2 2 3 3 3 4 4
7017 274/300 Flush
7018 275/300 Read block 2572 len 36864
7019 276/300 Read block 2009 len 36864
7020 277/300 Write block 1326 len 24576 data: 2 2 2 2 2 2
7021 278/300 Read block 3298 len 28672
7022 279/300 Flush
7023 280/300 Read block 2595 len 20480
7024 281/300 Read block 3766 len 4096
7025 282/300 Flush
7026 283/300 Read block 4969 len 24576
7027 284/300 Read block 220 len 20480
7028 285/300 Write block 1989 len 16384 data: 3 3 3 4
7029 286/300 Write block 297 len 4096 data: 3
7030 287/300 Write block 3262 len 36864 data: 2 2 2 2 2 2 2 3 3
7031 288/300 Write block 1680 len 24576 data: 4 4 4 4 4 4
7032 289/300 Write block 4917 len 28672 data: 2 2 2 2 2 3 3
7033 290/300 Write block 1457 len 20480 data: 3 3 4 3 3
7034 291/300 Read block 339 len 36864
7035 292/300 Read block 4208 len 28672
7036 293/300 Read block 2089 len 36864
7037 294/300 Read block 1922 len 8192
7038 295/300 Read block 2905 len 4096
7039 296/300 Write block 2039 len 24576 data: 3 3 2 2 2 3
7040 297/300 Write block 3813 len 16384 data: 4 3 2 3
7041 298/300 Write block 2946 len 12288 data: 3 3 3
7042 299/300 Read block 1308 len 32768
7043 300/300 Write block 4403 len 32768 data: 2 3 3 4 4 4 4 3
7044 301/300 Write block 1274 len 8192 data: 4 5
7045 Main task has work to do, go find it
7046 got dsc Start(0)
7047 start 0
7048 Mar 17 22:17:48.928 INFO request completed, latency_us: 218, response_code: 204, uri: /start/cid/0, method: POST, req_id: 40954503-476d-4822-a1a8-57a9ff94bb61, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
7049 Replay: started 0, returned:Ok(())
7050 ----------------------------------------------------------------
7051 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7052 States: Offline Active Active
7053 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7054 2811 Acked Read 9 Sent Done Done false
7055 2812 Acked Write 6 Sent Done Done false
7056 2813 Acked Read 8 Sent Done Done false
7057 2814 Acked Read 7 Sent Done Done false
7058 2815 Acked Write 1 Sent Done Done false
7059 2816 Acked Read 2 Sent Done Done false
7060 2817 Acked Read 1 Sent Done Done false
7061 2818 Acked Flush 0 Sent Done Done false
7062 2819 Acked Write 4 Sent Done Done false
7063 2820 Acked Write 2 Sent Done Done false
7064 2821 Acked Write 3 Sent Done Done false
7065 2822 Acked Flush 0 Sent Done Done false
7066 2823 Acked Read 6 Sent Done Done false
7067 2824 Acked Read 9 Sent Done Done false
7068 2825 Acked Read 7 Sent Done Done false
7069 2826 Acked Read 1 Sent Done Done false
7070 2827 Acked Write 2 Sent Done Done false
7071 2828 Acked Flush 0 Sent Done Done false
7072 2829 Acked Read 6 Sent Done Done false
7073 2830 Acked Write 5 Sent Done Done false
7074 2831 Acked Write 7 Sent Done Done false
7075 2832 Acked Write 6 Sent Done Done false
7076 2833 Acked Read 7 Sent Done Done false
7077 2834 Acked Read 8 Sent Done Done false
7078 2835 Acked Read 8 Sent Done Done false
7079 2836 Acked Read 1 Sent Done Done false
7080 2837 Acked Write 5 Sent Done Done false
7081 2838 Acked Read 3 Sent Done Done false
7082 2839 Acked Write 9 Sent Done Done false
7083 2840 Acked Read 4 Sent Done Done false
7084 2841 Acked Write 8 Sent Done Done false
7085 2842 Acked Write 2 Sent Done Done false
7086 2843 Acked Write 4 Sent Done Done false
7087 2844 Acked Write 1 Sent Done Done false
7088 2845 Acked Read 6 Sent Done Done false
7089 2846 Acked Write 8 Sent Done Done false
7090 2847 Acked Read 9 Sent Done Done false
7091 2848 Acked Read 5 Sent Done Done false
7092 2849 Acked Flush 0 Sent Done Done false
7093 2850 Acked Read 5 Sent Done Done false
7094 2851 Acked Read 6 Sent Done Done false
7095 2852 Acked Read 3 Sent Done Done false
7096 2853 Acked Read 5 Sent Done Done false
7097 2854 Acked Read 8 Sent Done Done false
7098 2855 Acked Read 10 Sent Done Done false
7099 2856 Acked Write 4 Sent Done Done false
7100 2857 Acked Read 6 Sent Done Done false
7101 2858 Acked Read 6 Sent Done Done false
7102 2859 Acked Read 3 Sent Done Done false
7103 2860 Acked Write 2 Sent Done Done false
7104 2861 Acked Write 4 Sent Done Done false
7105 2862 Acked Read 9 Sent Done Done false
7106 2863 Acked Flush 0 Sent Done Done false
7107 2864 Acked Flush 0 Sent Done Done false
7108 2865 Acked Read 10 Sent Done Done false
7109 2866 Acked Write 5 Sent Done Done false
7110 2867 Acked Write 2 Sent Done Done false
7111 2868 Acked Write 3 Sent Done Done false
7112 2869 Acked Read 3 Sent Done Done false
7113 2870 Acked Write 3 Sent Done Done false
7114 2871 Acked Read 1 Sent Done Done false
7115 2872 Acked Flush 0 Sent Done Done false
7116 2873 Acked Write 6 Sent Done Done false
7117 2874 Acked Write 4 Sent Done Done false
7118 2875 Acked Read 5 Sent Done Done false
7119 2876 Acked Write 9 Sent Done Done false
7120 2877 Acked Write 4 Sent Done Done false
7121 2878 Acked Write 9 Sent Done Done false
7122 2879 Acked Read 3 Sent Done Done false
7123 2880 Acked Read 9 Sent Done Done false
7124 2881 Acked Write 7 Sent Done Done false
7125 2882 Acked Write 2 Sent Done Done false
7126 2883 Acked Write 7 Sent Done Done false
7127 2884 Acked Write 3 Sent Done Done false
7128 2885 Acked Write 3 Sent Done Done false
7129 2886 Acked Read 4 Sent Done Done false
7130 2887 Acked Write 1 Sent Done Done false
7131 2888 Acked Read 1 Sent Done Done false
7132 2889 Acked Write 3 Sent Done Done false
7133 2890 Acked Write 5 Sent Done Done false
7134 2891 Acked Write 7 Sent Done Done false
7135 2892 Acked Read 6 Sent Done Done false
7136 2893 Acked Write 8 Sent Done Done false
7137 2894 Acked Read 9 Sent Done Done false
7138 2895 Acked Write 5 Sent Done Done false
7139 2896 Acked Read 7 Sent Done Done false
7140 2897 Acked Write 1 Sent Done Done false
7141 2898 Acked Write 1 Sent Done Done false
7142 2899 Acked Write 8 Sent Done Done false
7143 2900 Acked Read 4 Sent Done Done false
7144 2901 Acked Read 7 Sent Done Done false
7145 2902 Acked Flush 0 Sent Done Done false
7146 2903 Acked Write 1 Sent Done Done false
7147 2904 Acked Read 2 Sent Done Done false
7148 2905 Acked Write 3 Sent Done Done false
7149 2906 Acked Write 6 Sent Done Done false
7150 2907 Acked Read 7 Sent Done Done false
7151 2908 Acked Read 6 Sent Done Done false
7152 2909 Acked Read 2 Sent Done Done false
7153 2910 Acked Read 5 Sent Done Done false
7154 2911 Acked Write 8 Sent Done Done false
7155 2912 Acked Read 3 Sent Done Done false
7156 2913 Acked Read 2 Sent Done Done false
7157 2914 Acked Write 5 Sent Done Done false
7158 2915 Acked Flush 0 Sent Done Done false
7159 2916 Acked Read 9 Sent Done Done false
7160 2917 Acked Write 7 Sent Done Done false
7161 2918 Acked Write 1 Sent Done Done false
7162 2919 Acked Read 5 Sent Done Done false
7163 2920 Acked Read 10 Sent Done Done false
7164 2921 Acked Write 6 Sent Done Done false
7165 2922 Acked Write 1 Sent Done Done false
7166 2923 Acked Write 6 Sent Done Done false
7167 2924 Acked Read 7 Sent Done Done false
7168 2925 Acked Write 4 Sent Done Done false
7169 2926 Acked Read 4 Sent Done Done false
7170 2927 Acked Read 8 Sent Done Done false
7171 2928 Acked Write 8 Sent Done Done false
7172 2929 Acked Flush 0 Sent Done Done false
7173 2930 Acked Write 7 Sent Done Done false
7174 2931 Acked Read 3 Sent Done Done false
7175 2932 Acked Read 9 Sent Done Done false
7176 2933 Acked Read 7 Sent Done Done false
7177 2934 Acked Flush 0 Sent Done Done false
7178 2935 Acked Read 1 Sent Done Done false
7179 2936 Acked Read 5 Sent Done Done false
7180 2937 Acked Write 7 Sent Done Done false
7181 2938 Acked Write 2 Sent Done Done false
7182 2939 Acked Read 8 Sent Done Done false
7183 2940 Acked Write 7 Sent Done Done false
7184 2941 Acked Write 7 Sent Done Done false
7185 2942 Acked Write 9 Sent Done Done false
7186 2943 Acked Write 5 Sent Done Done false
7187 2944 Acked Write 3 Sent Done Done false
7188 2945 Acked Write 5 Sent Done Done false
7189 2946 Acked Write 4 Sent Done Done false
7190 2947 Acked Write 2 Sent Done Done false
7191 2948 Acked Read 1 Sent Done Done false
7192 2949 Acked Read 5 Sent Done Done false
7193 2950 Acked Write 1 Sent Done Done false
7194 2951 Acked Write 4 Sent Done Done false
7195 2952 Acked Read 3 Sent Done Done false
7196 2953 Acked Write 7 Sent Done Done false
7197 2954 Acked Read 8 Sent Done Done false
7198 2955 Acked Read 6 Sent Done Done false
7199 2956 Acked Read 6 Sent Done Done false
7200 2957 Acked Write 8 Sent Done Done false
7201 2958 Acked Write 10 Sent Done Done false
7202 2959 Acked Read 5 Sent Done Done false
7203 2960 Acked Read 7 Sent Done Done false
7204 2961 Acked Read 6 Sent Done Done false
7205 2962 Acked Read 10 Sent Done Done false
7206 2963 Acked Write 5 Sent Done Done false
7207 2964 Acked Flush 0 Sent Done Done false
7208 2965 Acked Write 7 Sent Done Done false
7209 2966 Acked Read 10 Sent Done Done false
7210 2967 Acked Write 1 Sent Done Done false
7211 2968 Acked Read 2 Sent Done Done false
7212 2969 Acked Read 8 Sent Done Done false
7213 2970 Acked Read 5 Sent Done Done false
7214 2971 Acked Write 3 Sent Done Done false
7215 2972 Acked Write 10 Sent Done Done false
7216 2973 Acked Read 8 Sent Done Done false
7217 2974 Acked Read 3 Sent Done Done false
7218 2975 Acked Write 1 Sent Done Done false
7219 2976 Acked Read 5 Sent Done Done false
7220 2977 Acked Read 6 Sent Done Done false
7221 2978 Acked Read 1 Sent Done Done false
7222 2979 Acked Write 10 Sent Done Done false
7223 2980 Acked Read 9 Sent Done Done false
7224 2981 Acked Read 6 Sent Done Done false
7225 2982 Acked Write 4 Sent Done Done false
7226 2983 Acked Write 3 Sent Done Done false
7227 2984 Acked Write 4 Sent Done Done false
7228 2985 Acked Read 8 Sent Done Done false
7229 2986 Acked Write 4 Sent Done Done false
7230 2987 Acked Read 8 Sent Done Done false
7231 2988 Acked Read 2 Sent Done Done false
7232 2989 Acked Read 6 Sent Done Done false
7233 2990 Acked Write 7 Sent Done Done false
7234 2991 Acked Flush 0 Sent Done Done false
7235 2992 Acked Write 6 Sent Done Done false
7236 2993 Acked Write 1 Sent Done Done false
7237 2994 Acked Read 7 Sent Done Done false
7238 2995 Acked Read 5 Sent Done Done false
7239 2996 Acked Flush 0 Sent Done Done false
7240 2997 Acked Flush 0 Sent Done Done false
7241 2998 Acked Read 4 Sent Done Done false
7242 2999 Acked Flush 0 Sent Done Done false
7243 3000 Acked Write 8 Sent Done Done false
7244 3001 Acked Flush 0 Sent Done Done false
7245 3002 Acked Write 2 Sent Done Done false
7246 3003 Acked Read 2 Sent Done Done false
7247 3004 Acked Read 5 Sent Done Done false
7248 3005 Acked Flush 0 Sent Done Done false
7249 3006 Acked Write 5 Sent Done Done false
7250 3007 Acked Flush 0 Sent Done Done false
7251 3008 Acked Write 2 Sent Done Done false
7252 3009 Acked Write 3 Sent Done Done false
7253 3010 Acked Read 6 Sent Done Done false
7254 3011 Acked Read 9 Sent Done Done false
7255 3012 Acked Read 7 Sent Done Done false
7256 3013 Acked Read 5 Sent Done Done false
7257 3014 Acked Flush 0 Sent Done Done false
7258 3015 Acked Write 1 Sent Done Done false
7259 3016 Acked Write 2 Sent Done Done false
7260 3017 Acked Read 9 Sent Done Done false
7261 3018 Acked Read 10 Sent Done Done false
7262 3019 Acked Read 8 Sent Done Done false
7263 3020 Acked Write 2 Sent Done Done false
7264 3021 Acked Read 9 Sent Done Done false
7265 3022 Acked Read 6 Sent Done Done false
7266 3023 Acked Read 7 Sent Done Done false
7267 3024 Acked Read 7 Sent Done Done false
7268 3025 Acked Write 4 Sent Done Done false
7269 3026 Acked Read 9 Sent Done Done false
7270 3027 Acked Read 3 Sent Done Done false
7271 3028 Acked Read 3 Sent Done Done false
7272 3029 Acked Read 1 Sent Done Done false
7273 3030 Acked Write 3 Sent Done Done false
7274 3031 Acked Write 1 Sent Done Done false
7275 3032 Acked Write 7 Sent Done Done false
7276 3033 Acked Write 7 Sent Done Done false
7277 3034 Acked Read 2 Sent Done Done false
7278 3035 Acked Read 1 Sent Done Done false
7279 3036 Acked Read 5 Sent Done Done false
7280 3037 Acked Write 4 Sent Done Done false
7281 3038 Acked Write 4 Sent Done Done false
7282 3039 Acked Write 2 Sent Done Done false
7283 3040 Acked Write 6 Sent Done Done false
7284 3041 Acked Write 5 Sent Done Done false
7285 3042 Acked Read 6 Sent Done Done false
7286 3043 Acked Write 10 Sent Done Done false
7287 3044 Acked Flush 0 Sent Done Done false
7288 3045 Acked Flush 0 Sent Done Done false
7289 3046 Acked Write 7 Sent Done Done false
7290 3047 Acked Flush 0 Sent Done Done false
7291 3048 Acked Read 9 Sent Done Done false
7292 3049 Acked Read 7 Sent Done Done false
7293 3050 Acked Write 5 Sent Done Done false
7294 3051 Acked Write 6 Sent Done Done false
7295 3052 Acked Read 4 Sent Done Done false
7296 3053 Acked Write 9 Sent Done Done false
7297 3054 Acked Read 8 Sent Done Done false
7298 3055 Acked Flush 0 Sent Done Done false
7299 3056 Acked Write 7 Sent Done Done false
7300 3057 Acked Write 8 Sent Done Done false
7301 3058 Acked Read 7 Sent Done Done false
7302 3059 Acked Read 9 Sent Done Done false
7303 3060 Acked Flush 0 Sent Done Done false
7304 3061 Acked Write 8 Sent Done Done false
7305 3062 Acked Write 6 Sent Done Done false
7306 3063 Acked Read 5 Sent Done Done false
7307 3064 Acked Read 2 Sent Done Done false
7308 3065 Acked Flush 0 Sent Done Done false
7309 3066 Acked Write 4 Sent Done Done false
7310 3067 Acked Write 7 Sent Done Done false
7311 3068 Acked Read 10 Sent Done Done false
7312 3069 Acked Read 3 Sent Done Done false
7313 3070 Acked Read 1 Sent Done Done false
7314 3071 Acked Read 8 Sent Done Done false
7315 3072 Acked Write 6 Sent Done Done false
7316 3073 Acked Read 1 Sent Done Done false
7317 3074 Acked Read 5 Sent Done Done false
7318 3075 Acked Read 4 Sent Done Done false
7319 3076 Acked Read 3 Sent Done Done false
7320 3077 Acked Write 6 Sent Done Done false
7321 3078 Acked Write 1 Sent Done Done false
7322 3079 Acked Read 9 Sent Done Done false
7323 3080 Acked Read 1 Sent Done Done false
7324 3081 Acked Write 4 Sent Done Done false
7325 3082 Acked Write 5 Sent Done Done false
7326 3083 Acked Write 8 Sent Done Done false
7327 3084 Acked Flush 0 Sent Done Done false
7328 3085 Acked Read 9 Sent Done Done false
7329 3086 Acked Read 9 Sent Done Done false
7330 3087 Acked Write 6 Sent Done Done false
7331 3088 Acked Read 7 Sent Done Done false
7332 3089 Acked Flush 0 Sent Done Done false
7333 3090 Acked Read 5 Sent Done Done false
7334 3091 Acked Read 1 Sent Done Done false
7335 3092 Acked Flush 0 Sent Done Done false
7336 3093 Acked Read 6 Sent Done Done false
7337 3094 Acked Read 5 Sent Done Done false
7338 3095 Acked Write 4 Sent Done Done false
7339 3096 Acked Write 1 Sent Done Done false
7340 3097 Acked Write 9 Sent Done Done false
7341 3098 Acked Write 6 Sent Done Done false
7342 3099 Acked Write 7 Sent Done Done false
7343 3100 Acked Write 5 Sent Done Done false
7344 3101 Acked Read 9 Sent Done Done false
7345 3102 Acked Read 7 Sent Done Done false
7346 3103 Acked Read 9 Sent Done Done false
7347 3104 Acked Read 2 Sent Done Done false
7348 3105 Acked Read 1 Sent Done Done false
7349 3106 Acked Write 6 Sent Done Done false
7350 3107 Acked Write 4 Sent Done Done false
7351 3108 Acked Write 3 Sent Done Done false
7352 3109 Acked Read 8 Sent Done Done false
7353 3110 Acked Write 8 Sent Done Done false
7354 3111 Acked Write 2 Sent Done Done false
7355 STATES DS:0 DS:1 DS:2 TOTAL
7356 Sent 301 0 0 301
7357 Done 0 301 301 602
7358 Skipped 0 0 0 0
7359 Error 0 0 0 0
7360 Last Flush: Some(JobId(2810)) Some(JobId(3092)) Some(JobId(3092))
7361 Downstairs last five completed: 2810 2809 2808 2807 2806
7362 Upstairs last five completed: 3111 3110 3109 3108 3107
7363 CLIENT: Up:0 ds:301 act:2
73642025-03-17T22:17:49.741ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
7365 [8810] got start action
7366 [8810] I am going to restart
7367 Starting downstairs at port 8810
7368 Make output file at "/tmp/dsc/downstairs-8810.txt"
7369 [8810][0] reports Starting
7370 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1213)
7371 [8810][0] reports Running
73722025-03-17T22:17:50.742ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
73732025-03-17T22:17:51.744ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
7374 ----------------------------------------------------------------
7375 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7376 States: Offline Active Active
7377 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
7378 2811 Acked Read 9 Sent Done Done false
7379 2812 Acked Write 6 Sent Done Done false
7380 2813 Acked Read 8 Sent Done Done false
7381 2814 Acked Read 7 Sent Done Done false
7382 2815 Acked Write 1 Sent Done Done false
7383 2816 Acked Read 2 Sent Done Done false
7384 2817 Acked Read 1 Sent Done Done false
7385 2818 Acked Flush 0 Sent Done Done false
7386 2819 Acked Write 4 Sent Done Done false
7387 2820 Acked Write 2 Sent Done Done false
7388 2821 Acked Write 3 Sent Done Done false
7389 2822 Acked Flush 0 Sent Done Done false
7390 2823 Acked Read 6 Sent Done Done false
7391 2824 Acked Read 9 Sent Done Done false
7392 2825 Acked Read 7 Sent Done Done false
7393 2826 Acked Read 1 Sent Done Done false
7394 2827 Acked Write 2 Sent Done Done false
7395 2828 Acked Flush 0 Sent Done Done false
7396 2829 Acked Read 6 Sent Done Done false
7397 2830 Acked Write 5 Sent Done Done false
7398 2831 Acked Write 7 Sent Done Done false
7399 2832 Acked Write 6 Sent Done Done false
7400 2833 Acked Read 7 Sent Done Done false
7401 2834 Acked Read 8 Sent Done Done false
7402 2835 Acked Read 8 Sent Done Done false
7403 2836 Acked Read 1 Sent Done Done false
7404 2837 Acked Write 5 Sent Done Done false
7405 2838 Acked Read 3 Sent Done Done false
7406 2839 Acked Write 9 Sent Done Done false
7407 2840 Acked Read 4 Sent Done Done false
7408 2841 Acked Write 8 Sent Done Done false
7409 2842 Acked Write 2 Sent Done Done false
7410 2843 Acked Write 4 Sent Done Done false
7411 2844 Acked Write 1 Sent Done Done false
7412 2845 Acked Read 6 Sent Done Done false
7413 2846 Acked Write 8 Sent Done Done false
7414 2847 Acked Read 9 Sent Done Done false
7415 2848 Acked Read 5 Sent Done Done false
7416 2849 Acked Flush 0 Sent Done Done false
7417 2850 Acked Read 5 Sent Done Done false
7418 2851 Acked Read 6 Sent Done Done false
7419 2852 Acked Read 3 Sent Done Done false
7420 2853 Acked Read 5 Sent Done Done false
7421 2854 Acked Read 8 Sent Done Done false
7422 2855 Acked Read 10 Sent Done Done false
7423 2856 Acked Write 4 Sent Done Done false
7424 2857 Acked Read 6 Sent Done Done false
7425 2858 Acked Read 6 Sent Done Done false
7426 2859 Acked Read 3 Sent Done Done false
7427 2860 Acked Write 2 Sent Done Done false
7428 2861 Acked Write 4 Sent Done Done false
7429 2862 Acked Read 9 Sent Done Done false
7430 2863 Acked Flush 0 Sent Done Done false
7431 2864 Acked Flush 0 Sent Done Done false
7432 2865 Acked Read 10 Sent Done Done false
7433 2866 Acked Write 5 Sent Done Done false
7434 2867 Acked Write 2 Sent Done Done false
7435 2868 Acked Write 3 Sent Done Done false
7436 2869 Acked Read 3 Sent Done Done false
7437 2870 Acked Write 3 Sent Done Done false
7438 2871 Acked Read 1 Sent Done Done false
7439 2872 Acked Flush 0 Sent Done Done false
7440 2873 Acked Write 6 Sent Done Done false
7441 2874 Acked Write 4 Sent Done Done false
7442 2875 Acked Read 5 Sent Done Done false
7443 2876 Acked Write 9 Sent Done Done false
7444 2877 Acked Write 4 Sent Done Done false
7445 2878 Acked Write 9 Sent Done Done false
7446 2879 Acked Read 3 Sent Done Done false
7447 2880 Acked Read 9 Sent Done Done false
7448 2881 Acked Write 7 Sent Done Done false
7449 2882 Acked Write 2 Sent Done Done false
7450 2883 Acked Write 7 Sent Done Done false
7451 2884 Acked Write 3 Sent Done Done false
7452 2885 Acked Write 3 Sent Done Done false
7453 2886 Acked Read 4 Sent Done Done false
7454 2887 Acked Write 1 Sent Done Done false
7455 2888 Acked Read 1 Sent Done Done false
7456 2889 Acked Write 3 Sent Done Done false
7457 2890 Acked Write 5 Sent Done Done false
7458 2891 Acked Write 7 Sent Done Done false
7459 2892 Acked Read 6 Sent Done Done false
7460 2893 Acked Write 8 Sent Done Done false
7461 2894 Acked Read 9 Sent Done Done false
7462 2895 Acked Write 5 Sent Done Done false
7463 2896 Acked Read 7 Sent Done Done false
7464 2897 Acked Write 1 Sent Done Done false
7465 2898 Acked Write 1 Sent Done Done false
7466 2899 Acked Write 8 Sent Done Done false
7467 2900 Acked Read 4 Sent Done Done false
7468 2901 Acked Read 7 Sent Done Done false
7469 2902 Acked Flush 0 Sent Done Done false
7470 2903 Acked Write 1 Sent Done Done false
7471 2904 Acked Read 2 Sent Done Done false
7472 2905 Acked Write 3 Sent Done Done false
7473 2906 Acked Write 6 Sent Done Done false
7474 2907 Acked Read 7 Sent Done Done false
7475 2908 Acked Read 6 Sent Done Done false
7476 2909 Acked Read 2 Sent Done Done false
7477 2910 Acked Read 5 Sent Done Done false
7478 2911 Acked Write 8 Sent Done Done false
7479 2912 Acked Read 3 Sent Done Done false
7480 2913 Acked Read 2 Sent Done Done false
7481 2914 Acked Write 5 Sent Done Done false
7482 2915 Acked Flush 0 Sent Done Done false
7483 2916 Acked Read 9 Sent Done Done false
7484 2917 Acked Write 7 Sent Done Done false
7485 2918 Acked Write 1 Sent Done Done false
7486 2919 Acked Read 5 Sent Done Done false
7487 2920 Acked Read 10 Sent Done Done false
7488 2921 Acked Write 6 Sent Done Done false
7489 2922 Acked Write 1 Sent Done Done false
7490 2923 Acked Write 6 Sent Done Done false
7491 2924 Acked Read 7 Sent Done Done false
7492 2925 Acked Write 4 Sent Done Done false
7493 2926 Acked Read 4 Sent Done Done false
7494 2927 Acked Read 8 Sent Done Done false
7495 2928 Acked Write 8 Sent Done Done false
7496 2929 Acked Flush 0 Sent Done Done false
7497 2930 Acked Write 7 Sent Done Done false
7498 2931 Acked Read 3 Sent Done Done false
7499 2932 Acked Read 9 Sent Done Done false
7500 2933 Acked Read 7 Sent Done Done false
7501 2934 Acked Flush 0 Sent Done Done false
7502 2935 Acked Read 1 Sent Done Done false
7503 2936 Acked Read 5 Sent Done Done false
7504 2937 Acked Write 7 Sent Done Done false
7505 2938 Acked Write 2 Sent Done Done false
7506 2939 Acked Read 8 Sent Done Done false
7507 2940 Acked Write 7 Sent Done Done false
7508 2941 Acked Write 7 Sent Done Done false
7509 2942 Acked Write 9 Sent Done Done false
7510 2943 Acked Write 5 Sent Done Done false
7511 2944 Acked Write 3 Sent Done Done false
7512 2945 Acked Write 5 Sent Done Done false
7513 2946 Acked Write 4 Sent Done Done false
7514 2947 Acked Write 2 Sent Done Done false
7515 2948 Acked Read 1 Sent Done Done false
7516 2949 Acked Read 5 Sent Done Done false
7517 2950 Acked Write 1 Sent Done Done false
7518 2951 Acked Write 4 Sent Done Done false
7519 2952 Acked Read 3 Sent Done Done false
7520 2953 Acked Write 7 Sent Done Done false
7521 2954 Acked Read 8 Sent Done Done false
7522 2955 Acked Read 6 Sent Done Done false
7523 2956 Acked Read 6 Sent Done Done false
7524 2957 Acked Write 8 Sent Done Done false
7525 2958 Acked Write 10 Sent Done Done false
7526 2959 Acked Read 5 Sent Done Done false
7527 2960 Acked Read 7 Sent Done Done false
7528 2961 Acked Read 6 Sent Done Done false
7529 2962 Acked Read 10 Sent Done Done false
7530 2963 Acked Write 5 Sent Done Done false
7531 2964 Acked Flush 0 Sent Done Done false
7532 2965 Acked Write 7 Sent Done Done false
7533 2966 Acked Read 10 Sent Done Done false
7534 2967 Acked Write 1 Sent Done Done false
7535 2968 Acked Read 2 Sent Done Done false
7536 2969 Acked Read 8 Sent Done Done false
7537 2970 Acked Read 5 Sent Done Done false
7538 2971 Acked Write 3 Sent Done Done false
7539 2972 Acked Write 10 Sent Done Done false
7540 2973 Acked Read 8 Sent Done Done false
7541 2974 Acked Read 3 Sent Done Done false
7542 2975 Acked Write 1 Sent Done Done false
7543 2976 Acked Read 5 Sent Done Done false
7544 2977 Acked Read 6 Sent Done Done false
7545 2978 Acked Read 1 Sent Done Done false
7546 2979 Acked Write 10 Sent Done Done false
7547 2980 Acked Read 9 Sent Done Done false
7548 2981 Acked Read 6 Sent Done Done false
7549 2982 Acked Write 4 Sent Done Done false
7550 2983 Acked Write 3 Sent Done Done false
7551 2984 Acked Write 4 Sent Done Done false
7552 2985 Acked Read 8 Sent Done Done false
7553 2986 Acked Write 4 Sent Done Done false
7554 2987 Acked Read 8 Sent Done Done false
7555 2988 Acked Read 2 Sent Done Done false
7556 2989 Acked Read 6 Sent Done Done false
7557 2990 Acked Write 7 Sent Done Done false
7558 2991 Acked Flush 0 Sent Done Done false
7559 2992 Acked Write 6 Sent Done Done false
7560 2993 Acked Write 1 Sent Done Done false
7561 2994 Acked Read 7 Sent Done Done false
7562 2995 Acked Read 5 Sent Done Done false
7563 2996 Acked Flush 0 Sent Done Done false
7564 2997 Acked Flush 0 Sent Done Done false
7565 2998 Acked Read 4 Sent Done Done false
7566 2999 Acked Flush 0 Sent Done Done false
7567 3000 Acked Write 8 Sent Done Done false
7568 3001 Acked Flush 0 Sent Done Done false
7569 3002 Acked Write 2 Sent Done Done false
7570 3003 Acked Read 2 Sent Done Done false
7571 3004 Acked Read 5 Sent Done Done false
7572 3005 Acked Flush 0 Sent Done Done false
7573 3006 Acked Write 5 Sent Done Done false
7574 3007 Acked Flush 0 Sent Done Done false
7575 3008 Acked Write 2 Sent Done Done false
7576 3009 Acked Write 3 Sent Done Done false
7577 3010 Acked Read 6 Sent Done Done false
7578 3011 Acked Read 9 Sent Done Done false
7579 3012 Acked Read 7 Sent Done Done false
7580 3013 Acked Read 5 Sent Done Done false
7581 3014 Acked Flush 0 Sent Done Done false
7582 3015 Acked Write 1 Sent Done Done false
7583 3016 Acked Write 2 Sent Done Done false
7584 3017 Acked Read 9 Sent Done Done false
7585 3018 Acked Read 10 Sent Done Done false
7586 3019 Acked Read 8 Sent Done Done false
7587 3020 Acked Write 2 Sent Done Done false
7588 3021 Acked Read 9 Sent Done Done false
7589 3022 Acked Read 6 Sent Done Done false
7590 3023 Acked Read 7 Sent Done Done false
7591 3024 Acked Read 7 Sent Done Done false
7592 3025 Acked Write 4 Sent Done Done false
7593 3026 Acked Read 9 Sent Done Done false
7594 3027 Acked Read 3 Sent Done Done false
7595 3028 Acked Read 3 Sent Done Done false
7596 3029 Acked Read 1 Sent Done Done false
7597 3030 Acked Write 3 Sent Done Done false
7598 3031 Acked Write 1 Sent Done Done false
7599 3032 Acked Write 7 Sent Done Done false
7600 3033 Acked Write 7 Sent Done Done false
7601 3034 Acked Read 2 Sent Done Done false
7602 3035 Acked Read 1 Sent Done Done false
7603 3036 Acked Read 5 Sent Done Done false
7604 3037 Acked Write 4 Sent Done Done false
7605 3038 Acked Write 4 Sent Done Done false
7606 3039 Acked Write 2 Sent Done Done false
7607 3040 Acked Write 6 Sent Done Done false
7608 3041 Acked Write 5 Sent Done Done false
7609 3042 Acked Read 6 Sent Done Done false
7610 3043 Acked Write 10 Sent Done Done false
7611 3044 Acked Flush 0 Sent Done Done false
7612 3045 Acked Flush 0 Sent Done Done false
7613 3046 Acked Write 7 Sent Done Done false
7614 3047 Acked Flush 0 Sent Done Done false
7615 3048 Acked Read 9 Sent Done Done false
7616 3049 Acked Read 7 Sent Done Done false
7617 3050 Acked Write 5 Sent Done Done false
7618 3051 Acked Write 6 Sent Done Done false
7619 3052 Acked Read 4 Sent Done Done false
7620 3053 Acked Write 9 Sent Done Done false
7621 3054 Acked Read 8 Sent Done Done false
7622 3055 Acked Flush 0 Sent Done Done false
7623 3056 Acked Write 7 Sent Done Done false
7624 3057 Acked Write 8 Sent Done Done false
7625 3058 Acked Read 7 Sent Done Done false
7626 3059 Acked Read 9 Sent Done Done false
7627 3060 Acked Flush 0 Sent Done Done false
7628 3061 Acked Write 8 Sent Done Done false
7629 3062 Acked Write 6 Sent Done Done false
7630 3063 Acked Read 5 Sent Done Done false
7631 3064 Acked Read 2 Sent Done Done false
7632 3065 Acked Flush 0 Sent Done Done false
7633 3066 Acked Write 4 Sent Done Done false
7634 3067 Acked Write 7 Sent Done Done false
7635 3068 Acked Read 10 Sent Done Done false
7636 3069 Acked Read 3 Sent Done Done false
7637 3070 Acked Read 1 Sent Done Done false
7638 3071 Acked Read 8 Sent Done Done false
7639 3072 Acked Write 6 Sent Done Done false
7640 3073 Acked Read 1 Sent Done Done false
7641 3074 Acked Read 5 Sent Done Done false
7642 3075 Acked Read 4 Sent Done Done false
7643 3076 Acked Read 3 Sent Done Done false
7644 3077 Acked Write 6 Sent Done Done false
7645 3078 Acked Write 1 Sent Done Done false
7646 3079 Acked Read 9 Sent Done Done false
7647 3080 Acked Read 1 Sent Done Done false
7648 3081 Acked Write 4 Sent Done Done false
7649 3082 Acked Write 5 Sent Done Done false
7650 3083 Acked Write 8 Sent Done Done false
7651 3084 Acked Flush 0 Sent Done Done false
7652 3085 Acked Read 9 Sent Done Done false
7653 3086 Acked Read 9 Sent Done Done false
7654 3087 Acked Write 6 Sent Done Done false
7655 3088 Acked Read 7 Sent Done Done false
7656 3089 Acked Flush 0 Sent Done Done false
7657 3090 Acked Read 5 Sent Done Done false
7658 3091 Acked Read 1 Sent Done Done false
7659 3092 Acked Flush 0 Sent Done Done false
7660 3093 Acked Read 6 Sent Done Done false
7661 3094 Acked Read 5 Sent Done Done false
7662 3095 Acked Write 4 Sent Done Done false
7663 3096 Acked Write 1 Sent Done Done false
7664 3097 Acked Write 9 Sent Done Done false
7665 3098 Acked Write 6 Sent Done Done false
7666 3099 Acked Write 7 Sent Done Done false
7667 3100 Acked Write 5 Sent Done Done false
7668 3101 Acked Read 9 Sent Done Done false
7669 3102 Acked Read 7 Sent Done Done false
7670 3103 Acked Read 9 Sent Done Done false
7671 3104 Acked Read 2 Sent Done Done false
7672 3105 Acked Read 1 Sent Done Done false
7673 3106 Acked Write 6 Sent Done Done false
7674 3107 Acked Write 4 Sent Done Done false
7675 3108 Acked Write 3 Sent Done Done false
7676 3109 Acked Read 8 Sent Done Done false
7677 3110 Acked Write 8 Sent Done Done false
7678 3111 Acked Write 2 Sent Done Done false
7679 STATES DS:0 DS:1 DS:2 TOTAL
7680 Sent 301 0 0 301
7681 Done 0 301 301 602
7682 Skipped 0 0 0 0
7683 Error 0 0 0 0
7684 Last Flush: Some(JobId(2810)) Some(JobId(3092)) Some(JobId(3092))
7685 Downstairs last five completed: 2810 2809 2808 2807 2806
7686 Upstairs last five completed: 3111 3110 3109 3108 3107
7687 CLIENT: Up:0 ds:301 act:2
76882025-03-17T22:17:53.747ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
76892025-03-17T22:17:54.691ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
76902025-03-17T22:17:54.692ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:38366) = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
76912025-03-17T22:17:54.693ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
76922025-03-17T22:17:54.693ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
76932025-03-17T22:17:54.693ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
76942025-03-17T22:17:54.693ZINFOcrucible: send last flush ID to this DS: Some(JobId(2810)) = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
76952025-03-17T22:17:54.693ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
76962025-03-17T22:17:54.693ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(2810)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
76972025-03-17T22:17:54.694ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(2810)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
7698 ----------------------------------------------------------------
7699 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
7700 Downstairs last five completed: 3112 3111 3110 3109 3108
7701 Upstairs last five completed: 3112 3111 3110 3109 3108
7702 CLIENT: Up:0 ds:0 act:3
7703 Replay: All jobs finished
7704 Main task has work to do, go find it
7705 got dsc Stop(1)
7706 stop 1
7707 Mar 17 22:17:56.934 INFO request completed, latency_us: [8820] Got stop action so:false kr:false
7708 325, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 8d5e51a2-7ac8-41d9-8eb4-dfe39a17844a, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
77092025-03-17T22:17:56.935ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
77102025-03-17T22:17:56.935ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
77112025-03-17T22:17:56.935ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
7712 Mar 17 22:17:56.935 INFO request completed, latency_us: 230, response_code: 200, uri: /state/cid/1, method: GET, req_id: 79cd196b-b68b-4cf1-8d6b-8da9b8943590, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
77132025-03-17T22:17:56.935ZINFOcrucible: client task is exiting = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
7714 [8820] Exited with: ExitStatus(unix_wait_status(9))
7715 [8820][1] reports Exit
7716 Mar 17 22:18:00.937 INFO request completed, latency_us: 217, response_code: 200, uri: /state/cid/1, method: GET, req_id: 369ed0c8-5752-4a3f-b28d-c1e80398a246, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
7717 001/300 Read block 817 len 24576
7718 002/300 Read block 160 len 28672
7719 003/300 Read block 4035 len 8192
7720 004/300 Read block 3787 len 20480
7721 005/300 Read block 4974 len 28672
7722 006/300 Write block 4149 len 4096 data: 3
7723 007/300 Read block 3791 len 20480
7724 008/300 Read block 4175 len 40960
7725 009/300 Read block 3343 len 20480
7726 010/300 Write block 4771 len 32768 data: 2 2 2 2 2 2 2 2
7727 011/300 Flush
7728 012/300 Flush
7729 013/300 Write block 797 len 20480 data: 2 2 2 2 2
7730 014/300 Read block 3716 len 4096
7731 015/300 Write block 3432 len 32768 data: 3 3 3 3 3 3 2 2
7732 016/300 Read block 2591 len 28672
7733 017/300 Write block 1961 len 40960 data: 5 5 6 5 5 3 3 3 3 3
7734 018/300 Read block 2549 len 4096
7735 019/300 Write block 1895 len 8192 data: 2 2
7736 020/300 Write block 2530 len 20480 data: 2 2 2 2 2
7737 021/300 Read block 4245 len 36864
7738 022/300 Write block 4608 len 32768 data: 6 5 5 4 3 3 3 3
7739 023/300 Read block 3309 len 12288
7740 024/300 Read block 2247 len 12288
7741 025/300 Read block 4181 len 4096
7742 026/300 Write block 2538 len 32768 data: 2 2 2 2 2 3 3 3
7743 027/300 Read block 884 len 8192
7744 028/300 Read block 3235 len 8192
7745 029/300 Write block 330 len 36864 data: 3 3 3 5 5 6 6 5 4
7746 030/300 Write block 4516 len 4096 data: 3
7747 031/300 Write block 3887 len 16384 data: 4 3 2 2
7748 032/300 Write block 4210 len 12288 data: 3 4 4
7749 033/300 Read block 266 len 36864
7750 034/300 Write block 4950 len 32768 data: 2 2 2 2 2 2 2 2
7751 035/300 Write block 3224 len 36864 data: 3 2 2 2 2 2 3 4 4
7752 036/300 Read block 4842 len 20480
7753 037/300 Read block 1682 len 16384
7754 038/300 Read block 1911 len 40960
7755 039/300 Write block 3309 len 24576 data: 3 3 2 2 2 2
7756 040/300 Write block 3843 len 16384 data: 3 2 2 2
7757 041/300 Read block 4538 len 12288
7758 042/300 Read block 469 len 4096
7759 043/300 Read block 4034 len 16384
7760 044/300 Write block 873 len 12288 data: 3 3 3
7761 045/300 Read block 4645 len 40960
7762 046/300 Flush
7763 047/300 Read block 3024 len 40960
7764 048/300 Write block 3877 len 8192 data: 2 2
7765 049/300 Write block 4371 len 32768 data: 3 3 3 3 3 3 3 2
7766 050/300 Write block 1672 len 4096 data: 5
7767 051/300 Flush
7768 052/300 Read block 3666 len 40960
7769 053/300 Read block 4213 len 20480
7770 054/300 Read block 2188 len 24576
7771 055/300 Flush
7772 056/300 Flush
7773 057/300 Read block 30 len 12288
7774 058/300 Write block 347 len 8192 data: 2 2
7775 059/300 Read block 4476 len 40960
7776 060/300 Read block 649 len 20480
7777 061/300 Write block 829 len 28672 data: 4 3 3 3 3 2 2
7778 062/300 Read block 4253 len 40960
7779 063/300 Read block 1432 len 16384
7780 064/300 Write block 736 len 20480 data: 2 2 2 2 2
7781 065/300 Write block 2667 len 8192 data: 4 4
7782 066/300 Write block 2417 len 4096 data: 2
7783 067/300 Write block 2450 len 4096 data: 2
7784 068/300 Read block 3104 len 40960
7785 069/300 Read block 973 len 36864
7786 070/300 Write block 346 len 40960 data: 2 3 3 2 2 3 3 3 3 3
7787 071/300 Read block 3431 len 16384
7788 072/300 Read block 4035 len 4096
7789 073/300 Write block 4177 len 24576 data: 2 2 2 2 2 3
7790 074/300 Read block 4749 len 8192
7791 075/300 Write block 1931 len 16384 data: 3 3 2 2
7792 076/300 Flush
7793 077/300 Write block 3483 len 12288 data: 2 2 2
7794 078/300 Write block 3543 len 36864 data: 2 2 2 2 2 2 2 2 2
7795 079/300 Write block 4493 len 4096 data: 3
7796 080/300 Read block 4138 len 36864
7797 081/300 Read block 187 len 24576
7798 082/300 Read block 3606 len 40960
7799 083/300 Flush
7800 084/300 Read block 1139 len 12288
7801 085/300 Write block 2703 len 4096 data: 3
7802 086/300 Read block 3498 len 16384
7803 087/300 Read block 124 len 16384
7804 088/300 Read block 2288 len 36864
7805 089/300 Write block 2126 len 32768 data: 2 2 2 2 2 3 3 3
7806 090/300 Read block 4100 len 28672
7807 091/300 Write block 1391 len 28672 data: 3 3 4 3 3 3 3
7808 092/300 Flush
7809 093/300 Read block 1400 len 36864
7810 094/300 Write block 2072 len 12288 data: 4 4 4
7811 095/300 Read block 4223 len 4096
7812 096/300 Write block 2071 len 32768 data: 3 5 5 5 3 2 2 2
7813 097/300 Read block 2855 len 8192
7814 098/300 Read block 3383 len 36864
7815 099/300 Write block 3449 len 12288 data: 3 2 2
7816 100/300 Read block 773 len 20480
7817 101/300 Write block 237 len 32768 data: 4 3 3 3 3 2 2 2
7818 102/300 Write block 588 len 32768 data: 4 4 4 3 3 2 3 3
7819 103/300 Read block 4852 len 32768
7820 104/300 Read block 4758 len 8192
7821 105/300 Read block 2161 len 40960
7822 106/300 Read block 1575 len 12288
7823 107/300 Write block 3385 len 28672 data: 2 3 4 4 3 3 3
7824 108/300 Flush
7825 109/300 Write block 4859 len 36864 data: 3 4 5 5 5 4 4 4 3
7826 110/300 Read block 3471 len 28672
7827 111/300 Read block 395 len 12288
7828 112/300 Read block 4513 len 12288
7829 113/300 Read block 1085 len 36864
7830 114/300 Write block 589 len 16384 data: 5 5 4 4
7831 115/300 Read block 708 len 16384
7832 116/300 Read block 1789 len 20480
7833 117/300 Read block 373 len 36864
7834 118/300 Read block 4586 len 16384
7835 119/300 Write block 4366 len 32768 data: 2 3 3 3 3 4 4 4
7836 120/300 Write block 951 len 28672 data: 2 2 2 2 2 2 2
7837 121/300 Write block 4822 len 40960 data: 3 2 2 2 2 2 2 2 2 2
7838 122/300 Flush
7839 123/300 Read block 1101 len 40960
7840 124/300 Write block 2326 len 12288 data: 4 3 3
7841 125/300 Write block 443 len 20480 data: 2 2 3 3 3
7842 126/300 Write block 1952 len 20480 data: 3 3 2 2 3
7843 127/300 Read block 38 len 32768
7844 128/300 Write block 4497 len 28672 data: 2 2 2 2 2 2 2
7845 129/300 Read block 2211 len 36864
7846 130/300 Read block 804 len 12288
7847 131/300 Write block 255 len 32768 data: 3 3 3 3 3 3 3 3
7848 132/300 Write block 1423 len 24576 data: 2 2 2 2 2 2
7849 133/300 Read block 4864 len 32768
7850 134/300 Read block 702 len 24576
7851 135/300 Write block 2148 len 8192 data: 4 4
7852 136/300 Flush
7853 137/300 Write block 837 len 24576 data: 2 3 3 3 3 2
7854 138/300 Read block 249 len 32768
7855 139/300 Write block 1807 len 4096 data: 3
7856 140/300 Flush
7857 141/300 Flush
7858 142/300 Write block 2785 len 40960 data: 2 2 2 2 2 2 3 3 3 2
7859 143/300 Read block 4851 len 8192
7860 144/300 Read block 3542 len 24576
7861 145/300 Read block 963 len 20480
7862 146/300 Read block 3440 len 36864
7863 147/300 Read block 3973 len 20480
7864 148/300 Read block 1072 len 36864
7865 149/300 Write block 1613 len 12288 data: 2 2 2
7866 150/300 Read block 1182 len 36864
7867 151/300 Flush
7868 152/300 Read block 3110 len 32768
7869 153/300 Read block 2138 len 32768
7870 154/300 Flush
7871 155/300 Flush
7872 156/300 Read block 1651 len 20480
7873 157/300 Read block 4037 len 36864
7874 158/300 Write block 3293 len 12288 data: 4 2 2
7875 159/300 Read block 3443 len 36864
7876 160/300 Read block 302 len 36864
7877 161/300 Write block 1862 len 12288 data: 3 3 3
7878 162/300 Write block 3612 len 32768 data: 2 3 2 2 2 2 2 2
7879 163/300 Read block 3076 len 4096
7880 164/300 Read block 1545 len 36864
7881 165/300 Read block 2910 len 20480
7882 166/300 Write block 1472 len 40960 data: 3 3 2 2 2 3 3 3 3 4
7883 167/300 Read block 4303 len 16384
7884 168/300 Read block 2211 len 28672
7885 169/300 Write block 4243 len 24576 data: 3 2 2 2 3 3
7886 170/300 Read block 3297 len 36864
7887 171/300 Write block 3558 len 16384 data: 2 3 3 3
7888 172/300 Write block 57 len 36864 data: 2 2 3 3 3 3 3 3 3
7889 173/300 Read block 4166 len 8192
7890 174/300 Read block 4907 len 12288
7891 175/300 Write block 415 len 12288 data: 2 2 2
7892 176/300 Read block 1716 len 4096
7893 177/300 Read block 3343 len 36864
7894 178/300 Read block 2060 len 40960
7895 179/300 Read block 200 len 12288
7896 180/300 Read block 2866 len 24576
7897 181/300 Flush
7898 182/300 Read block 3379 len 36864
7899 183/300 Read block 4101 len 36864
7900 184/300 Write block 4883 len 16384 data: 4 5 4 4
7901 185/300 Flush
7902 186/300 Read block 1578 len 32768
7903 187/300 Write block 1397 len 28672 data: 4 3 3 3 3 3 3
7904 188/300 Write block 744 len 16384 data: 2 2 2 2
7905 189/300 Flush
7906 190/300 Write block 1822 len 4096 data: 2
7907 191/300 Read block 1540 len 16384
7908 192/300 Write block 2933 len 36864 data: 6 6 4 3 4 4 4 4 4
7909 193/300 Read block 2258 len 4096
7910 194/300 Flush
7911 195/300 Read block 504 len 28672
7912 196/300 Read block 4436 len 40960
7913 197/300 Read block 3986 len 40960
7914 198/300 Read block 2564 len 12288
7915 199/300 Read block 4715 len 4096
7916 200/300 Write block 2818 len 28672 data: 3 3 4 4 4 4 5
7917 201/300 Write block 4002 len 12288 data: 3 3 3
7918 202/300 Write block 2700 len 20480 data: 3 2 2 4 4
7919 203/300 Read block 1407 len 16384
7920 204/300 Read block 1994 len 8192
7921 205/300 Read block 3863 len 40960
7922 206/300 Write block 31 len 24576 data: 2 2 2 2 2 3
7923 207/300 Read block 1094 len 20480
7924 208/300 Write block 809 len 32768 data: 3 2 2 2 3 3 3 2
7925 209/300 Write block 4783 len 20480 data: 3 3 3 3 3
7926 210/300 Read block 3155 len 16384
7927 211/300 Read block 1886 len 36864
7928 212/300 Read block 735 len 20480
7929 213/300 Read block 2813 len 16384
7930 214/300 Read block 1822 len 12288
7931 215/300 Read block 3678 len 28672
7932 216/300 Write block 4222 len 8192 data: 2 2
7933 217/300 Read block 1118 len 24576
7934 218/300 Write block 3238 len 28672 data: 4 4 3 3 3 2 2
7935 219/300 Read block 1061 len 28672
7936 220/300 Read block 1648 len 32768
7937 221/300 Write block 3940 len 8192 data: 3 3
7938 222/300 Flush
7939 223/300 Read block 836 len 28672
7940 224/300 Read block 2657 len 32768
7941 225/300 Write block 3512 len 12288 data: 2 2 2
7942 226/300 Write block 3854 len 12288 data: 2 2 2
7943 227/300 Write block 1984 len 36864 data: 2 3 3 2 2 4 4 4 5
7944 228/300 Read block 3871 len 24576
7945 229/300 Read block 3964 len 8192
7946 230/300 Write block 3013 len 24576 data: 3 3 4 4 4 4
7947 231/300 Read block 2972 len 32768
7948 232/300 Write block 2603 len 4096 data: 3
7949 233/300 Write block 1307 len 36864 data: 2 3 3 3 3 3 3 3 3
7950 234/300 Read block 105 len 4096
7951 235/300 Read block 141 len 16384
7952 236/300 Write block 3291 len 36864 data: 4 4 5 3 3 2 2 2 2
7953 237/300 Read block 3091 len 20480
7954 238/300 Read block 4534 len 16384
7955 239/300 Read block 1228 len 36864
7956 240/300 Write block 2958 len 36864 data: 4 4 4 3 2 2 2 3 3
7957 241/300 Write block 2847 len 16384 data: 4 3 2 3
7958 242/300 Write block 4144 len 32768 data: 2 2 3 3 3 4 3 3
7959 243/300 Write block 4873 len 12288 data: 4 4 4
7960 244/300 Read block 1708 len 16384
7961 245/300 Read block 3852 len 4096
7962 246/300 Read block 993 len 4096
7963 247/300 Write block 60 len 40960 data: 4 4 4 4 4 4 3 2 2 2
7964 248/300 Read block 2419 len 40960
7965 249/300 Read block 3037 len 36864
7966 250/300 Write block 2225 len 36864 data: 3 3 2 2 3 3 3 3 3
7967 251/300 Read block 4216 len 24576
7968 252/300 Read block 739 len 20480
7969 253/300 Read block 3433 len 12288
7970 254/300 Write block 3917 len 12288 data: 4 4 3
7971 255/300 Read block 1569 len 8192
7972 256/300 Write block 2265 len 12288 data: 2 2 2
7973 257/300 Read block 3587 len 16384
7974 258/300 Write block 4393 len 8192 data: 3 3
7975 259/300 Flush
7976 260/300 Write block 4886 len 16384 data: 5 3 4 3
7977 261/300 Write block 79 len 8192 data: 4 4
7978 262/300 Read block 647 len 36864
7979 263/300 Read block 2219 len 24576
7980 264/300 Write block 4490 len 4096 data: 3
7981 265/300 Flush
7982 266/300 Write block 3127 len 24576 data: 2 2 3 3 3 3
7983 267/300 Read block 2618 len 20480
7984 268/300 Read block 2547 len 4096
7985 269/300 Write block 4851 len 40960 data: 3 3 2 2 2 2 2 2 4 5
7986 270/300 Read block 4371 len 32768
7987 271/300 Write block 1445 len 8192 data: 2 2
7988 272/300 Read block 1345 len 36864
7989 273/300 Flush
7990 274/300 Read block 3821 len 16384
7991 275/300 Read block 3862 len 12288
7992 276/300 Write block 2175 len 28672 data: 2 2 3 4 3 3 4
7993 277/300 Read block 2528 len 36864
7994 278/300 Write block 2956 len 36864 data: 4 5 5 5 5 4 3 3 3
7995 279/300 Write block 1583 len 32768 data: 3 3 3 3 2 2 2 2
7996 280/300 Read block 699 len 12288
7997 281/300 Read block 1310 len 16384
7998 282/300 Read block 608 len 32768
7999 283/300 Read block 3776 len 8192
8000 284/300 Write block 1041 len 16384 data: 4 4 2 2
8001 285/300 Read block 2482 len 16384
8002 286/300 Read block 4944 len 24576
8003 287/300 Write block 4447 len 40960 data: 2 2 3 3 3 3 2 2 2 2
8004 288/300 Write block 3636 len 40960 data: 2 2 2 2 2 4 4 5 5 4
8005 289/300 Read block 744 len 16384
8006 290/300 Write block 954 len 36864 data: 3 3 3 3 2 3 3 3 3
8007 291/300 Flush
8008 292/300 Read block 958 len 8192
8009 293/300 Read block 4901 len 40960
8010 294/300 Write block 4753 len 32768 data: 3 3 3 4 4 4 4 4
8011 295/300 Read block 4827 len 24576
8012 296/300 Read block 4736 len 36864
8013 297/300 Read block 3671 len 40960
8014 298/300 Read block 3595 len 4096
8015 299/300 Read block 1685 len 4096
8016 300/300 Write block 3432 len 24576 data: 4 4 4 4 4 4
8017 301/300 Read block 844 len 20480
8018 Main task has work to do, go find it
8019 got dsc Start(1)
8020 start 1
8021 Mar 17 22:18:01.198 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/1, method: POST, req_id: 8b6a5571-1253-4508-bd0a-b6538fba39a9, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
8022 Replay: started 1, returned:Ok(())
8023 ----------------------------------------------------------------
8024 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8025 States: Active Offline Active
8026 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8027 3113 Acked Read 6 Done Sent Done false
8028 3114 Acked Read 7 Done Sent Done false
8029 3115 Acked Read 2 Done Sent Done false
8030 3116 Acked Read 5 Done Sent Done false
8031 3117 Acked Read 7 Done Sent Done false
8032 3118 Acked Write 1 Done Sent Done false
8033 3119 Acked Read 5 Done Sent Done false
8034 3120 Acked Read 10 Done Sent Done false
8035 3121 Acked Read 5 Done Sent Done false
8036 3122 Acked Write 8 Done Sent Done false
8037 3123 Acked Flush 0 Done Sent Done false
8038 3124 Acked Flush 0 Done Sent Done false
8039 3125 Acked Write 5 Done Sent Done false
8040 3126 Acked Read 1 Done Sent Done false
8041 3127 Acked Write 8 Done Sent Done false
8042 3128 Acked Read 7 Done Sent Done false
8043 3129 Acked Write 10 Done Sent Done false
8044 3130 Acked Read 1 Done Sent Done false
8045 3131 Acked Write 2 Done Sent Done false
8046 3132 Acked Write 5 Done Sent Done false
8047 3133 Acked Read 9 Done Sent Done false
8048 3134 Acked Write 8 Done Sent Done false
8049 3135 Acked Read 3 Done Sent Done false
8050 3136 Acked Read 3 Done Sent Done false
8051 3137 Acked Read 1 Done Sent Done false
8052 3138 Acked Write 8 Done Sent Done false
8053 3139 Acked Read 2 Done Sent Done false
8054 3140 Acked Read 2 Done Sent Done false
8055 3141 Acked Write 9 Done Sent Done false
8056 3142 Acked Write 1 Done Sent Done false
8057 3143 Acked Write 4 Done Sent Done false
8058 3144 Acked Write 3 Done Sent Done false
8059 3145 Acked Read 9 Done Sent Done false
8060 3146 Acked Write 8 Done Sent Done false
8061 3147 Acked Write 9 Done Sent Done false
8062 3148 Acked Read 5 Done Sent Done false
8063 3149 Acked Read 4 Done Sent Done false
8064 3150 Acked Read 10 Done Sent Done false
8065 3151 Acked Write 6 Done Sent Done false
8066 3152 Acked Write 4 Done Sent Done false
8067 3153 Acked Read 3 Done Sent Done false
8068 3154 Acked Read 1 Done Sent Done false
8069 3155 Acked Read 4 Done Sent Done false
8070 3156 Acked Write 3 Done Sent Done false
8071 3157 Acked Read 10 Done Sent Done false
8072 3158 Acked Flush 0 Done Sent Done false
8073 3159 Acked Read 10 Done Sent Done false
8074 3160 Acked Write 2 Done Sent Done false
8075 3161 Acked Write 8 Done Sent Done false
8076 3162 Acked Write 1 Done Sent Done false
8077 3163 Acked Flush 0 Done Sent Done false
8078 3164 Acked Read 10 Done Sent Done false
8079 3165 Acked Read 5 Done Sent Done false
8080 3166 Acked Read 6 Done Sent Done false
8081 3167 Acked Flush 0 Done Sent Done false
8082 3168 Acked Flush 0 Done Sent Done false
8083 3169 Acked Read 3 Done Sent Done false
8084 3170 Acked Write 2 Done Sent Done false
8085 3171 Acked Read 10 Done Sent Done false
8086 3172 Acked Read 5 Done Sent Done false
8087 3173 Acked Write 7 Done Sent Done false
8088 3174 Acked Read 10 Done Sent Done false
8089 3175 Acked Read 4 Done Sent Done false
8090 3176 Acked Write 5 Done Sent Done false
8091 3177 Acked Write 2 Done Sent Done false
8092 3178 Acked Write 1 Done Sent Done false
8093 3179 Acked Write 1 Done Sent Done false
8094 3180 Acked Read 10 Done Sent Done false
8095 3181 Acked Read 9 Done Sent Done false
8096 3182 Acked Write 10 Done Sent Done false
8097 3183 Acked Read 4 Done Sent Done false
8098 3184 Acked Read 1 Done Sent Done false
8099 3185 Acked Write 6 Done Sent Done false
8100 3186 Acked Read 2 Done Sent Done false
8101 3187 Acked Write 4 Done Sent Done false
8102 3188 Acked Flush 0 Done Sent Done false
8103 3189 Acked Write 3 Done Sent Done false
8104 3190 Acked Write 9 Done Sent Done false
8105 3191 Acked Write 1 Done Sent Done false
8106 3192 Acked Read 9 Done Sent Done false
8107 3193 Acked Read 6 Done Sent Done false
8108 3194 Acked Read 10 Done Sent Done false
8109 3195 Acked Flush 0 Done Sent Done false
8110 3196 Acked Read 3 Done Sent Done false
8111 3197 Acked Write 1 Done Sent Done false
8112 3198 Acked Read 4 Done Sent Done false
8113 3199 Acked Read 4 Done Sent Done false
8114 3200 Acked Read 9 Done Sent Done false
8115 3201 Acked Write 8 Done Sent Done false
8116 3202 Acked Read 7 Done Sent Done false
8117 3203 Acked Write 7 Done Sent Done false
8118 3204 Acked Flush 0 Done Sent Done false
8119 3205 Acked Read 9 Done Sent Done false
8120 3206 Acked Write 3 Done Sent Done false
8121 3207 Acked Read 1 Done Sent Done false
8122 3208 Acked Write 8 Done Sent Done false
8123 3209 Acked Read 2 Done Sent Done false
8124 3210 Acked Read 9 Done Sent Done false
8125 3211 Acked Write 3 Done Sent Done false
8126 3212 Acked Read 5 Done Sent Done false
8127 3213 Acked Write 8 Done Sent Done false
8128 3214 Acked Write 8 Done Sent Done false
8129 3215 Acked Read 8 Done Sent Done false
8130 3216 Acked Read 2 Done Sent Done false
8131 3217 Acked Read 10 Done Sent Done false
8132 3218 Acked Read 3 Done Sent Done false
8133 3219 Acked Write 7 Done Sent Done false
8134 3220 Acked Flush 0 Done Sent Done false
8135 3221 Acked Write 9 Done Sent Done false
8136 3222 Acked Read 7 Done Sent Done false
8137 3223 Acked Read 3 Done Sent Done false
8138 3224 Acked Read 3 Done Sent Done false
8139 3225 Acked Read 9 Done Sent Done false
8140 3226 Acked Write 4 Done Sent Done false
8141 3227 Acked Read 4 Done Sent Done false
8142 3228 Acked Read 5 Done Sent Done false
8143 3229 Acked Read 9 Done Sent Done false
8144 3230 Acked Read 4 Done Sent Done false
8145 3231 Acked Write 8 Done Sent Done false
8146 3232 Acked Write 7 Done Sent Done false
8147 3233 Acked Write 10 Done Sent Done false
8148 3234 Acked Flush 0 Done Sent Done false
8149 3235 Acked Read 10 Done Sent Done false
8150 3236 Acked Write 3 Done Sent Done false
8151 3237 Acked Write 5 Done Sent Done false
8152 3238 Acked Write 5 Done Sent Done false
8153 3239 Acked Read 8 Done Sent Done false
8154 3240 Acked Write 7 Done Sent Done false
8155 3241 Acked Read 9 Done Sent Done false
8156 3242 Acked Read 3 Done Sent Done false
8157 3243 Acked Write 8 Done Sent Done false
8158 3244 Acked Write 6 Done Sent Done false
8159 3245 Acked Read 8 Done Sent Done false
8160 3246 Acked Read 6 Done Sent Done false
8161 3247 Acked Write 2 Done Sent Done false
8162 3248 Acked Flush 0 Done Sent Done false
8163 3249 Acked Write 6 Done Sent Done false
8164 3250 Acked Read 8 Done Sent Done false
8165 3251 Acked Write 1 Done Sent Done false
8166 3252 Acked Flush 0 Done Sent Done false
8167 3253 Acked Flush 0 Done Sent Done false
8168 3254 Acked Write 10 Done Sent Done false
8169 3255 Acked Read 2 Done Sent Done false
8170 3256 Acked Read 6 Done Sent Done false
8171 3257 Acked Read 5 Done Sent Done false
8172 3258 Acked Read 9 Done Sent Done false
8173 3259 Acked Read 5 Done Sent Done false
8174 3260 Acked Read 9 Done Sent Done false
8175 3261 Acked Write 3 Done Sent Done false
8176 3262 Acked Read 9 Done Sent Done false
8177 3263 Acked Flush 0 Done Sent Done false
8178 3264 Acked Read 8 Done Sent Done false
8179 3265 Acked Read 8 Done Sent Done false
8180 3266 Acked Flush 0 Done Sent Done false
8181 3267 Acked Flush 0 Done Sent Done false
8182 3268 Acked Read 5 Done Sent Done false
8183 3269 Acked Read 9 Done Sent Done false
8184 3270 Acked Write 3 Done Sent Done false
8185 3271 Acked Read 9 Done Sent Done false
8186 3272 Acked Read 9 Done Sent Done false
8187 3273 Acked Write 3 Done Sent Done false
8188 3274 Acked Write 8 Done Sent Done false
8189 3275 Acked Read 1 Done Sent Done false
8190 3276 Acked Read 9 Done Sent Done false
8191 3277 Acked Read 5 Done Sent Done false
8192 3278 Acked Write 10 Done Sent Done false
8193 3279 Acked Read 4 Done Sent Done false
8194 3280 Acked Read 7 Done Sent Done false
8195 3281 Acked Write 6 Done Sent Done false
8196 3282 Acked Read 9 Done Sent Done false
8197 3283 Acked Write 4 Done Sent Done false
8198 3284 Acked Write 9 Done Sent Done false
8199 3285 Acked Read 2 Done Sent Done false
8200 3286 Acked Read 3 Done Sent Done false
8201 3287 Acked Write 3 Done Sent Done false
8202 3288 Acked Read 1 Done Sent Done false
8203 3289 Acked Read 9 Done Sent Done false
8204 3290 Acked Read 10 Done Sent Done false
8205 3291 Acked Read 3 Done Sent Done false
8206 3292 Acked Read 6 Done Sent Done false
8207 3293 Acked Flush 0 Done Sent Done false
8208 3294 Acked Read 9 Done Sent Done false
8209 3295 Acked Read 9 Done Sent Done false
8210 3296 Acked Write 4 Done Sent Done false
8211 3297 Acked Flush 0 Done Sent Done false
8212 3298 Acked Read 8 Done Sent Done false
8213 3299 Acked Write 7 Done Sent Done false
8214 3300 Acked Write 4 Done Sent Done false
8215 3301 Acked Flush 0 Done Sent Done false
8216 3302 Acked Write 1 Done Sent Done false
8217 3303 Acked Read 4 Done Sent Done false
8218 3304 Acked Write 9 Done Sent Done false
8219 3305 Acked Read 1 Done Sent Done false
8220 3306 Acked Flush 0 Done Sent Done false
8221 3307 Acked Read 7 Done Sent Done false
8222 3308 Acked Read 10 Done Sent Done false
8223 3309 Acked Read 10 Done Sent Done false
8224 3310 Acked Read 3 Done Sent Done false
8225 3311 Acked Read 1 Done Sent Done false
8226 3312 Acked Write 7 Done Sent Done false
8227 3313 Acked Write 3 Done Sent Done false
8228 3314 Acked Write 5 Done Sent Done false
8229 3315 Acked Read 4 Done Sent Done false
8230 3316 Acked Read 2 Done Sent Done false
8231 3317 Acked Read 10 Done Sent Done false
8232 3318 Acked Write 6 Done Sent Done false
8233 3319 Acked Read 5 Done Sent Done false
8234 3320 Acked Write 8 Done Sent Done false
8235 3321 Acked Write 5 Done Sent Done false
8236 3322 Acked Read 4 Done Sent Done false
8237 3323 Acked Read 9 Done Sent Done false
8238 3324 Acked Read 5 Done Sent Done false
8239 3325 Acked Read 4 Done Sent Done false
8240 3326 Acked Read 3 Done Sent Done false
8241 3327 Acked Read 7 Done Sent Done false
8242 3328 Acked Write 2 Done Sent Done false
8243 3329 Acked Read 6 Done Sent Done false
8244 3330 Acked Write 7 Done Sent Done false
8245 3331 Acked Read 7 Done Sent Done false
8246 3332 Acked Read 8 Done Sent Done false
8247 3333 Acked Write 2 Done Sent Done false
8248 3334 Acked Flush 0 Done Sent Done false
8249 3335 Acked Read 7 Done Sent Done false
8250 3336 Acked Read 8 Done Sent Done false
8251 3337 Acked Write 3 Done Sent Done false
8252 3338 Acked Write 3 Done Sent Done false
8253 3339 Acked Write 9 Done Sent Done false
8254 3340 Acked Read 6 Done Sent Done false
8255 3341 Acked Read 2 Done Sent Done false
8256 3342 Acked Write 6 Done Sent Done false
8257 3343 Acked Read 8 Done Sent Done false
8258 3344 Acked Write 1 Done Sent Done false
8259 3345 Acked Write 9 Done Sent Done false
8260 3346 Acked Read 1 Done Sent Done false
8261 3347 Acked Read 4 Done Sent Done false
8262 3348 Acked Write 9 Done Sent Done false
8263 3349 Acked Read 5 Done Sent Done false
8264 3350 Acked Read 4 Done Sent Done false
8265 3351 Acked Read 9 Done Sent Done false
8266 3352 Acked Write 9 Done Sent Done false
8267 3353 Acked Write 4 Done Sent Done false
8268 3354 Acked Write 8 Done Sent Done false
8269 3355 Acked Write 3 Done Sent Done false
8270 3356 Acked Read 4 Done Sent Done false
8271 3357 Acked Read 1 Done Sent Done false
8272 3358 Acked Read 1 Done Sent Done false
8273 3359 Acked Write 10 Done Sent Done false
8274 3360 Acked Read 10 Done Sent Done false
8275 3361 Acked Read 9 Done Sent Done false
8276 3362 Acked Write 9 Done Sent Done false
8277 3363 Acked Read 6 Done Sent Done false
8278 3364 Acked Read 5 Done Sent Done false
8279 3365 Acked Read 3 Done Sent Done false
8280 3366 Acked Write 3 Done Sent Done false
8281 3367 Acked Read 2 Done Sent Done false
8282 3368 Acked Write 3 Done Sent Done false
8283 3369 Acked Read 4 Done Sent Done false
8284 3370 Acked Write 2 Done Sent Done false
8285 3371 Acked Flush 0 Done Sent Done false
8286 3372 Acked Write 4 Done Sent Done false
8287 3373 Acked Write 2 Done Sent Done false
8288 3374 Acked Read 9 Done Sent Done false
8289 3375 Acked Read 6 Done Sent Done false
8290 3376 Acked Write 1 Done Sent Done false
8291 3377 Acked Flush 0 Done Sent Done false
8292 3378 Acked Write 6 Done Sent Done false
8293 3379 Acked Read 5 Done Sent Done false
8294 3380 Acked Read 1 Done Sent Done false
8295 3381 Acked Write 10 Done Sent Done false
8296 3382 Acked Read 8 Done Sent Done false
8297 3383 Acked Write 2 Done Sent Done false
8298 3384 Acked Read 9 Done Sent Done false
8299 3385 Acked Flush 0 Done Sent Done false
8300 3386 Acked Read 4 Done Sent Done false
8301 3387 Acked Read 3 Done Sent Done false
8302 3388 Acked Write 7 Done Sent Done false
8303 3389 Acked Read 9 Done Sent Done false
8304 3390 Acked Write 9 Done Sent Done false
8305 3391 Acked Write 8 Done Sent Done false
8306 3392 Acked Read 3 Done Sent Done false
8307 3393 Acked Read 4 Done Sent Done false
8308 3394 Acked Read 8 Done Sent Done false
8309 3395 Acked Read 2 Done Sent Done false
8310 3396 Acked Write 4 Done Sent Done false
8311 3397 Acked Read 4 Done Sent Done false
8312 3398 Acked Read 6 Done Sent Done false
8313 3399 Acked Write 10 Done Sent Done false
8314 3400 Acked Write 10 Done Sent Done false
8315 3401 Acked Read 4 Done Sent Done false
8316 3402 Acked Write 9 Done Sent Done false
8317 3403 Acked Flush 0 Done Sent Done false
8318 3404 Acked Read 2 Done Sent Done false
8319 3405 Acked Read 10 Done Sent Done false
8320 3406 Acked Write 8 Done Sent Done false
8321 3407 Acked Read 6 Done Sent Done false
8322 3408 Acked Read 9 Done Sent Done false
8323 3409 Acked Read 10 Done Sent Done false
8324 3410 Acked Read 1 Done Sent Done false
8325 3411 Acked Read 1 Done Sent Done false
8326 3412 Acked Write 6 Done Sent Done false
8327 3413 Acked Read 5 Done Sent Done false
8328 STATES DS:0 DS:1 DS:2 TOTAL
8329 Sent 0 301 0 301
8330 Done 301 0 301 602
8331 Skipped 0 0 0 0
8332 Error 0 0 0 0
8333 Last Flush: Some(JobId(3403)) Some(JobId(3112)) Some(JobId(3403))
8334 Downstairs last five completed: 3112 3111 3110 3109 3108
8335 Upstairs last five completed: 3413 3412 3411 3410 3409
8336 CLIENT: Up:0 ds:301 act:2
83372025-03-17T22:18:01.762ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
83382025-03-17T22:18:02.262ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
83392025-03-17T22:18:02.763ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
83402025-03-17T22:18:03.766ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
8341 ----------------------------------------------------------------
8342 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8343 States: Active Offline Active
8344 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8345 3113 Acked Read 6 Done Sent Done false
8346 3114 Acked Read 7 Done Sent Done false
8347 3115 Acked Read 2 Done Sent Done false
8348 3116 Acked Read 5 Done Sent Done false
8349 3117 Acked Read 7 Done Sent Done false
8350 3118 Acked Write 1 Done Sent Done false
8351 3119 Acked Read 5 Done Sent Done false
8352 3120 Acked Read 10 Done Sent Done false
8353 3121 Acked Read 5 Done Sent Done false
8354 3122 Acked Write 8 Done Sent Done false
8355 3123 Acked Flush 0 Done Sent Done false
8356 3124 Acked Flush 0 Done Sent Done false
8357 3125 Acked Write 5 Done Sent Done false
8358 3126 Acked Read 1 Done Sent Done false
8359 3127 Acked Write 8 Done Sent Done false
8360 3128 Acked Read 7 Done Sent Done false
8361 3129 Acked Write 10 Done Sent Done false
8362 3130 Acked Read 1 Done Sent Done false
8363 3131 Acked Write 2 Done Sent Done false
8364 3132 Acked Write 5 Done Sent Done false
8365 3133 Acked Read 9 Done Sent Done false
8366 3134 Acked Write 8 Done Sent Done false
8367 3135 Acked Read 3 Done Sent Done false
8368 3136 Acked Read 3 Done Sent Done false
8369 3137 Acked Read 1 Done Sent Done false
8370 3138 Acked Write 8 Done Sent Done false
8371 3139 Acked Read 2 Done Sent Done false
8372 3140 Acked Read 2 Done Sent Done false
8373 3141 Acked Write 9 Done Sent Done false
8374 3142 Acked Write 1 Done Sent Done false
8375 3143 Acked Write 4 Done Sent Done false
8376 3144 Acked Write 3 Done Sent Done false
8377 3145 Acked Read 9 Done Sent Done false
8378 3146 Acked Write 8 Done Sent Done false
8379 3147 Acked Write 9 Done Sent Done false
8380 3148 Acked Read 5 Done Sent Done false
8381 3149 Acked Read 4 Done Sent Done false
8382 3150 Acked Read 10 Done Sent Done false
8383 3151 Acked Write 6 Done Sent Done false
8384 3152 Acked Write 4 Done Sent Done false
8385 3153 Acked Read 3 Done Sent Done false
8386 3154 Acked Read 1 Done Sent Done false
8387 3155 Acked Read 4 Done Sent Done false
8388 3156 Acked Write 3 Done Sent Done false
8389 3157 Acked Read 10 Done Sent Done false
8390 3158 Acked Flush 0 Done Sent Done false
8391 3159 Acked Read 10 Done Sent Done false
8392 3160 Acked Write 2 Done Sent Done false
8393 3161 Acked Write 8 Done Sent Done false
8394 3162 Acked Write 1 Done Sent Done false
8395 3163 Acked Flush 0 Done Sent Done false
8396 3164 Acked Read 10 Done Sent Done false
8397 3165 Acked Read 5 Done Sent Done false
8398 3166 Acked Read 6 Done Sent Done false
8399 3167 Acked Flush 0 Done Sent Done false
8400 3168 Acked Flush 0 Done Sent Done false
8401 3169 Acked Read 3 Done Sent Done false
8402 3170 Acked Write 2 Done Sent Done false
8403 3171 Acked Read 10 Done Sent Done false
8404 3172 Acked Read 5 Done Sent Done false
8405 3173 Acked Write 7 Done Sent Done false
8406 3174 Acked Read 10 Done Sent Done false
8407 3175 Acked Read 4 Done Sent Done false
8408 3176 Acked Write 5 Done Sent Done false
8409 3177 Acked Write 2 Done Sent Done false
8410 3178 Acked Write 1 Done Sent Done false
8411 3179 Acked Write 1 Done Sent Done false
8412 3180 Acked Read 10 Done Sent Done false
8413 3181 Acked Read 9 Done Sent Done false
8414 3182 Acked Write 10 Done Sent Done false
8415 3183 Acked Read 4 Done Sent Done false
8416 3184 Acked Read 1 Done Sent Done false
8417 3185 Acked Write 6 Done Sent Done false
8418 3186 Acked Read 2 Done Sent Done false
8419 3187 Acked Write 4 Done Sent Done false
8420 3188 Acked Flush 0 Done Sent Done false
8421 3189 Acked Write 3 Done Sent Done false
8422 3190 Acked Write 9 Done Sent Done false
8423 3191 Acked Write 1 Done Sent Done false
8424 3192 Acked Read 9 Done Sent Done false
8425 3193 Acked Read 6 Done Sent Done false
8426 3194 Acked Read 10 Done Sent Done false
8427 3195 Acked Flush 0 Done Sent Done false
8428 3196 Acked Read 3 Done Sent Done false
8429 3197 Acked Write 1 Done Sent Done false
8430 3198 Acked Read 4 Done Sent Done false
8431 3199 Acked Read 4 Done Sent Done false
8432 3200 Acked Read 9 Done Sent Done false
8433 3201 Acked Write 8 Done Sent Done false
8434 3202 Acked Read 7 Done Sent Done false
8435 3203 Acked Write 7 Done Sent Done false
8436 3204 Acked Flush 0 Done Sent Done false
8437 3205 Acked Read 9 Done Sent Done false
8438 3206 Acked Write 3 Done Sent Done false
8439 3207 Acked Read 1 Done Sent Done false
8440 3208 Acked Write 8 Done Sent Done false
8441 3209 Acked Read 2 Done Sent Done false
8442 3210 Acked Read 9 Done Sent Done false
8443 3211 Acked Write 3 Done Sent Done false
8444 3212 Acked Read 5 Done Sent Done false
8445 3213 Acked Write 8 Done Sent Done false
8446 3214 Acked Write 8 Done Sent Done false
8447 3215 Acked Read 8 Done Sent Done false
8448 3216 Acked Read 2 Done Sent Done false
8449 3217 Acked Read 10 Done Sent Done false
8450 3218 Acked Read 3 Done Sent Done false
8451 3219 Acked Write 7 Done Sent Done false
8452 3220 Acked Flush 0 Done Sent Done false
8453 3221 Acked Write 9 Done Sent Done false
8454 3222 Acked Read 7 Done Sent Done false
8455 3223 Acked Read 3 Done Sent Done false
8456 3224 Acked Read 3 Done Sent Done false
8457 3225 Acked Read 9 Done Sent Done false
8458 3226 Acked Write 4 Done Sent Done false
8459 3227 Acked Read 4 Done Sent Done false
8460 3228 Acked Read 5 Done Sent Done false
8461 3229 Acked Read 9 Done Sent Done false
8462 3230 Acked Read 4 Done Sent Done false
8463 3231 Acked Write 8 Done Sent Done false
8464 3232 Acked Write 7 Done Sent Done false
8465 3233 Acked Write 10 Done Sent Done false
8466 3234 Acked Flush 0 Done Sent Done false
8467 3235 Acked Read 10 Done Sent Done false
8468 3236 Acked Write 3 Done Sent Done false
8469 3237 Acked Write 5 Done Sent Done false
8470 3238 Acked Write 5 Done Sent Done false
8471 3239 Acked Read 8 Done Sent Done false
8472 3240 Acked Write 7 Done Sent Done false
8473 3241 Acked Read 9 Done Sent Done false
8474 3242 Acked Read 3 Done Sent Done false
8475 3243 Acked Write 8 Done Sent Done false
8476 3244 Acked Write 6 Done Sent Done false
8477 3245 Acked Read 8 Done Sent Done false
8478 3246 Acked Read 6 Done Sent Done false
8479 3247 Acked Write 2 Done Sent Done false
8480 3248 Acked Flush 0 Done Sent Done false
8481 3249 Acked Write 6 Done Sent Done false
8482 3250 Acked Read 8 Done Sent Done false
8483 3251 Acked Write 1 Done Sent Done false
8484 3252 Acked Flush 0 Done Sent Done false
8485 3253 Acked Flush 0 Done Sent Done false
8486 3254 Acked Write 10 Done Sent Done false
8487 3255 Acked Read 2 Done Sent Done false
8488 3256 Acked Read 6 Done Sent Done false
8489 3257 Acked Read 5 Done Sent Done false
8490 3258 Acked Read 9 Done Sent Done false
8491 3259 Acked Read 5 Done Sent Done false
8492 3260 Acked Read 9 Done Sent Done false
8493 3261 Acked Write 3 Done Sent Done false
8494 3262 Acked Read 9 Done Sent Done false
8495 3263 Acked Flush 0 Done Sent Done false
8496 3264 Acked Read 8 Done Sent Done false
8497 3265 Acked Read 8 Done Sent Done false
8498 3266 Acked Flush 0 Done Sent Done false
8499 3267 Acked Flush 0 Done Sent Done false
8500 3268 Acked Read 5 Done Sent Done false
8501 3269 Acked Read 9 Done Sent Done false
8502 3270 Acked Write 3 Done Sent Done false
8503 3271 Acked Read 9 Done Sent Done false
8504 3272 Acked Read 9 Done Sent Done false
8505 3273 Acked Write 3 Done Sent Done false
8506 3274 Acked Write 8 Done Sent Done false
8507 3275 Acked Read 1 Done Sent Done false
8508 3276 Acked Read 9 Done Sent Done false
8509 3277 Acked Read 5 Done Sent Done false
8510 3278 Acked Write 10 Done Sent Done false
8511 3279 Acked Read 4 Done Sent Done false
8512 3280 Acked Read 7 Done Sent Done false
8513 3281 Acked Write 6 Done Sent Done false
8514 3282 Acked Read 9 Done Sent Done false
8515 3283 Acked Write 4 Done Sent Done false
8516 3284 Acked Write 9 Done Sent Done false
8517 3285 Acked Read 2 Done Sent Done false
8518 3286 Acked Read 3 Done Sent Done false
8519 3287 Acked Write 3 Done Sent Done false
8520 3288 Acked Read 1 Done Sent Done false
8521 3289 Acked Read 9 Done Sent Done false
8522 3290 Acked Read 10 Done Sent Done false
8523 3291 Acked Read 3 Done Sent Done false
8524 3292 Acked Read 6 Done Sent Done false
8525 3293 Acked Flush 0 Done Sent Done false
8526 3294 Acked Read 9 Done Sent Done false
8527 3295 Acked Read 9 Done Sent Done false
8528 3296 Acked Write 4 Done Sent Done false
8529 3297 Acked Flush 0 Done Sent Done false
8530 3298 Acked Read 8 Done Sent Done false
8531 3299 Acked Write 7 Done Sent Done false
8532 3300 Acked Write 4 Done Sent Done false
8533 3301 Acked Flush 0 Done Sent Done false
8534 3302 Acked Write 1 Done Sent Done false
8535 3303 Acked Read 4 Done Sent Done false
8536 3304 Acked Write 9 Done Sent Done false
8537 3305 Acked Read 1 Done Sent Done false
8538 3306 Acked Flush 0 Done Sent Done false
8539 3307 Acked Read 7 Done Sent Done false
8540 3308 Acked Read 10 Done Sent Done false
8541 3309 Acked Read 10 Done Sent Done false
8542 3310 Acked Read 3 Done Sent Done false
8543 3311 Acked Read 1 Done Sent Done false
8544 3312 Acked Write 7 Done Sent Done false
8545 3313 Acked Write 3 Done Sent Done false
8546 3314 Acked Write 5 Done Sent Done false
8547 3315 Acked Read 4 Done Sent Done false
8548 3316 Acked Read 2 Done Sent Done false
8549 3317 Acked Read 10 Done Sent Done false
8550 3318 Acked Write 6 Done Sent Done false
8551 3319 Acked Read 5 Done Sent Done false
8552 3320 Acked Write 8 Done Sent Done false
8553 3321 Acked Write 5 Done Sent Done false
8554 3322 Acked Read 4 Done Sent Done false
8555 3323 Acked Read 9 Done Sent Done false
8556 3324 Acked Read 5 Done Sent Done false
8557 3325 Acked Read 4 Done Sent Done false
8558 3326 Acked Read 3 Done Sent Done false
8559 3327 Acked Read 7 Done Sent Done false
8560 3328 Acked Write 2 Done Sent Done false
8561 3329 Acked Read 6 Done Sent Done false
8562 3330 Acked Write 7 Done Sent Done false
8563 3331 Acked Read 7 Done Sent Done false
8564 3332 Acked Read 8 Done Sent Done false
8565 3333 Acked Write 2 Done Sent Done false
8566 3334 Acked Flush 0 Done Sent Done false
8567 3335 Acked Read 7 Done Sent Done false
8568 3336 Acked Read 8 Done Sent Done false
8569 3337 Acked Write 3 Done Sent Done false
8570 3338 Acked Write 3 Done Sent Done false
8571 3339 Acked Write 9 Done Sent Done false
8572 3340 Acked Read 6 Done Sent Done false
8573 3341 Acked Read 2 Done Sent Done false
8574 3342 Acked Write 6 Done Sent Done false
8575 3343 Acked Read 8 Done Sent Done false
8576 3344 Acked Write 1 Done Sent Done false
8577 3345 Acked Write 9 Done Sent Done false
8578 3346 Acked Read 1 Done Sent Done false
8579 3347 Acked Read 4 Done Sent Done false
8580 3348 Acked Write 9 Done Sent Done false
8581 3349 Acked Read 5 Done Sent Done false
8582 3350 Acked Read 4 Done Sent Done false
8583 3351 Acked Read 9 Done Sent Done false
8584 3352 Acked Write 9 Done Sent Done false
8585 3353 Acked Write 4 Done Sent Done false
8586 3354 Acked Write 8 Done Sent Done false
8587 3355 Acked Write 3 Done Sent Done false
8588 3356 Acked Read 4 Done Sent Done false
8589 3357 Acked Read 1 Done Sent Done false
8590 3358 Acked Read 1 Done Sent Done false
8591 3359 Acked Write 10 Done Sent Done false
8592 3360 Acked Read 10 Done Sent Done false
8593 3361 Acked Read 9 Done Sent Done false
8594 3362 Acked Write 9 Done Sent Done false
8595 3363 Acked Read 6 Done Sent Done false
8596 3364 Acked Read 5 Done Sent Done false
8597 3365 Acked Read 3 Done Sent Done false
8598 3366 Acked Write 3 Done Sent Done false
8599 3367 Acked Read 2 Done Sent Done false
8600 3368 Acked Write 3 Done Sent Done false
8601 3369 Acked Read 4 Done Sent Done false
8602 3370 Acked Write 2 Done Sent Done false
8603 3371 Acked Flush 0 Done Sent Done false
8604 3372 Acked Write 4 Done Sent Done false
8605 3373 Acked Write 2 Done Sent Done false
8606 3374 Acked Read 9 Done Sent Done false
8607 3375 Acked Read 6 Done Sent Done false
8608 3376 Acked Write 1 Done Sent Done false
8609 3377 Acked Flush 0 Done Sent Done false
8610 3378 Acked Write 6 Done Sent Done false
8611 3379 Acked Read 5 Done Sent Done false
8612 3380 Acked Read 1 Done Sent Done false
8613 3381 Acked Write 10 Done Sent Done false
8614 3382 Acked Read 8 Done Sent Done false
8615 3383 Acked Write 2 Done Sent Done false
8616 3384 Acked Read 9 Done Sent Done false
8617 3385 Acked Flush 0 Done Sent Done false
8618 3386 Acked Read 4 Done Sent Done false
8619 3387 Acked Read 3 Done Sent Done false
8620 3388 Acked Write 7 Done Sent Done false
8621 3389 Acked Read 9 Done Sent Done false
8622 3390 Acked Write 9 Done Sent Done false
8623 3391 Acked Write 8 Done Sent Done false
8624 3392 Acked Read 3 Done Sent Done false
8625 3393 Acked Read 4 Done Sent Done false
8626 3394 Acked Read 8 Done Sent Done false
8627 3395 Acked Read 2 Done Sent Done false
8628 3396 Acked Write 4 Done Sent Done false
8629 3397 Acked Read 4 Done Sent Done false
8630 3398 Acked Read 6 Done Sent Done false
8631 3399 Acked Write 10 Done Sent Done false
8632 3400 Acked Write 10 Done Sent Done false
8633 3401 Acked Read 4 Done Sent Done false
8634 3402 Acked Write 9 Done Sent Done false
8635 3403 Acked Flush 0 Done Sent Done false
8636 3404 Acked Read 2 Done Sent Done false
8637 3405 Acked Read 10 Done Sent Done false
8638 3406 Acked Write 8 Done Sent Done false
8639 3407 Acked Read 6 Done Sent Done false
8640 3408 Acked Read 9 Done Sent Done false
8641 3409 Acked Read 10 Done Sent Done false
8642 3410 Acked Read 1 Done Sent Done false
8643 3411 Acked Read 1 Done Sent Done false
8644 3412 Acked Write 6 Done Sent Done false
8645 3413 Acked Read 5 Done Sent Done false
8646 STATES DS:0 DS:1 DS:2 TOTAL
8647 Sent 0 301 0 301
8648 Done 301 0 301 602
8649 Skipped 0 0 0 0
8650 Error 0 0 0 0
8651 Last Flush: Some(JobId(3403)) Some(JobId(3112)) Some(JobId(3403))
8652 Downstairs last five completed: 3112 3111 3110 3109 3108
8653 Upstairs last five completed: 3413 3412 3411 3410 3409
8654 CLIENT: Up:0 ds:301 act:2
86552025-03-17T22:18:05.769ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
86562025-03-17T22:18:06.770ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
86572025-03-17T22:18:06.937ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
86582025-03-17T22:18:06.937ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
86592025-03-17T22:18:06.937ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
86602025-03-17T22:18:06.937ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
86612025-03-17T22:18:06.937ZINFOcrucible: client task is exiting = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
8662 [8820] got start action
8663 [8820] I am going to restart
8664 Starting downstairs at port 8820
8665 Make output file at "/tmp/dsc/downstairs-8820.txt"
8666 [8820][1] reports Starting
8667 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1214)
8668 [8820][1] reports Running
86692025-03-17T22:18:07.771ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
86702025-03-17T22:18:08.774ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
8671 ----------------------------------------------------------------
8672 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8673 States: Active Offline Active
8674 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8675 3113 Acked Read 6 Done Sent Done false
8676 3114 Acked Read 7 Done Sent Done false
8677 3115 Acked Read 2 Done Sent Done false
8678 3116 Acked Read 5 Done Sent Done false
8679 3117 Acked Read 7 Done Sent Done false
8680 3118 Acked Write 1 Done Sent Done false
8681 3119 Acked Read 5 Done Sent Done false
8682 3120 Acked Read 10 Done Sent Done false
8683 3121 Acked Read 5 Done Sent Done false
8684 3122 Acked Write 8 Done Sent Done false
8685 3123 Acked Flush 0 Done Sent Done false
8686 3124 Acked Flush 0 Done Sent Done false
8687 3125 Acked Write 5 Done Sent Done false
8688 3126 Acked Read 1 Done Sent Done false
8689 3127 Acked Write 8 Done Sent Done false
8690 3128 Acked Read 7 Done Sent Done false
8691 3129 Acked Write 10 Done Sent Done false
8692 3130 Acked Read 1 Done Sent Done false
8693 3131 Acked Write 2 Done Sent Done false
8694 3132 Acked Write 5 Done Sent Done false
8695 3133 Acked Read 9 Done Sent Done false
8696 3134 Acked Write 8 Done Sent Done false
8697 3135 Acked Read 3 Done Sent Done false
8698 3136 Acked Read 3 Done Sent Done false
8699 3137 Acked Read 1 Done Sent Done false
8700 3138 Acked Write 8 Done Sent Done false
8701 3139 Acked Read 2 Done Sent Done false
8702 3140 Acked Read 2 Done Sent Done false
8703 3141 Acked Write 9 Done Sent Done false
8704 3142 Acked Write 1 Done Sent Done false
8705 3143 Acked Write 4 Done Sent Done false
8706 3144 Acked Write 3 Done Sent Done false
8707 3145 Acked Read 9 Done Sent Done false
8708 3146 Acked Write 8 Done Sent Done false
8709 3147 Acked Write 9 Done Sent Done false
8710 3148 Acked Read 5 Done Sent Done false
8711 3149 Acked Read 4 Done Sent Done false
8712 3150 Acked Read 10 Done Sent Done false
8713 3151 Acked Write 6 Done Sent Done false
8714 3152 Acked Write 4 Done Sent Done false
8715 3153 Acked Read 3 Done Sent Done false
8716 3154 Acked Read 1 Done Sent Done false
8717 3155 Acked Read 4 Done Sent Done false
8718 3156 Acked Write 3 Done Sent Done false
8719 3157 Acked Read 10 Done Sent Done false
8720 3158 Acked Flush 0 Done Sent Done false
8721 3159 Acked Read 10 Done Sent Done false
8722 3160 Acked Write 2 Done Sent Done false
8723 3161 Acked Write 8 Done Sent Done false
8724 3162 Acked Write 1 Done Sent Done false
8725 3163 Acked Flush 0 Done Sent Done false
8726 3164 Acked Read 10 Done Sent Done false
8727 3165 Acked Read 5 Done Sent Done false
8728 3166 Acked Read 6 Done Sent Done false
8729 3167 Acked Flush 0 Done Sent Done false
8730 3168 Acked Flush 0 Done Sent Done false
8731 3169 Acked Read 3 Done Sent Done false
8732 3170 Acked Write 2 Done Sent Done false
8733 3171 Acked Read 10 Done Sent Done false
8734 3172 Acked Read 5 Done Sent Done false
8735 3173 Acked Write 7 Done Sent Done false
8736 3174 Acked Read 10 Done Sent Done false
8737 3175 Acked Read 4 Done Sent Done false
8738 3176 Acked Write 5 Done Sent Done false
8739 3177 Acked Write 2 Done Sent Done false
8740 3178 Acked Write 1 Done Sent Done false
8741 3179 Acked Write 1 Done Sent Done false
8742 3180 Acked Read 10 Done Sent Done false
8743 3181 Acked Read 9 Done Sent Done false
8744 3182 Acked Write 10 Done Sent Done false
8745 3183 Acked Read 4 Done Sent Done false
8746 3184 Acked Read 1 Done Sent Done false
8747 3185 Acked Write 6 Done Sent Done false
8748 3186 Acked Read 2 Done Sent Done false
8749 3187 Acked Write 4 Done Sent Done false
8750 3188 Acked Flush 0 Done Sent Done false
8751 3189 Acked Write 3 Done Sent Done false
8752 3190 Acked Write 9 Done Sent Done false
8753 3191 Acked Write 1 Done Sent Done false
8754 3192 Acked Read 9 Done Sent Done false
8755 3193 Acked Read 6 Done Sent Done false
8756 3194 Acked Read 10 Done Sent Done false
8757 3195 Acked Flush 0 Done Sent Done false
8758 3196 Acked Read 3 Done Sent Done false
8759 3197 Acked Write 1 Done Sent Done false
8760 3198 Acked Read 4 Done Sent Done false
8761 3199 Acked Read 4 Done Sent Done false
8762 3200 Acked Read 9 Done Sent Done false
8763 3201 Acked Write 8 Done Sent Done false
8764 3202 Acked Read 7 Done Sent Done false
8765 3203 Acked Write 7 Done Sent Done false
8766 3204 Acked Flush 0 Done Sent Done false
8767 3205 Acked Read 9 Done Sent Done false
8768 3206 Acked Write 3 Done Sent Done false
8769 3207 Acked Read 1 Done Sent Done false
8770 3208 Acked Write 8 Done Sent Done false
8771 3209 Acked Read 2 Done Sent Done false
8772 3210 Acked Read 9 Done Sent Done false
8773 3211 Acked Write 3 Done Sent Done false
8774 3212 Acked Read 5 Done Sent Done false
8775 3213 Acked Write 8 Done Sent Done false
8776 3214 Acked Write 8 Done Sent Done false
8777 3215 Acked Read 8 Done Sent Done false
8778 3216 Acked Read 2 Done Sent Done false
8779 3217 Acked Read 10 Done Sent Done false
8780 3218 Acked Read 3 Done Sent Done false
8781 3219 Acked Write 7 Done Sent Done false
8782 3220 Acked Flush 0 Done Sent Done false
8783 3221 Acked Write 9 Done Sent Done false
8784 3222 Acked Read 7 Done Sent Done false
8785 3223 Acked Read 3 Done Sent Done false
8786 3224 Acked Read 3 Done Sent Done false
8787 3225 Acked Read 9 Done Sent Done false
8788 3226 Acked Write 4 Done Sent Done false
8789 3227 Acked Read 4 Done Sent Done false
8790 3228 Acked Read 5 Done Sent Done false
8791 3229 Acked Read 9 Done Sent Done false
8792 3230 Acked Read 4 Done Sent Done false
8793 3231 Acked Write 8 Done Sent Done false
8794 3232 Acked Write 7 Done Sent Done false
8795 3233 Acked Write 10 Done Sent Done false
8796 3234 Acked Flush 0 Done Sent Done false
8797 3235 Acked Read 10 Done Sent Done false
8798 3236 Acked Write 3 Done Sent Done false
8799 3237 Acked Write 5 Done Sent Done false
8800 3238 Acked Write 5 Done Sent Done false
8801 3239 Acked Read 8 Done Sent Done false
8802 3240 Acked Write 7 Done Sent Done false
8803 3241 Acked Read 9 Done Sent Done false
8804 3242 Acked Read 3 Done Sent Done false
8805 3243 Acked Write 8 Done Sent Done false
8806 3244 Acked Write 6 Done Sent Done false
8807 3245 Acked Read 8 Done Sent Done false
8808 3246 Acked Read 6 Done Sent Done false
8809 3247 Acked Write 2 Done Sent Done false
8810 3248 Acked Flush 0 Done Sent Done false
8811 3249 Acked Write 6 Done Sent Done false
8812 3250 Acked Read 8 Done Sent Done false
8813 3251 Acked Write 1 Done Sent Done false
8814 3252 Acked Flush 0 Done Sent Done false
8815 3253 Acked Flush 0 Done Sent Done false
8816 3254 Acked Write 10 Done Sent Done false
8817 3255 Acked Read 2 Done Sent Done false
8818 3256 Acked Read 6 Done Sent Done false
8819 3257 Acked Read 5 Done Sent Done false
8820 3258 Acked Read 9 Done Sent Done false
8821 3259 Acked Read 5 Done Sent Done false
8822 3260 Acked Read 9 Done Sent Done false
8823 3261 Acked Write 3 Done Sent Done false
8824 3262 Acked Read 9 Done Sent Done false
8825 3263 Acked Flush 0 Done Sent Done false
8826 3264 Acked Read 8 Done Sent Done false
8827 3265 Acked Read 8 Done Sent Done false
8828 3266 Acked Flush 0 Done Sent Done false
8829 3267 Acked Flush 0 Done Sent Done false
8830 3268 Acked Read 5 Done Sent Done false
8831 3269 Acked Read 9 Done Sent Done false
8832 3270 Acked Write 3 Done Sent Done false
8833 3271 Acked Read 9 Done Sent Done false
8834 3272 Acked Read 9 Done Sent Done false
8835 3273 Acked Write 3 Done Sent Done false
8836 3274 Acked Write 8 Done Sent Done false
8837 3275 Acked Read 1 Done Sent Done false
8838 3276 Acked Read 9 Done Sent Done false
8839 3277 Acked Read 5 Done Sent Done false
8840 3278 Acked Write 10 Done Sent Done false
8841 3279 Acked Read 4 Done Sent Done false
8842 3280 Acked Read 7 Done Sent Done false
8843 3281 Acked Write 6 Done Sent Done false
8844 3282 Acked Read 9 Done Sent Done false
8845 3283 Acked Write 4 Done Sent Done false
8846 3284 Acked Write 9 Done Sent Done false
8847 3285 Acked Read 2 Done Sent Done false
8848 3286 Acked Read 3 Done Sent Done false
8849 3287 Acked Write 3 Done Sent Done false
8850 3288 Acked Read 1 Done Sent Done false
8851 3289 Acked Read 9 Done Sent Done false
8852 3290 Acked Read 10 Done Sent Done false
8853 3291 Acked Read 3 Done Sent Done false
8854 3292 Acked Read 6 Done Sent Done false
8855 3293 Acked Flush 0 Done Sent Done false
8856 3294 Acked Read 9 Done Sent Done false
8857 3295 Acked Read 9 Done Sent Done false
8858 3296 Acked Write 4 Done Sent Done false
8859 3297 Acked Flush 0 Done Sent Done false
8860 3298 Acked Read 8 Done Sent Done false
8861 3299 Acked Write 7 Done Sent Done false
8862 3300 Acked Write 4 Done Sent Done false
8863 3301 Acked Flush 0 Done Sent Done false
8864 3302 Acked Write 1 Done Sent Done false
8865 3303 Acked Read 4 Done Sent Done false
8866 3304 Acked Write 9 Done Sent Done false
8867 3305 Acked Read 1 Done Sent Done false
8868 3306 Acked Flush 0 Done Sent Done false
8869 3307 Acked Read 7 Done Sent Done false
8870 3308 Acked Read 10 Done Sent Done false
8871 3309 Acked Read 10 Done Sent Done false
8872 3310 Acked Read 3 Done Sent Done false
8873 3311 Acked Read 1 Done Sent Done false
8874 3312 Acked Write 7 Done Sent Done false
8875 3313 Acked Write 3 Done Sent Done false
8876 3314 Acked Write 5 Done Sent Done false
8877 3315 Acked Read 4 Done Sent Done false
8878 3316 Acked Read 2 Done Sent Done false
8879 3317 Acked Read 10 Done Sent Done false
8880 3318 Acked Write 6 Done Sent Done false
8881 3319 Acked Read 5 Done Sent Done false
8882 3320 Acked Write 8 Done Sent Done false
8883 3321 Acked Write 5 Done Sent Done false
8884 3322 Acked Read 4 Done Sent Done false
8885 3323 Acked Read 9 Done Sent Done false
8886 3324 Acked Read 5 Done Sent Done false
8887 3325 Acked Read 4 Done Sent Done false
8888 3326 Acked Read 3 Done Sent Done false
8889 3327 Acked Read 7 Done Sent Done false
8890 3328 Acked Write 2 Done Sent Done false
8891 3329 Acked Read 6 Done Sent Done false
8892 3330 Acked Write 7 Done Sent Done false
8893 3331 Acked Read 7 Done Sent Done false
8894 3332 Acked Read 8 Done Sent Done false
8895 3333 Acked Write 2 Done Sent Done false
8896 3334 Acked Flush 0 Done Sent Done false
8897 3335 Acked Read 7 Done Sent Done false
8898 3336 Acked Read 8 Done Sent Done false
8899 3337 Acked Write 3 Done Sent Done false
8900 3338 Acked Write 3 Done Sent Done false
8901 3339 Acked Write 9 Done Sent Done false
8902 3340 Acked Read 6 Done Sent Done false
8903 3341 Acked Read 2 Done Sent Done false
8904 3342 Acked Write 6 Done Sent Done false
8905 3343 Acked Read 8 Done Sent Done false
8906 3344 Acked Write 1 Done Sent Done false
8907 3345 Acked Write 9 Done Sent Done false
8908 3346 Acked Read 1 Done Sent Done false
8909 3347 Acked Read 4 Done Sent Done false
8910 3348 Acked Write 9 Done Sent Done false
8911 3349 Acked Read 5 Done Sent Done false
8912 3350 Acked Read 4 Done Sent Done false
8913 3351 Acked Read 9 Done Sent Done false
8914 3352 Acked Write 9 Done Sent Done false
8915 3353 Acked Write 4 Done Sent Done false
8916 3354 Acked Write 8 Done Sent Done false
8917 3355 Acked Write 3 Done Sent Done false
8918 3356 Acked Read 4 Done Sent Done false
8919 3357 Acked Read 1 Done Sent Done false
8920 3358 Acked Read 1 Done Sent Done false
8921 3359 Acked Write 10 Done Sent Done false
8922 3360 Acked Read 10 Done Sent Done false
8923 3361 Acked Read 9 Done Sent Done false
8924 3362 Acked Write 9 Done Sent Done false
8925 3363 Acked Read 6 Done Sent Done false
8926 3364 Acked Read 5 Done Sent Done false
8927 3365 Acked Read 3 Done Sent Done false
8928 3366 Acked Write 3 Done Sent Done false
8929 3367 Acked Read 2 Done Sent Done false
8930 3368 Acked Write 3 Done Sent Done false
8931 3369 Acked Read 4 Done Sent Done false
8932 3370 Acked Write 2 Done Sent Done false
8933 3371 Acked Flush 0 Done Sent Done false
8934 3372 Acked Write 4 Done Sent Done false
8935 3373 Acked Write 2 Done Sent Done false
8936 3374 Acked Read 9 Done Sent Done false
8937 3375 Acked Read 6 Done Sent Done false
8938 3376 Acked Write 1 Done Sent Done false
8939 3377 Acked Flush 0 Done Sent Done false
8940 3378 Acked Write 6 Done Sent Done false
8941 3379 Acked Read 5 Done Sent Done false
8942 3380 Acked Read 1 Done Sent Done false
8943 3381 Acked Write 10 Done Sent Done false
8944 3382 Acked Read 8 Done Sent Done false
8945 3383 Acked Write 2 Done Sent Done false
8946 3384 Acked Read 9 Done Sent Done false
8947 3385 Acked Flush 0 Done Sent Done false
8948 3386 Acked Read 4 Done Sent Done false
8949 3387 Acked Read 3 Done Sent Done false
8950 3388 Acked Write 7 Done Sent Done false
8951 3389 Acked Read 9 Done Sent Done false
8952 3390 Acked Write 9 Done Sent Done false
8953 3391 Acked Write 8 Done Sent Done false
8954 3392 Acked Read 3 Done Sent Done false
8955 3393 Acked Read 4 Done Sent Done false
8956 3394 Acked Read 8 Done Sent Done false
8957 3395 Acked Read 2 Done Sent Done false
8958 3396 Acked Write 4 Done Sent Done false
8959 3397 Acked Read 4 Done Sent Done false
8960 3398 Acked Read 6 Done Sent Done false
8961 3399 Acked Write 10 Done Sent Done false
8962 3400 Acked Write 10 Done Sent Done false
8963 3401 Acked Read 4 Done Sent Done false
8964 3402 Acked Write 9 Done Sent Done false
8965 3403 Acked Flush 0 Done Sent Done false
8966 3404 Acked Read 2 Done Sent Done false
8967 3405 Acked Read 10 Done Sent Done false
8968 3406 Acked Write 8 Done Sent Done false
8969 3407 Acked Read 6 Done Sent Done false
8970 3408 Acked Read 9 Done Sent Done false
8971 3409 Acked Read 10 Done Sent Done false
8972 3410 Acked Read 1 Done Sent Done false
8973 3411 Acked Read 1 Done Sent Done false
8974 3412 Acked Write 6 Done Sent Done false
8975 3413 Acked Read 5 Done Sent Done false
8976 STATES DS:0 DS:1 DS:2 TOTAL
8977 Sent 0 301 0 301
8978 Done 301 0 301 602
8979 Skipped 0 0 0 0
8980 Error 0 0 0 0
8981 Last Flush: Some(JobId(3403)) Some(JobId(3112)) Some(JobId(3403))
8982 Downstairs last five completed: 3112 3111 3110 3109 3108
8983 Upstairs last five completed: 3413 3412 3411 3410 3409
8984 CLIENT: Up:0 ds:301 act:2
89852025-03-17T22:18:10.776ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
89862025-03-17T22:18:11.778ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
89872025-03-17T22:18:12.780ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
8988 ----------------------------------------------------------------
8989 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8990 States: Active Offline Active
8991 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
8992 3113 Acked Read 6 Done Sent Done false
8993 3114 Acked Read 7 Done Sent Done false
8994 3115 Acked Read 2 Done Sent Done false
8995 3116 Acked Read 5 Done Sent Done false
8996 3117 Acked Read 7 Done Sent Done false
8997 3118 Acked Write 1 Done Sent Done false
8998 3119 Acked Read 5 Done Sent Done false
8999 3120 Acked Read 10 Done Sent Done false
9000 3121 Acked Read 5 Done Sent Done false
9001 3122 Acked Write 8 Done Sent Done false
9002 3123 Acked Flush 0 Done Sent Done false
9003 3124 Acked Flush 0 Done Sent Done false
9004 3125 Acked Write 5 Done Sent Done false
9005 3126 Acked Read 1 Done Sent Done false
9006 3127 Acked Write 8 Done Sent Done false
9007 3128 Acked Read 7 Done Sent Done false
9008 3129 Acked Write 10 Done Sent Done false
9009 3130 Acked Read 1 Done Sent Done false
9010 3131 Acked Write 2 Done Sent Done false
9011 3132 Acked Write 5 Done Sent Done false
9012 3133 Acked Read 9 Done Sent Done false
9013 3134 Acked Write 8 Done Sent Done false
9014 3135 Acked Read 3 Done Sent Done false
9015 3136 Acked Read 3 Done Sent Done false
9016 3137 Acked Read 1 Done Sent Done false
9017 3138 Acked Write 8 Done Sent Done false
9018 3139 Acked Read 2 Done Sent Done false
9019 3140 Acked Read 2 Done Sent Done false
9020 3141 Acked Write 9 Done Sent Done false
9021 3142 Acked Write 1 Done Sent Done false
9022 3143 Acked Write 4 Done Sent Done false
9023 3144 Acked Write 3 Done Sent Done false
9024 3145 Acked Read 9 Done Sent Done false
9025 3146 Acked Write 8 Done Sent Done false
9026 3147 Acked Write 9 Done Sent Done false
9027 3148 Acked Read 5 Done Sent Done false
9028 3149 Acked Read 4 Done Sent Done false
9029 3150 Acked Read 10 Done Sent Done false
9030 3151 Acked Write 6 Done Sent Done false
9031 3152 Acked Write 4 Done Sent Done false
9032 3153 Acked Read 3 Done Sent Done false
9033 3154 Acked Read 1 Done Sent Done false
9034 3155 Acked Read 4 Done Sent Done false
9035 3156 Acked Write 3 Done Sent Done false
9036 3157 Acked Read 10 Done Sent Done false
9037 3158 Acked Flush 0 Done Sent Done false
9038 3159 Acked Read 10 Done Sent Done false
9039 3160 Acked Write 2 Done Sent Done false
9040 3161 Acked Write 8 Done Sent Done false
9041 3162 Acked Write 1 Done Sent Done false
9042 3163 Acked Flush 0 Done Sent Done false
9043 3164 Acked Read 10 Done Sent Done false
9044 3165 Acked Read 5 Done Sent Done false
9045 3166 Acked Read 6 Done Sent Done false
9046 3167 Acked Flush 0 Done Sent Done false
9047 3168 Acked Flush 0 Done Sent Done false
9048 3169 Acked Read 3 Done Sent Done false
9049 3170 Acked Write 2 Done Sent Done false
9050 3171 Acked Read 10 Done Sent Done false
9051 3172 Acked Read 5 Done Sent Done false
9052 3173 Acked Write 7 Done Sent Done false
9053 3174 Acked Read 10 Done Sent Done false
9054 3175 Acked Read 4 Done Sent Done false
9055 3176 Acked Write 5 Done Sent Done false
9056 3177 Acked Write 2 Done Sent Done false
9057 3178 Acked Write 1 Done Sent Done false
9058 3179 Acked Write 1 Done Sent Done false
9059 3180 Acked Read 10 Done Sent Done false
9060 3181 Acked Read 9 Done Sent Done false
9061 3182 Acked Write 10 Done Sent Done false
9062 3183 Acked Read 4 Done Sent Done false
9063 3184 Acked Read 1 Done Sent Done false
9064 3185 Acked Write 6 Done Sent Done false
9065 3186 Acked Read 2 Done Sent Done false
9066 3187 Acked Write 4 Done Sent Done false
9067 3188 Acked Flush 0 Done Sent Done false
9068 3189 Acked Write 3 Done Sent Done false
9069 3190 Acked Write 9 Done Sent Done false
9070 3191 Acked Write 1 Done Sent Done false
9071 3192 Acked Read 9 Done Sent Done false
9072 3193 Acked Read 6 Done Sent Done false
9073 3194 Acked Read 10 Done Sent Done false
9074 3195 Acked Flush 0 Done Sent Done false
9075 3196 Acked Read 3 Done Sent Done false
9076 3197 Acked Write 1 Done Sent Done false
9077 3198 Acked Read 4 Done Sent Done false
9078 3199 Acked Read 4 Done Sent Done false
9079 3200 Acked Read 9 Done Sent Done false
9080 3201 Acked Write 8 Done Sent Done false
9081 3202 Acked Read 7 Done Sent Done false
9082 3203 Acked Write 7 Done Sent Done false
9083 3204 Acked Flush 0 Done Sent Done false
9084 3205 Acked Read 9 Done Sent Done false
9085 3206 Acked Write 3 Done Sent Done false
9086 3207 Acked Read 1 Done Sent Done false
9087 3208 Acked Write 8 Done Sent Done false
9088 3209 Acked Read 2 Done Sent Done false
9089 3210 Acked Read 9 Done Sent Done false
9090 3211 Acked Write 3 Done Sent Done false
9091 3212 Acked Read 5 Done Sent Done false
9092 3213 Acked Write 8 Done Sent Done false
9093 3214 Acked Write 8 Done Sent Done false
9094 3215 Acked Read 8 Done Sent Done false
9095 3216 Acked Read 2 Done Sent Done false
9096 3217 Acked Read 10 Done Sent Done false
9097 3218 Acked Read 3 Done Sent Done false
9098 3219 Acked Write 7 Done Sent Done false
9099 3220 Acked Flush 0 Done Sent Done false
9100 3221 Acked Write 9 Done Sent Done false
9101 3222 Acked Read 7 Done Sent Done false
9102 3223 Acked Read 3 Done Sent Done false
9103 3224 Acked Read 3 Done Sent Done false
9104 3225 Acked Read 9 Done Sent Done false
9105 3226 Acked Write 4 Done Sent Done false
9106 3227 Acked Read 4 Done Sent Done false
9107 3228 Acked Read 5 Done Sent Done false
9108 3229 Acked Read 9 Done Sent Done false
9109 3230 Acked Read 4 Done Sent Done false
9110 3231 Acked Write 8 Done Sent Done false
9111 3232 Acked Write 7 Done Sent Done false
9112 3233 Acked Write 10 Done Sent Done false
9113 3234 Acked Flush 0 Done Sent Done false
9114 3235 Acked Read 10 Done Sent Done false
9115 3236 Acked Write 3 Done Sent Done false
9116 3237 Acked Write 5 Done Sent Done false
9117 3238 Acked Write 5 Done Sent Done false
9118 3239 Acked Read 8 Done Sent Done false
9119 3240 Acked Write 7 Done Sent Done false
9120 3241 Acked Read 9 Done Sent Done false
9121 3242 Acked Read 3 Done Sent Done false
9122 3243 Acked Write 8 Done Sent Done false
9123 3244 Acked Write 6 Done Sent Done false
9124 3245 Acked Read 8 Done Sent Done false
9125 3246 Acked Read 6 Done Sent Done false
9126 3247 Acked Write 2 Done Sent Done false
9127 3248 Acked Flush 0 Done Sent Done false
9128 3249 Acked Write 6 Done Sent Done false
9129 3250 Acked Read 8 Done Sent Done false
9130 3251 Acked Write 1 Done Sent Done false
9131 3252 Acked Flush 0 Done Sent Done false
9132 3253 Acked Flush 0 Done Sent Done false
9133 3254 Acked Write 10 Done Sent Done false
9134 3255 Acked Read 2 Done Sent Done false
9135 3256 Acked Read 6 Done Sent Done false
9136 3257 Acked Read 5 Done Sent Done false
9137 3258 Acked Read 9 Done Sent Done false
9138 3259 Acked Read 5 Done Sent Done false
9139 3260 Acked Read 9 Done Sent Done false
9140 3261 Acked Write 3 Done Sent Done false
9141 3262 Acked Read 9 Done Sent Done false
9142 3263 Acked Flush 0 Done Sent Done false
9143 3264 Acked Read 8 Done Sent Done false
9144 3265 Acked Read 8 Done Sent Done false
9145 3266 Acked Flush 0 Done Sent Done false
9146 3267 Acked Flush 0 Done Sent Done false
9147 3268 Acked Read 5 Done Sent Done false
9148 3269 Acked Read 9 Done Sent Done false
9149 3270 Acked Write 3 Done Sent Done false
9150 3271 Acked Read 9 Done Sent Done false
9151 3272 Acked Read 9 Done Sent Done false
9152 3273 Acked Write 3 Done Sent Done false
9153 3274 Acked Write 8 Done Sent Done false
9154 3275 Acked Read 1 Done Sent Done false
9155 3276 Acked Read 9 Done Sent Done false
9156 3277 Acked Read 5 Done Sent Done false
9157 3278 Acked Write 10 Done Sent Done false
9158 3279 Acked Read 4 Done Sent Done false
9159 3280 Acked Read 7 Done Sent Done false
9160 3281 Acked Write 6 Done Sent Done false
9161 3282 Acked Read 9 Done Sent Done false
9162 3283 Acked Write 4 Done Sent Done false
9163 3284 Acked Write 9 Done Sent Done false
9164 3285 Acked Read 2 Done Sent Done false
9165 3286 Acked Read 3 Done Sent Done false
9166 3287 Acked Write 3 Done Sent Done false
9167 3288 Acked Read 1 Done Sent Done false
9168 3289 Acked Read 9 Done Sent Done false
9169 3290 Acked Read 10 Done Sent Done false
9170 3291 Acked Read 3 Done Sent Done false
9171 3292 Acked Read 6 Done Sent Done false
9172 3293 Acked Flush 0 Done Sent Done false
9173 3294 Acked Read 9 Done Sent Done false
9174 3295 Acked Read 9 Done Sent Done false
9175 3296 Acked Write 4 Done Sent Done false
9176 3297 Acked Flush 0 Done Sent Done false
9177 3298 Acked Read 8 Done Sent Done false
9178 3299 Acked Write 7 Done Sent Done false
9179 3300 Acked Write 4 Done Sent Done false
9180 3301 Acked Flush 0 Done Sent Done false
9181 3302 Acked Write 1 Done Sent Done false
9182 3303 Acked Read 4 Done Sent Done false
9183 3304 Acked Write 9 Done Sent Done false
9184 3305 Acked Read 1 Done Sent Done false
9185 3306 Acked Flush 0 Done Sent Done false
9186 3307 Acked Read 7 Done Sent Done false
9187 3308 Acked Read 10 Done Sent Done false
9188 3309 Acked Read 10 Done Sent Done false
9189 3310 Acked Read 3 Done Sent Done false
9190 3311 Acked Read 1 Done Sent Done false
9191 3312 Acked Write 7 Done Sent Done false
9192 3313 Acked Write 3 Done Sent Done false
9193 3314 Acked Write 5 Done Sent Done false
9194 3315 Acked Read 4 Done Sent Done false
9195 3316 Acked Read 2 Done Sent Done false
9196 3317 Acked Read 10 Done Sent Done false
9197 3318 Acked Write 6 Done Sent Done false
9198 3319 Acked Read 5 Done Sent Done false
9199 3320 Acked Write 8 Done Sent Done false
9200 3321 Acked Write 5 Done Sent Done false
9201 3322 Acked Read 4 Done Sent Done false
9202 3323 Acked Read 9 Done Sent Done false
9203 3324 Acked Read 5 Done Sent Done false
9204 3325 Acked Read 4 Done Sent Done false
9205 3326 Acked Read 3 Done Sent Done false
9206 3327 Acked Read 7 Done Sent Done false
9207 3328 Acked Write 2 Done Sent Done false
9208 3329 Acked Read 6 Done Sent Done false
9209 3330 Acked Write 7 Done Sent Done false
9210 3331 Acked Read 7 Done Sent Done false
9211 3332 Acked Read 8 Done Sent Done false
9212 3333 Acked Write 2 Done Sent Done false
9213 3334 Acked Flush 0 Done Sent Done false
9214 3335 Acked Read 7 Done Sent Done false
9215 3336 Acked Read 8 Done Sent Done false
9216 3337 Acked Write 3 Done Sent Done false
9217 3338 Acked Write 3 Done Sent Done false
9218 3339 Acked Write 9 Done Sent Done false
9219 3340 Acked Read 6 Done Sent Done false
9220 3341 Acked Read 2 Done Sent Done false
9221 3342 Acked Write 6 Done Sent Done false
9222 3343 Acked Read 8 Done Sent Done false
9223 3344 Acked Write 1 Done Sent Done false
9224 3345 Acked Write 9 Done Sent Done false
9225 3346 Acked Read 1 Done Sent Done false
9226 3347 Acked Read 4 Done Sent Done false
9227 3348 Acked Write 9 Done Sent Done false
9228 3349 Acked Read 5 Done Sent Done false
9229 3350 Acked Read 4 Done Sent Done false
9230 3351 Acked Read 9 Done Sent Done false
9231 3352 Acked Write 9 Done Sent Done false
9232 3353 Acked Write 4 Done Sent Done false
9233 3354 Acked Write 8 Done Sent Done false
9234 3355 Acked Write 3 Done Sent Done false
9235 3356 Acked Read 4 Done Sent Done false
9236 3357 Acked Read 1 Done Sent Done false
9237 3358 Acked Read 1 Done Sent Done false
9238 3359 Acked Write 10 Done Sent Done false
9239 3360 Acked Read 10 Done Sent Done false
9240 3361 Acked Read 9 Done Sent Done false
9241 3362 Acked Write 9 Done Sent Done false
9242 3363 Acked Read 6 Done Sent Done false
9243 3364 Acked Read 5 Done Sent Done false
9244 3365 Acked Read 3 Done Sent Done false
9245 3366 Acked Write 3 Done Sent Done false
9246 3367 Acked Read 2 Done Sent Done false
9247 3368 Acked Write 3 Done Sent Done false
9248 3369 Acked Read 4 Done Sent Done false
9249 3370 Acked Write 2 Done Sent Done false
9250 3371 Acked Flush 0 Done Sent Done false
9251 3372 Acked Write 4 Done Sent Done false
9252 3373 Acked Write 2 Done Sent Done false
9253 3374 Acked Read 9 Done Sent Done false
9254 3375 Acked Read 6 Done Sent Done false
9255 3376 Acked Write 1 Done Sent Done false
9256 3377 Acked Flush 0 Done Sent Done false
9257 3378 Acked Write 6 Done Sent Done false
9258 3379 Acked Read 5 Done Sent Done false
9259 3380 Acked Read 1 Done Sent Done false
9260 3381 Acked Write 10 Done Sent Done false
9261 3382 Acked Read 8 Done Sent Done false
9262 3383 Acked Write 2 Done Sent Done false
9263 3384 Acked Read 9 Done Sent Done false
9264 3385 Acked Flush 0 Done Sent Done false
9265 3386 Acked Read 4 Done Sent Done false
9266 3387 Acked Read 3 Done Sent Done false
9267 3388 Acked Write 7 Done Sent Done false
9268 3389 Acked Read 9 Done Sent Done false
9269 3390 Acked Write 9 Done Sent Done false
9270 3391 Acked Write 8 Done Sent Done false
9271 3392 Acked Read 3 Done Sent Done false
9272 3393 Acked Read 4 Done Sent Done false
9273 3394 Acked Read 8 Done Sent Done false
9274 3395 Acked Read 2 Done Sent Done false
9275 3396 Acked Write 4 Done Sent Done false
9276 3397 Acked Read 4 Done Sent Done false
9277 3398 Acked Read 6 Done Sent Done false
9278 3399 Acked Write 10 Done Sent Done false
9279 3400 Acked Write 10 Done Sent Done false
9280 3401 Acked Read 4 Done Sent Done false
9281 3402 Acked Write 9 Done Sent Done false
9282 3403 Acked Flush 0 Done Sent Done false
9283 3404 Acked Read 2 Done Sent Done false
9284 3405 Acked Read 10 Done Sent Done false
9285 3406 Acked Write 8 Done Sent Done false
9286 3407 Acked Read 6 Done Sent Done false
9287 3408 Acked Read 9 Done Sent Done false
9288 3409 Acked Read 10 Done Sent Done false
9289 3410 Acked Read 1 Done Sent Done false
9290 3411 Acked Read 1 Done Sent Done false
9291 3412 Acked Write 6 Done Sent Done false
9292 3413 Acked Read 5 Done Sent Done false
9293 STATES DS:0 DS:1 DS:2 TOTAL
9294 Sent 0 301 0 301
9295 Done 301 0 301 602
9296 Skipped 0 0 0 0
9297 Error 0 0 0 0
9298 Last Flush: Some(JobId(3403)) Some(JobId(3112)) Some(JobId(3403))
9299 Downstairs last five completed: 3112 3111 3110 3109 3108
9300 Upstairs last five completed: 3413 3412 3411 3410 3409
9301 CLIENT: Up:0 ds:301 act:2
93022025-03-17T22:18:13.782ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
93032025-03-17T22:18:15.786ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
93042025-03-17T22:18:16.787ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
93052025-03-17T22:18:16.938ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
93062025-03-17T22:18:16.938ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:32849) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
93072025-03-17T22:18:16.939ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
93082025-03-17T22:18:16.940ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
93092025-03-17T22:18:16.940ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
93102025-03-17T22:18:16.940ZINFOcrucible: send last flush ID to this DS: Some(JobId(3112)) = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
93112025-03-17T22:18:16.940ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
93122025-03-17T22:18:16.940ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(3112)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
93132025-03-17T22:18:16.940ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(3112)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
9314 ----------------------------------------------------------------
9315 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:10
9316 States: Active Active Active
9317 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9318 3404 Acked Read 2 Done Done Done true
9319 3405 Acked Read 10 Done Done Done true
9320 3406 Acked Write 8 Done Done Done true
9321 3407 Acked Read 6 Done Done Done true
9322 3408 Acked Read 9 Done Done Done true
9323 3409 Acked Read 10 Done Done Done true
9324 3410 Acked Read 1 Done Done Done true
9325 3411 Acked Read 1 Done Done Done true
9326 3412 Acked Write 6 Done Done Done true
9327 3413 Acked Read 5 Done Done Done true
9328 STATES DS:0 DS:1 DS:2 TOTAL
9329 Sent 0 0 0 0
9330 Done 10 10 10 30
9331 Skipped 0 0 0 0
9332 Error 0 0 0 0
9333 Last Flush: Some(JobId(3403)) Some(JobId(3403)) Some(JobId(3403))
9334 Downstairs last five completed: 3403 3402 3401 3400 3399
9335 Upstairs last five completed: 3413 3412 3411 3410 3409
9336 CLIENT: Up:0 ds:10 act:3
9337 ----------------------------------------------------------------
9338 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
9339 Downstairs last five completed: 3414 3413 3412 3411 3410
9340 Upstairs last five completed: 3414 3413 3412 3411 3410
9341 CLIENT: Up:0 ds:0 act:3
9342 Replay: All jobs finished
9343 Main task has work to do, go find it
9344 got dsc Stop(2)
9345 stop 2
9346 Mar 17 22:18:21.210 INFO request completed, latency_us: 374, response_code: 204, [8830] Got stop action so:false kr:false
9347 uri: /stop/cid/2, method: POST, req_id: adda8bfa-220e-4372-848e-d9df6b115d63, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
93482025-03-17T22:18:21.211ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
93492025-03-17T22:18:21.211ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
93502025-03-17T22:18:21.211ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
9351 Mar 17 22:18:21.211 INFO request completed, latency_us: 205, response_code: 200, uri: /state/cid/2, method: GET, req_id: 0c271616-459f-4862-85be-1005f266a5b3, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
93522025-03-17T22:18:21.211ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
9353 [8830] Exited with: ExitStatus(unix_wait_status(9))
9354 [8830][2] reports Exit
9355 Mar 17 22:18:25.213 INFO request completed, latency_us: 216, response_code: 200, uri: /state/cid/2, method: GET, req_id: 13cfba18-9240-4948-ae29-ab9045375483, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
9356 001/300 Write block 576 len 40960 data: 2 2 2 2 2 2 2 2 2 3
9357 002/300 Write block 3676 len 4096 data: 2
9358 003/300 Read block 4765 len 24576
9359 004/300 Write block 4250 len 40960 data: 3 3 3 3 3 3 2 3 4 4
9360 005/300 Write block 3319 len 36864 data: 2 2 2 2 3 3 3 3 4
9361 006/300 Read block 1546 len 20480
9362 007/300 Write block 4039 len 36864 data: 2 2 2 2 2 2 2 3 4
9363 008/300 Read block 103 len 32768
9364 009/300 Flush
9365 010/300 Read block 2723 len 16384
9366 011/300 Write block 1722 len 8192 data: 2 2
9367 012/300 Read block 408 len 20480
9368 013/300 Read block 983 len 40960
9369 014/300 Flush
9370 015/300 Read block 4751 len 24576
9371 016/300 Read block 1304 len 12288
9372 017/300 Read block 1399 len 8192
9373 018/300 Read block 1405 len 40960
9374 019/300 Read block 3205 len 40960
9375 020/300 Read block 480 len 36864
9376 021/300 Read block 1725 len 36864
9377 022/300 Read block 4664 len 16384
9378 023/300 Write block 3473 len 4096 data: 2
9379 024/300 Read block 4168 len 32768
9380 025/300 Write block 4976 len 40960 data: 2 2 2 2 3 2 2 2 2 2
9381 026/300 Read block 3886 len 40960
9382 027/300 Read block 3984 len 28672
9383 028/300 Read block 3267 len 28672
9384 029/300 Read block 2875 len 16384
9385 030/300 Read block 3123 len 16384
9386 031/300 Write block 1801 len 40960 data: 4 3 3 3 3 2 4 3 3 3
9387 032/300 Read block 2763 len 24576
9388 033/300 Write block 2652 len 24576 data: 2 2 2 2 3 3
9389 034/300 Read block 1680 len 12288
9390 035/300 Read block 2923 len 20480
9391 036/300 Flush
9392 037/300 Write block 4774 len 40960 data: 3 3 3 3 3 4 4 4 4 4
9393 038/300 Write block 411 len 28672 data: 2 2 2 2 3 3 3
9394 039/300 Read block 295 len 4096
9395 040/300 Read block 4412 len 36864
9396 041/300 Write block 3871 len 4096 data: 3
9397 042/300 Read block 3244 len 36864
9398 043/300 Write block 3068 len 40960 data: 3 3 3 3 2 3 2 2 2 2
9399 044/300 Read block 2444 len 8192
9400 045/300 Read block 3694 len 36864
9401 046/300 Read block 2966 len 16384
9402 047/300 Read block 4191 len 8192
9403 048/300 Write block 2131 len 32768 data: 4 4 4 3 3 3 3 3
9404 049/300 Write block 80 len 20480 data: 5 3 3 2 2
9405 050/300 Flush
9406 051/300 Read block 4921 len 4096
9407 052/300 Read block 2309 len 36864
9408 053/300 Read block 258 len 12288
9409 054/300 Read block 2494 len 36864
9410 055/300 Write block 3154 len 28672 data: 2 2 2 2 2 3 3
9411 056/300 Write block 1176 len 36864 data: 5 5 4 2 2 2 2 2 2
9412 057/300 Write block 2308 len 16384 data: 2 3 3 3
9413 058/300 Flush
9414 059/300 Read block 2356 len 40960
9415 060/300 Read block 2765 len 36864
9416 061/300 Read block 3231 len 36864
9417 062/300 Read block 1880 len 8192
9418 063/300 Write block 3052 len 40960 data: 3 2 2 2 2 2 2 2 2 2
9419 064/300 Write block 4021 len 4096 data: 3
9420 065/300 Read block 1109 len 12288
9421 066/300 Read block 2174 len 8192
9422 067/300 Write block 3913 len 8192 data: 2 3
9423 068/300 Read block 3615 len 36864
9424 069/300 Read block 4862 len 4096
9425 070/300 Write block 3402 len 20480 data: 2 3 3 3 3
9426 071/300 Read block 3823 len 20480
9427 072/300 Write block 3253 len 24576 data: 2 2 2 2 2 2
9428 073/300 Read block 1107 len 4096
9429 074/300 Read block 1026 len 12288
9430 075/300 Read block 3024 len 32768
9431 076/300 Write block 3196 len 4096 data: 2
9432 077/300 Read block 3246 len 20480
9433 078/300 Write block 2545 len 24576 data: 4 3 3 3 3 3
9434 079/300 Write block 2107 len 12288 data: 4 3 3
9435 080/300 Write block 3198 len 8192 data: 2 3
9436 081/300 Write block 1034 len 32768 data: 4 4 4 3 3 3 4 5
9437 082/300 Write block 3768 len 24576 data: 2 3 3 3 3 3
9438 083/300 Write block 2562 len 16384 data: 2 2 2 4
9439 084/300 Flush
9440 085/300 Write block 1161 len 40960 data: 3 3 2 2 2 2 2 2 2 2
9441 086/300 Write block 3512 len 12288 data: 3 3 3
9442 087/300 Write block 4865 len 4096 data: 5
9443 088/300 Read block 3757 len 4096
9444 089/300 Read block 4062 len 8192
9445 090/300 Write block 2778 len 32768 data: 2 2 2 2 2 2 2 3
9446 091/300 Read block 63 len 16384
9447 092/300 Read block 4134 len 4096
9448 093/300 Read block 4293 len 36864
9449 094/300 Read block 96 len 28672
9450 095/300 Flush
9451 096/300 Read block 2184 len 24576
9452 097/300 Read block 878 len 8192
9453 098/300 Write block 812 len 8192 data: 3 4
9454 099/300 Read block 3690 len 4096
9455 100/300 Write block 1797 len 4096 data: 4
9456 101/300 Write block 790 len 36864 data: 2 2 2 2 2 2 2 3 3
9457 102/300 Write block 2697 len 36864 data: 3 3 3 4 3 3 5 5 4
9458 103/300 Write block 853 len 12288 data: 3 4 3
9459 104/300 Read block 2948 len 4096
9460 105/300 Write block 834 len 32768 data: 3 3 2 3 4 4 4 4
9461 106/300 Flush
9462 107/300 Read block 2698 len 20480
9463 108/300 Write block 3180 len 16384 data: 3 2 2 2
9464 109/300 Write block 2689 len 40960 data: 3 3 3 3 3 3 3 3 4 4
9465 110/300 Read block 791 len 20480
9466 111/300 Write block 3112 len 4096 data: 2
9467 112/300 Read block 2997 len 12288
9468 113/300 Flush
9469 114/300 Read block 4561 len 28672
9470 115/300 Flush
9471 116/300 Read block 3844 len 36864
9472 117/300 Write block 1902 len 8192 data: 2 2
9473 118/300 Read block 1572 len 40960
9474 119/300 Read block 2332 len 8192
9475 120/300 Write block 2476 len 32768 data: 5 4 5 5 3 3 3 3
9476 121/300 Write block 1549 len 16384 data: 2 3 3 3
9477 122/300 Write block 4049 len 24576 data: 2 2 2 2 2 2
9478 123/300 Write block 3440 len 8192 data: 2 2
9479 124/300 Read block 3346 len 8192
9480 125/300 Write block 1664 len 4096 data: 3
9481 126/300 Read block 377 len 12288
9482 127/300 Write block 158 len 4096 data: 4
9483 128/300 Read block 4733 len 20480
9484 129/300 Read block 4167 len 4096
9485 130/300 Read block 279 len 4096
9486 131/300 Read block 4489 len 36864
9487 132/300 Read block 3100 len 40960
9488 133/300 Read block 2411 len 32768
9489 134/300 Write block 4958 len 28672 data: 2 2 2 2 2 3 2
9490 135/300 Read block 1214 len 36864
9491 136/300 Write block 1209 len 8192 data: 2 3
9492 137/300 Write block 1291 len 8192 data: 2 2
9493 138/300 Read block 71 len 32768
9494 139/300 Read block 4195 len 28672
9495 140/300 Write block 233 len 36864 data: 5 4 4 4 5 4 4 4 4
9496 141/300 Flush
9497 142/300 Flush
9498 143/300 Read block 2757 len 12288
9499 144/300 Read block 838 len 32768
9500 145/300 Read block 766 len 12288
9501 146/300 Read block 1935 len 4096
9502 147/300 Write block 1608 len 24576 data: 2 2 2 2 2 3
9503 148/300 Read block 3662 len 8192
9504 149/300 Read block 1892 len 36864
9505 150/300 Read block 4277 len 20480
9506 151/300 Write block 250 len 8192 data: 2 2
9507 152/300 Read block 1485 len 4096
9508 153/300 Write block 3922 len 8192 data: 2 2
9509 154/300 Read block 937 len 4096
9510 155/300 Write block 99 len 20480 data: 2 2 2 2 2
9511 156/300 Read block 1441 len 40960
9512 157/300 Write block 1831 len 32768 data: 3 3 3 3 3 2 2 2
9513 158/300 Read block 576 len 20480
9514 159/300 Write block 944 len 20480 data: 3 3 2 2 2
9515 160/300 Write block 2674 len 4096 data: 4
9516 161/300 Write block 4347 len 12288 data: 2 2 2
9517 162/300 Write block 4724 len 24576 data: 5 3 3 3 3 4
9518 163/300 Write block 681 len 12288 data: 4 4 5
9519 164/300 Read block 949 len 16384
9520 165/300 Read block 1649 len 4096
9521 166/300 Read block 4948 len 24576
9522 167/300 Read block 2035 len 20480
9523 168/300 Write block 1580 len 28672 data: 4 4 4 4 4 4 4
9524 169/300 Read block 745 len 12288
9525 170/300 Read block 599 len 8192
9526 171/300 Write block 3564 len 28672 data: 3 3 3 2 2 2 2
9527 172/300 Write block 3247 len 24576 data: 2 2 2 3 3 2
9528 173/300 Write block 2279 len 20480 data: 5 5 5 5 5
9529 174/300 Read block 28 len 28672
9530 175/300 Write block 4721 len 24576 data: 4 4 5 6 4 4
9531 176/300 Flush
9532 177/300 Read block 1597 len 32768
9533 178/300 Read block 3628 len 24576
9534 179/300 Read block 4165 len 32768
9535 180/300 Write block 1096 len 36864 data: 3 3 3 3 3 3 4 3 3
9536 181/300 Write block 4703 len 40960 data: 2 2 2 3 3 3 3 3 3 2
9537 182/300 Read block 155 len 8192
9538 183/300 Flush
9539 184/300 Read block 1137 len 20480
9540 185/300 Read block 2063 len 8192
9541 186/300 Write block 1011 len 8192 data: 3 3
9542 187/300 Write block 2756 len 20480 data: 2 3 3 3 3
9543 188/300 Read block 3479 len 28672
9544 189/300 Read block 2822 len 36864
9545 190/300 Write block 3799 len 40960 data: 5 5 5 5 4 3 3 4 3 3
9546 191/300 Write block 1863 len 24576 data: 4 4 3 3 3 3
9547 192/300 Write block 3651 len 12288 data: 3 3 3
9548 193/300 Write block 553 len 4096 data: 2
9549 194/300 Write block 1282 len 32768 data: 3 3 3 2 2 2 2 2
9550 195/300 Write block 905 len 12288 data: 2 2 2
9551 196/300 Read block 2366 len 36864
9552 197/300 Read block 2806 len 32768
9553 198/300 Write block 4521 len 20480 data: 2 2 2 2 2
9554 199/300 Read block 3189 len 28672
9555 200/300 Read block 1688 len 4096
9556 201/300 Read block 243 len 32768
9557 202/300 Read block 2001 len 16384
9558 203/300 Write block 2996 len 32768 data: 4 3 3 3 2 3 3 3
9559 204/300 Write block 2720 len 8192 data: 2 2
9560 205/300 Read block 1912 len 32768
9561 206/300 Write block 4949 len 12288 data: 2 3 3
9562 207/300 Write block 2905 len 8192 data: 3 3
9563 208/300 Read block 269 len 8192
9564 209/300 Write block 1295 len 36864 data: 2 2 2 2 2 2 2 2 2
9565 210/300 Read block 3781 len 28672
9566 211/300 Write block 4105 len 12288 data: 3 3 4
9567 212/300 Read block 1141 len 20480
9568 213/300 Read block 3387 len 36864
9569 214/300 Write block 2915 len 24576 data: 2 2 2 2 2 2
9570 215/300 Flush
9571 216/300 Write block 3013 len 36864 data: 4 4 5 5 5 5 4 4 4
9572 217/300 Write block 2132 len 32768 data: 5 5 4 4 4 4 4 3
9573 218/300 Write block 367 len 24576 data: 2 2 2 2 2 3
9574 219/300 Write block 3450 len 8192 data: 3 3
9575 220/300 Flush
9576 221/300 Write block 3130 len 24576 data: 4 4 4 4 4 4
9577 222/300 Read block 4924 len 16384
9578 223/300 Write block 28 len 32768 data: 4 2 2 3 3 3 3 3
9579 224/300 Write block 1221 len 4096 data: 4
9580 225/300 Flush
9581 226/300 Write block 2884 len 16384 data: 4 5 6 6
9582 227/300 Read block 2156 len 4096
9583 228/300 Write block 1518 len 4096 data: 2
9584 229/300 Write block 279 len 36864 data: 3 2 2 3 3 3 3 3 3
9585 230/300 Write block 3010 len 40960 data: 3 3 3 5 5 6 6 6 6 5
9586 231/300 Read block 2578 len 40960
9587 232/300 Flush
9588 233/300 Write block 16 len 28672 data: 3 3 3 3 3 3 3
9589 234/300 Read block 1738 len 4096
9590 235/300 Read block 4142 len 28672
9591 236/300 Read block 4350 len 36864
9592 237/300 Read block 3968 len 8192
9593 238/300 Read block 1567 len 40960
9594 239/300 Write block 424 len 12288 data: 2 2 2
9595 240/300 Write block 1869 len 36864 data: 4 4 5 3 3 3 2 2 2
9596 241/300 Write block 1474 len 20480 data: 3 3 3 4 4
9597 242/300 Read block 461 len 24576
9598 243/300 Flush
9599 244/300 Read block 2695 len 28672
9600 245/300 Read block 1659 len 36864
9601 246/300 Write block 1860 len 4096 data: 5
9602 247/300 Read block 2699 len 20480
9603 248/300 Write block 993 len 12288 data: 4 4 3
9604 249/300 Read block 21 len 20480
9605 250/300 Read block 3692 len 12288
9606 251/300 Read block 1971 len 28672
9607 252/300 Read block 3100 len 40960
9608 253/300 Read block 3472 len 20480
9609 254/300 Read block 1632 len 12288
9610 255/300 Write block 234 len 28672 data: 5 5 5 6 5 5 5
9611 256/300 Read block 4240 len 40960
9612 257/300 Read block 1014 len 8192
9613 258/300 Write block 4507 len 24576 data: 2 2 2 2 2 2
9614 259/300 Write block 4421 len 20480 data: 4 3 3 3 3
9615 260/300 Read block 400 len 4096
9616 261/300 Write block 1033 len 8192 data: 4 5
9617 262/300 Write block 114 len 40960 data: 2 3 3 3 3 3 3 3 3 3
9618 263/300 Write block 4199 len 24576 data: 3 2 2 2 2 2
9619 264/300 Write block 3611 len 36864 data: 2 3 4 3 3 3 3 3 3
9620 265/300 Read block 1822 len 8192
9621 266/300 Read block 2353 len 36864
9622 267/300 Read block 3578 len 12288
9623 268/300 Read block 1841 len 20480
9624 269/300 Read block 518 len 12288
9625 270/300 Read block 4712 len 8192
9626 271/300 Write block 4761 len 4096 data: 4
9627 272/300 Write block 2112 len 40960 data: 5 4 4 4 4 5 6 6 5 5
9628 273/300 Read block 4805 len 12288
9629 274/300 Flush
9630 275/300 Write block 1357 len 40960 data: 4 4 4 4 4 4 4 3 3 3
9631 276/300 Write block 4060 len 12288 data: 5 5 4
9632 277/300 Write block 4628 len 12288 data: 2 2 3
9633 278/300 Write block 2802 len 16384 data: 3 4 4 4
9634 279/300 Read block 1894 len 24576
9635 280/300 Flush
9636 281/300 Read block 2514 len 20480
9637 282/300 Write block 1897 len 24576 data: 2 2 2 2 2 3
9638 283/300 Read block 2714 len 36864
9639 284/300 Read block 221 len 40960
9640 285/300 Write block 2074 len 20480 data: 6 4 3 3 3
9641 286/300 Read block 3406 len 4096
9642 287/300 Write block 2950 len 4096 data: 2
9643 288/300 Read block 926 len 20480
9644 289/300 Write block 2409 len 20480 data: 2 2 2 2 2
9645 290/300 Read block 4137 len 32768
9646 291/300 Write block 91 len 36864 data: 4 3 3 3 3 2 2 2 3
9647 292/300 Write block 2637 len 4096 data: 2
9648 293/300 Write block 1449 len 4096 data: 2
9649 294/300 Read block 1086 len 40960
9650 295/300 Write block 3354 len 24576 data: 5 5 5 3 3 3
9651 296/300 Write block 3597 len 4096 data: 4
9652 297/300 Write block 2606 len 24576 data: 5 6 5 5 5 5
9653 298/300 Read block 376 len 8192
9654 299/300 Read block 2154 len 32768
9655 300/300 Read block 4869 len 36864
9656 301/300 Read block 2939 len 36864
9657 Main task has work to do, go find it
9658 got dsc Start(2)
9659 start 2
9660 Mar 17 22:18:25.456 INFO request completed, latency_us: 296, response_code: 204, uri: /start/cid/2, method: POST, req_id: d6fa2728-1a0b-4db6-8f59-4e97930da1ae, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
9661 Replay: started 2, returned:Ok(())
9662 ----------------------------------------------------------------
9663 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9664 States: Active Active Offline
9665 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9666 3415 Acked Write 10 Done Done Sent false
9667 3416 Acked Write 1 Done Done Sent false
9668 3417 Acked Read 6 Done Done Sent false
9669 3418 Acked Write 10 Done Done Sent false
9670 3419 Acked Write 9 Done Done Sent false
9671 3420 Acked Read 5 Done Done Sent false
9672 3421 Acked Write 9 Done Done Sent false
9673 3422 Acked Read 8 Done Done Sent false
9674 3423 Acked Flush 0 Done Done Sent false
9675 3424 Acked Read 4 Done Done Sent false
9676 3425 Acked Write 2 Done Done Sent false
9677 3426 Acked Read 5 Done Done Sent false
9678 3427 Acked Read 10 Done Done Sent false
9679 3428 Acked Flush 0 Done Done Sent false
9680 3429 Acked Read 6 Done Done Sent false
9681 3430 Acked Read 3 Done Done Sent false
9682 3431 Acked Read 2 Done Done Sent false
9683 3432 Acked Read 10 Done Done Sent false
9684 3433 Acked Read 10 Done Done Sent false
9685 3434 Acked Read 9 Done Done Sent false
9686 3435 Acked Read 9 Done Done Sent false
9687 3436 Acked Read 4 Done Done Sent false
9688 3437 Acked Write 1 Done Done Sent false
9689 3438 Acked Read 8 Done Done Sent false
9690 3439 Acked Write 10 Done Done Sent false
9691 3440 Acked Read 10 Done Done Sent false
9692 3441 Acked Read 7 Done Done Sent false
9693 3442 Acked Read 7 Done Done Sent false
9694 3443 Acked Read 4 Done Done Sent false
9695 3444 Acked Read 4 Done Done Sent false
9696 3445 Acked Write 10 Done Done Sent false
9697 3446 Acked Read 6 Done Done Sent false
9698 3447 Acked Write 6 Done Done Sent false
9699 3448 Acked Read 3 Done Done Sent false
9700 3449 Acked Read 5 Done Done Sent false
9701 3450 Acked Flush 0 Done Done Sent false
9702 3451 Acked Write 10 Done Done Sent false
9703 3452 Acked Write 7 Done Done Sent false
9704 3453 Acked Read 1 Done Done Sent false
9705 3454 Acked Read 9 Done Done Sent false
9706 3455 Acked Write 1 Done Done Sent false
9707 3456 Acked Read 9 Done Done Sent false
9708 3457 Acked Write 10 Done Done Sent false
9709 3458 Acked Read 2 Done Done Sent false
9710 3459 Acked Read 9 Done Done Sent false
9711 3460 Acked Read 4 Done Done Sent false
9712 3461 Acked Read 2 Done Done Sent false
9713 3462 Acked Write 8 Done Done Sent false
9714 3463 Acked Write 5 Done Done Sent false
9715 3464 Acked Flush 0 Done Done Sent false
9716 3465 Acked Read 1 Done Done Sent false
9717 3466 Acked Read 9 Done Done Sent false
9718 3467 Acked Read 3 Done Done Sent false
9719 3468 Acked Read 9 Done Done Sent false
9720 3469 Acked Write 7 Done Done Sent false
9721 3470 Acked Write 9 Done Done Sent false
9722 3471 Acked Write 4 Done Done Sent false
9723 3472 Acked Flush 0 Done Done Sent false
9724 3473 Acked Read 10 Done Done Sent false
9725 3474 Acked Read 9 Done Done Sent false
9726 3475 Acked Read 9 Done Done Sent false
9727 3476 Acked Read 2 Done Done Sent false
9728 3477 Acked Write 10 Done Done Sent false
9729 3478 Acked Write 1 Done Done Sent false
9730 3479 Acked Read 3 Done Done Sent false
9731 3480 Acked Read 2 Done Done Sent false
9732 3481 Acked Write 2 Done Done Sent false
9733 3482 Acked Read 9 Done Done Sent false
9734 3483 Acked Read 1 Done Done Sent false
9735 3484 Acked Write 5 Done Done Sent false
9736 3485 Acked Read 5 Done Done Sent false
9737 3486 Acked Write 6 Done Done Sent false
9738 3487 Acked Read 1 Done Done Sent false
9739 3488 Acked Read 3 Done Done Sent false
9740 3489 Acked Read 8 Done Done Sent false
9741 3490 Acked Write 1 Done Done Sent false
9742 3491 Acked Read 5 Done Done Sent false
9743 3492 Acked Write 6 Done Done Sent false
9744 3493 Acked Write 3 Done Done Sent false
9745 3494 Acked Write 2 Done Done Sent false
9746 3495 Acked Write 8 Done Done Sent false
9747 3496 Acked Write 6 Done Done Sent false
9748 3497 Acked Write 4 Done Done Sent false
9749 3498 Acked Flush 0 Done Done Sent false
9750 3499 Acked Write 10 Done Done Sent false
9751 3500 Acked Write 3 Done Done Sent false
9752 3501 Acked Write 1 Done Done Sent false
9753 3502 Acked Read 1 Done Done Sent false
9754 3503 Acked Read 2 Done Done Sent false
9755 3504 Acked Write 8 Done Done Sent false
9756 3505 Acked Read 4 Done Done Sent false
9757 3506 Acked Read 1 Done Done Sent false
9758 3507 Acked Read 9 Done Done Sent false
9759 3508 Acked Read 7 Done Done Sent false
9760 3509 Acked Flush 0 Done Done Sent false
9761 3510 Acked Read 6 Done Done Sent false
9762 3511 Acked Read 2 Done Done Sent false
9763 3512 Acked Write 2 Done Done Sent false
9764 3513 Acked Read 1 Done Done Sent false
9765 3514 Acked Write 1 Done Done Sent false
9766 3515 Acked Write 9 Done Done Sent false
9767 3516 Acked Write 9 Done Done Sent false
9768 3517 Acked Write 3 Done Done Sent false
9769 3518 Acked Read 1 Done Done Sent false
9770 3519 Acked Write 8 Done Done Sent false
9771 3520 Acked Flush 0 Done Done Sent false
9772 3521 Acked Read 5 Done Done Sent false
9773 3522 Acked Write 4 Done Done Sent false
9774 3523 Acked Write 10 Done Done Sent false
9775 3524 Acked Read 5 Done Done Sent false
9776 3525 Acked Write 1 Done Done Sent false
9777 3526 Acked Read 3 Done Done Sent false
9778 3527 Acked Flush 0 Done Done Sent false
9779 3528 Acked Read 7 Done Done Sent false
9780 3529 Acked Flush 0 Done Done Sent false
9781 3530 Acked Read 9 Done Done Sent false
9782 3531 Acked Write 2 Done Done Sent false
9783 3532 Acked Read 10 Done Done Sent false
9784 3533 Acked Read 2 Done Done Sent false
9785 3534 Acked Write 8 Done Done Sent false
9786 3535 Acked Write 4 Done Done Sent false
9787 3536 Acked Write 6 Done Done Sent false
9788 3537 Acked Write 2 Done Done Sent false
9789 3538 Acked Read 2 Done Done Sent false
9790 3539 Acked Write 1 Done Done Sent false
9791 3540 Acked Read 3 Done Done Sent false
9792 3541 Acked Write 1 Done Done Sent false
9793 3542 Acked Read 5 Done Done Sent false
9794 3543 Acked Read 1 Done Done Sent false
9795 3544 Acked Read 1 Done Done Sent false
9796 3545 Acked Read 9 Done Done Sent false
9797 3546 Acked Read 10 Done Done Sent false
9798 3547 Acked Read 8 Done Done Sent false
9799 3548 Acked Write 7 Done Done Sent false
9800 3549 Acked Read 9 Done Done Sent false
9801 3550 Acked Write 2 Done Done Sent false
9802 3551 Acked Write 2 Done Done Sent false
9803 3552 Acked Read 8 Done Done Sent false
9804 3553 Acked Read 7 Done Done Sent false
9805 3554 Acked Write 9 Done Done Sent false
9806 3555 Acked Flush 0 Done Done Sent false
9807 3556 Acked Flush 0 Done Done Sent false
9808 3557 Acked Read 3 Done Done Sent false
9809 3558 Acked Read 8 Done Done Sent false
9810 3559 Acked Read 3 Done Done Sent false
9811 3560 Acked Read 1 Done Done Sent false
9812 3561 Acked Write 6 Done Done Sent false
9813 3562 Acked Read 2 Done Done Sent false
9814 3563 Acked Read 9 Done Done Sent false
9815 3564 Acked Read 5 Done Done Sent false
9816 3565 Acked Write 2 Done Done Sent false
9817 3566 Acked Read 1 Done Done Sent false
9818 3567 Acked Write 2 Done Done Sent false
9819 3568 Acked Read 1 Done Done Sent false
9820 3569 Acked Write 5 Done Done Sent false
9821 3570 Acked Read 10 Done Done Sent false
9822 3571 Acked Write 8 Done Done Sent false
9823 3572 Acked Read 5 Done Done Sent false
9824 3573 Acked Write 5 Done Done Sent false
9825 3574 Acked Write 1 Done Done Sent false
9826 3575 Acked Write 3 Done Done Sent false
9827 3576 Acked Write 6 Done Done Sent false
9828 3577 Acked Write 3 Done Done Sent false
9829 3578 Acked Read 4 Done Done Sent false
9830 3579 Acked Read 1 Done Done Sent false
9831 3580 Acked Read 6 Done Done Sent false
9832 3581 Acked Read 5 Done Done Sent false
9833 3582 Acked Write 7 Done Done Sent false
9834 3583 Acked Read 3 Done Done Sent false
9835 3584 Acked Read 2 Done Done Sent false
9836 3585 Acked Write 7 Done Done Sent false
9837 3586 Acked Write 6 Done Done Sent false
9838 3587 Acked Write 5 Done Done Sent false
9839 3588 Acked Read 7 Done Done Sent false
9840 3589 Acked Write 6 Done Done Sent false
9841 3590 Acked Flush 0 Done Done Sent false
9842 3591 Acked Read 8 Done Done Sent false
9843 3592 Acked Read 6 Done Done Sent false
9844 3593 Acked Read 8 Done Done Sent false
9845 3594 Acked Write 9 Done Done Sent false
9846 3595 Acked Write 10 Done Done Sent false
9847 3596 Acked Read 2 Done Done Sent false
9848 3597 Acked Flush 0 Done Done Sent false
9849 3598 Acked Read 5 Done Done Sent false
9850 3599 Acked Read 2 Done Done Sent false
9851 3600 Acked Write 2 Done Done Sent false
9852 3601 Acked Write 5 Done Done Sent false
9853 3602 Acked Read 7 Done Done Sent false
9854 3603 Acked Read 9 Done Done Sent false
9855 3604 Acked Write 10 Done Done Sent false
9856 3605 Acked Write 6 Done Done Sent false
9857 3606 Acked Write 3 Done Done Sent false
9858 3607 Acked Write 1 Done Done Sent false
9859 3608 Acked Write 8 Done Done Sent false
9860 3609 Acked Write 3 Done Done Sent false
9861 3610 Acked Read 9 Done Done Sent false
9862 3611 Acked Read 8 Done Done Sent false
9863 3612 Acked Write 5 Done Done Sent false
9864 3613 Acked Read 7 Done Done Sent false
9865 3614 Acked Read 1 Done Done Sent false
9866 3615 Acked Read 8 Done Done Sent false
9867 3616 Acked Read 4 Done Done Sent false
9868 3617 Acked Write 8 Done Done Sent false
9869 3618 Acked Write 2 Done Done Sent false
9870 3619 Acked Read 8 Done Done Sent false
9871 3620 Acked Write 3 Done Done Sent false
9872 3621 Acked Write 2 Done Done Sent false
9873 3622 Acked Read 2 Done Done Sent false
9874 3623 Acked Write 9 Done Done Sent false
9875 3624 Acked Read 7 Done Done Sent false
9876 3625 Acked Write 3 Done Done Sent false
9877 3626 Acked Read 5 Done Done Sent false
9878 3627 Acked Read 9 Done Done Sent false
9879 3628 Acked Write 6 Done Done Sent false
9880 3629 Acked Flush 0 Done Done Sent false
9881 3630 Acked Write 9 Done Done Sent false
9882 3631 Acked Write 8 Done Done Sent false
9883 3632 Acked Write 6 Done Done Sent false
9884 3633 Acked Write 2 Done Done Sent false
9885 3634 Acked Flush 0 Done Done Sent false
9886 3635 Acked Write 6 Done Done Sent false
9887 3636 Acked Read 4 Done Done Sent false
9888 3637 Acked Write 8 Done Done Sent false
9889 3638 Acked Write 1 Done Done Sent false
9890 3639 Acked Flush 0 Done Done Sent false
9891 3640 Acked Write 4 Done Done Sent false
9892 3641 Acked Read 1 Done Done Sent false
9893 3642 Acked Write 1 Done Done Sent false
9894 3643 Acked Write 9 Done Done Sent false
9895 3644 Acked Write 10 Done Done Sent false
9896 3645 Acked Read 10 Done Done Sent false
9897 3646 Acked Flush 0 Done Done Sent false
9898 3647 Acked Write 7 Done Done Sent false
9899 3648 Acked Read 1 Done Done Sent false
9900 3649 Acked Read 7 Done Done Sent false
9901 3650 Acked Read 9 Done Done Sent false
9902 3651 Acked Read 2 Done Done Sent false
9903 3652 Acked Read 10 Done Done Sent false
9904 3653 Acked Write 3 Done Done Sent false
9905 3654 Acked Write 9 Done Done Sent false
9906 3655 Acked Write 5 Done Done Sent false
9907 3656 Acked Read 6 Done Done Sent false
9908 3657 Acked Flush 0 Done Done Sent false
9909 3658 Acked Read 7 Done Done Sent false
9910 3659 Acked Read 9 Done Done Sent false
9911 3660 Acked Write 1 Done Done Sent false
9912 3661 Acked Read 5 Done Done Sent false
9913 3662 Acked Write 3 Done Done Sent false
9914 3663 Acked Read 5 Done Done Sent false
9915 3664 Acked Read 3 Done Done Sent false
9916 3665 Acked Read 7 Done Done Sent false
9917 3666 Acked Read 10 Done Done Sent false
9918 3667 Acked Read 5 Done Done Sent false
9919 3668 Acked Read 3 Done Done Sent false
9920 3669 Acked Write 7 Done Done Sent false
9921 3670 Acked Read 10 Done Done Sent false
9922 3671 Acked Read 2 Done Done Sent false
9923 3672 Acked Write 6 Done Done Sent false
9924 3673 Acked Write 5 Done Done Sent false
9925 3674 Acked Read 1 Done Done Sent false
9926 3675 Acked Write 2 Done Done Sent false
9927 3676 Acked Write 10 Done Done Sent false
9928 3677 Acked Write 6 Done Done Sent false
9929 3678 Acked Write 9 Done Done Sent false
9930 3679 Acked Read 2 Done Done Sent false
9931 3680 Acked Read 9 Done Done Sent false
9932 3681 Acked Read 3 Done Done Sent false
9933 3682 Acked Read 5 Done Done Sent false
9934 3683 Acked Read 3 Done Done Sent false
9935 3684 Acked Read 2 Done Done Sent false
9936 3685 Acked Write 1 Done Done Sent false
9937 3686 Acked Write 10 Done Done Sent false
9938 3687 Acked Read 3 Done Done Sent false
9939 3688 Acked Flush 0 Done Done Sent false
9940 3689 Acked Write 10 Done Done Sent false
9941 3690 Acked Write 3 Done Done Sent false
9942 3691 Acked Write 3 Done Done Sent false
9943 3692 Acked Write 4 Done Done Sent false
9944 3693 Acked Read 6 Done Done Sent false
9945 3694 Acked Flush 0 Done Done Sent false
9946 3695 Acked Read 5 Done Done Sent false
9947 3696 Acked Write 6 Done Done Sent false
9948 3697 Acked Read 9 Done Done Sent false
9949 3698 Acked Read 10 Done Done Sent false
9950 3699 Acked Write 5 Done Done Sent false
9951 3700 Acked Read 1 Done Done Sent false
9952 3701 Acked Write 1 Done Done Sent false
9953 3702 Acked Read 5 Done Done Sent false
9954 3703 Acked Write 5 Done Done Sent false
9955 3704 Acked Read 8 Done Done Sent false
9956 3705 Acked Write 9 Done Done Sent false
9957 3706 Acked Write 1 Done Done Sent false
9958 3707 Acked Write 1 Done Done Sent false
9959 3708 Acked Read 10 Done Done Sent false
9960 3709 Acked Write 6 Done Done Sent false
9961 3710 Acked Write 1 Done Done Sent false
9962 3711 Acked Write 6 Done Done Sent false
9963 3712 Acked Read 2 Done Done Sent false
9964 3713 Acked Read 8 Done Done Sent false
9965 3714 Acked Read 9 Done Done Sent false
9966 3715 Acked Read 9 Done Done Sent false
9967 STATES DS:0 DS:1 DS:2 TOTAL
9968 Sent 0 0 301 301
9969 Done 301 301 0 602
9970 Skipped 0 0 0 0
9971 Error 0 0 0 0
9972 Last Flush: Some(JobId(3694)) Some(JobId(3694)) Some(JobId(3414))
9973 Downstairs last five completed: 3414 3413 3412 3411 3410
9974 Upstairs last five completed: 3715 3714 3713 3712 3711
9975 CLIENT: Up:0 ds:301 act:2
99762025-03-17T22:18:25.958ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
99772025-03-17T22:18:26.799ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
99782025-03-17T22:18:27.800ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
99792025-03-17T22:18:28.802ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
9980 ----------------------------------------------------------------
9981 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9982 States: Active Active Offline
9983 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
9984 3415 Acked Write 10 Done Done Sent false
9985 3416 Acked Write 1 Done Done Sent false
9986 3417 Acked Read 6 Done Done Sent false
9987 3418 Acked Write 10 Done Done Sent false
9988 3419 Acked Write 9 Done Done Sent false
9989 3420 Acked Read 5 Done Done Sent false
9990 3421 Acked Write 9 Done Done Sent false
9991 3422 Acked Read 8 Done Done Sent false
9992 3423 Acked Flush 0 Done Done Sent false
9993 3424 Acked Read 4 Done Done Sent false
9994 3425 Acked Write 2 Done Done Sent false
9995 3426 Acked Read 5 Done Done Sent false
9996 3427 Acked Read 10 Done Done Sent false
9997 3428 Acked Flush 0 Done Done Sent false
9998 3429 Acked Read 6 Done Done Sent false
9999 3430 Acked Read 3 Done Done Sent false
10000 3431 Acked Read 2 Done Done Sent false
10001 3432 Acked Read 10 Done Done Sent false
10002 3433 Acked Read 10 Done Done Sent false
10003 3434 Acked Read 9 Done Done Sent false
10004 3435 Acked Read 9 Done Done Sent false
10005 3436 Acked Read 4 Done Done Sent false
10006 3437 Acked Write 1 Done Done Sent false
10007 3438 Acked Read 8 Done Done Sent false
10008 3439 Acked Write 10 Done Done Sent false
10009 3440 Acked Read 10 Done Done Sent false
10010 3441 Acked Read 7 Done Done Sent false
10011 3442 Acked Read 7 Done Done Sent false
10012 3443 Acked Read 4 Done Done Sent false
10013 3444 Acked Read 4 Done Done Sent false
10014 3445 Acked Write 10 Done Done Sent false
10015 3446 Acked Read 6 Done Done Sent false
10016 3447 Acked Write 6 Done Done Sent false
10017 3448 Acked Read 3 Done Done Sent false
10018 3449 Acked Read 5 Done Done Sent false
10019 3450 Acked Flush 0 Done Done Sent false
10020 3451 Acked Write 10 Done Done Sent false
10021 3452 Acked Write 7 Done Done Sent false
10022 3453 Acked Read 1 Done Done Sent false
10023 3454 Acked Read 9 Done Done Sent false
10024 3455 Acked Write 1 Done Done Sent false
10025 3456 Acked Read 9 Done Done Sent false
10026 3457 Acked Write 10 Done Done Sent false
10027 3458 Acked Read 2 Done Done Sent false
10028 3459 Acked Read 9 Done Done Sent false
10029 3460 Acked Read 4 Done Done Sent false
10030 3461 Acked Read 2 Done Done Sent false
10031 3462 Acked Write 8 Done Done Sent false
10032 3463 Acked Write 5 Done Done Sent false
10033 3464 Acked Flush 0 Done Done Sent false
10034 3465 Acked Read 1 Done Done Sent false
10035 3466 Acked Read 9 Done Done Sent false
10036 3467 Acked Read 3 Done Done Sent false
10037 3468 Acked Read 9 Done Done Sent false
10038 3469 Acked Write 7 Done Done Sent false
10039 3470 Acked Write 9 Done Done Sent false
10040 3471 Acked Write 4 Done Done Sent false
10041 3472 Acked Flush 0 Done Done Sent false
10042 3473 Acked Read 10 Done Done Sent false
10043 3474 Acked Read 9 Done Done Sent false
10044 3475 Acked Read 9 Done Done Sent false
10045 3476 Acked Read 2 Done Done Sent false
10046 3477 Acked Write 10 Done Done Sent false
10047 3478 Acked Write 1 Done Done Sent false
10048 3479 Acked Read 3 Done Done Sent false
10049 3480 Acked Read 2 Done Done Sent false
10050 3481 Acked Write 2 Done Done Sent false
10051 3482 Acked Read 9 Done Done Sent false
10052 3483 Acked Read 1 Done Done Sent false
10053 3484 Acked Write 5 Done Done Sent false
10054 3485 Acked Read 5 Done Done Sent false
10055 3486 Acked Write 6 Done Done Sent false
10056 3487 Acked Read 1 Done Done Sent false
10057 3488 Acked Read 3 Done Done Sent false
10058 3489 Acked Read 8 Done Done Sent false
10059 3490 Acked Write 1 Done Done Sent false
10060 3491 Acked Read 5 Done Done Sent false
10061 3492 Acked Write 6 Done Done Sent false
10062 3493 Acked Write 3 Done Done Sent false
10063 3494 Acked Write 2 Done Done Sent false
10064 3495 Acked Write 8 Done Done Sent false
10065 3496 Acked Write 6 Done Done Sent false
10066 3497 Acked Write 4 Done Done Sent false
10067 3498 Acked Flush 0 Done Done Sent false
10068 3499 Acked Write 10 Done Done Sent false
10069 3500 Acked Write 3 Done Done Sent false
10070 3501 Acked Write 1 Done Done Sent false
10071 3502 Acked Read 1 Done Done Sent false
10072 3503 Acked Read 2 Done Done Sent false
10073 3504 Acked Write 8 Done Done Sent false
10074 3505 Acked Read 4 Done Done Sent false
10075 3506 Acked Read 1 Done Done Sent false
10076 3507 Acked Read 9 Done Done Sent false
10077 3508 Acked Read 7 Done Done Sent false
10078 3509 Acked Flush 0 Done Done Sent false
10079 3510 Acked Read 6 Done Done Sent false
10080 3511 Acked Read 2 Done Done Sent false
10081 3512 Acked Write 2 Done Done Sent false
10082 3513 Acked Read 1 Done Done Sent false
10083 3514 Acked Write 1 Done Done Sent false
10084 3515 Acked Write 9 Done Done Sent false
10085 3516 Acked Write 9 Done Done Sent false
10086 3517 Acked Write 3 Done Done Sent false
10087 3518 Acked Read 1 Done Done Sent false
10088 3519 Acked Write 8 Done Done Sent false
10089 3520 Acked Flush 0 Done Done Sent false
10090 3521 Acked Read 5 Done Done Sent false
10091 3522 Acked Write 4 Done Done Sent false
10092 3523 Acked Write 10 Done Done Sent false
10093 3524 Acked Read 5 Done Done Sent false
10094 3525 Acked Write 1 Done Done Sent false
10095 3526 Acked Read 3 Done Done Sent false
10096 3527 Acked Flush 0 Done Done Sent false
10097 3528 Acked Read 7 Done Done Sent false
10098 3529 Acked Flush 0 Done Done Sent false
10099 3530 Acked Read 9 Done Done Sent false
10100 3531 Acked Write 2 Done Done Sent false
10101 3532 Acked Read 10 Done Done Sent false
10102 3533 Acked Read 2 Done Done Sent false
10103 3534 Acked Write 8 Done Done Sent false
10104 3535 Acked Write 4 Done Done Sent false
10105 3536 Acked Write 6 Done Done Sent false
10106 3537 Acked Write 2 Done Done Sent false
10107 3538 Acked Read 2 Done Done Sent false
10108 3539 Acked Write 1 Done Done Sent false
10109 3540 Acked Read 3 Done Done Sent false
10110 3541 Acked Write 1 Done Done Sent false
10111 3542 Acked Read 5 Done Done Sent false
10112 3543 Acked Read 1 Done Done Sent false
10113 3544 Acked Read 1 Done Done Sent false
10114 3545 Acked Read 9 Done Done Sent false
10115 3546 Acked Read 10 Done Done Sent false
10116 3547 Acked Read 8 Done Done Sent false
10117 3548 Acked Write 7 Done Done Sent false
10118 3549 Acked Read 9 Done Done Sent false
10119 3550 Acked Write 2 Done Done Sent false
10120 3551 Acked Write 2 Done Done Sent false
10121 3552 Acked Read 8 Done Done Sent false
10122 3553 Acked Read 7 Done Done Sent false
10123 3554 Acked Write 9 Done Done Sent false
10124 3555 Acked Flush 0 Done Done Sent false
10125 3556 Acked Flush 0 Done Done Sent false
10126 3557 Acked Read 3 Done Done Sent false
10127 3558 Acked Read 8 Done Done Sent false
10128 3559 Acked Read 3 Done Done Sent false
10129 3560 Acked Read 1 Done Done Sent false
10130 3561 Acked Write 6 Done Done Sent false
10131 3562 Acked Read 2 Done Done Sent false
10132 3563 Acked Read 9 Done Done Sent false
10133 3564 Acked Read 5 Done Done Sent false
10134 3565 Acked Write 2 Done Done Sent false
10135 3566 Acked Read 1 Done Done Sent false
10136 3567 Acked Write 2 Done Done Sent false
10137 3568 Acked Read 1 Done Done Sent false
10138 3569 Acked Write 5 Done Done Sent false
10139 3570 Acked Read 10 Done Done Sent false
10140 3571 Acked Write 8 Done Done Sent false
10141 3572 Acked Read 5 Done Done Sent false
10142 3573 Acked Write 5 Done Done Sent false
10143 3574 Acked Write 1 Done Done Sent false
10144 3575 Acked Write 3 Done Done Sent false
10145 3576 Acked Write 6 Done Done Sent false
10146 3577 Acked Write 3 Done Done Sent false
10147 3578 Acked Read 4 Done Done Sent false
10148 3579 Acked Read 1 Done Done Sent false
10149 3580 Acked Read 6 Done Done Sent false
10150 3581 Acked Read 5 Done Done Sent false
10151 3582 Acked Write 7 Done Done Sent false
10152 3583 Acked Read 3 Done Done Sent false
10153 3584 Acked Read 2 Done Done Sent false
10154 3585 Acked Write 7 Done Done Sent false
10155 3586 Acked Write 6 Done Done Sent false
10156 3587 Acked Write 5 Done Done Sent false
10157 3588 Acked Read 7 Done Done Sent false
10158 3589 Acked Write 6 Done Done Sent false
10159 3590 Acked Flush 0 Done Done Sent false
10160 3591 Acked Read 8 Done Done Sent false
10161 3592 Acked Read 6 Done Done Sent false
10162 3593 Acked Read 8 Done Done Sent false
10163 3594 Acked Write 9 Done Done Sent false
10164 3595 Acked Write 10 Done Done Sent false
10165 3596 Acked Read 2 Done Done Sent false
10166 3597 Acked Flush 0 Done Done Sent false
10167 3598 Acked Read 5 Done Done Sent false
10168 3599 Acked Read 2 Done Done Sent false
10169 3600 Acked Write 2 Done Done Sent false
10170 3601 Acked Write 5 Done Done Sent false
10171 3602 Acked Read 7 Done Done Sent false
10172 3603 Acked Read 9 Done Done Sent false
10173 3604 Acked Write 10 Done Done Sent false
10174 3605 Acked Write 6 Done Done Sent false
10175 3606 Acked Write 3 Done Done Sent false
10176 3607 Acked Write 1 Done Done Sent false
10177 3608 Acked Write 8 Done Done Sent false
10178 3609 Acked Write 3 Done Done Sent false
10179 3610 Acked Read 9 Done Done Sent false
10180 3611 Acked Read 8 Done Done Sent false
10181 3612 Acked Write 5 Done Done Sent false
10182 3613 Acked Read 7 Done Done Sent false
10183 3614 Acked Read 1 Done Done Sent false
10184 3615 Acked Read 8 Done Done Sent false
10185 3616 Acked Read 4 Done Done Sent false
10186 3617 Acked Write 8 Done Done Sent false
10187 3618 Acked Write 2 Done Done Sent false
10188 3619 Acked Read 8 Done Done Sent false
10189 3620 Acked Write 3 Done Done Sent false
10190 3621 Acked Write 2 Done Done Sent false
10191 3622 Acked Read 2 Done Done Sent false
10192 3623 Acked Write 9 Done Done Sent false
10193 3624 Acked Read 7 Done Done Sent false
10194 3625 Acked Write 3 Done Done Sent false
10195 3626 Acked Read 5 Done Done Sent false
10196 3627 Acked Read 9 Done Done Sent false
10197 3628 Acked Write 6 Done Done Sent false
10198 3629 Acked Flush 0 Done Done Sent false
10199 3630 Acked Write 9 Done Done Sent false
10200 3631 Acked Write 8 Done Done Sent false
10201 3632 Acked Write 6 Done Done Sent false
10202 3633 Acked Write 2 Done Done Sent false
10203 3634 Acked Flush 0 Done Done Sent false
10204 3635 Acked Write 6 Done Done Sent false
10205 3636 Acked Read 4 Done Done Sent false
10206 3637 Acked Write 8 Done Done Sent false
10207 3638 Acked Write 1 Done Done Sent false
10208 3639 Acked Flush 0 Done Done Sent false
10209 3640 Acked Write 4 Done Done Sent false
10210 3641 Acked Read 1 Done Done Sent false
10211 3642 Acked Write 1 Done Done Sent false
10212 3643 Acked Write 9 Done Done Sent false
10213 3644 Acked Write 10 Done Done Sent false
10214 3645 Acked Read 10 Done Done Sent false
10215 3646 Acked Flush 0 Done Done Sent false
10216 3647 Acked Write 7 Done Done Sent false
10217 3648 Acked Read 1 Done Done Sent false
10218 3649 Acked Read 7 Done Done Sent false
10219 3650 Acked Read 9 Done Done Sent false
10220 3651 Acked Read 2 Done Done Sent false
10221 3652 Acked Read 10 Done Done Sent false
10222 3653 Acked Write 3 Done Done Sent false
10223 3654 Acked Write 9 Done Done Sent false
10224 3655 Acked Write 5 Done Done Sent false
10225 3656 Acked Read 6 Done Done Sent false
10226 3657 Acked Flush 0 Done Done Sent false
10227 3658 Acked Read 7 Done Done Sent false
10228 3659 Acked Read 9 Done Done Sent false
10229 3660 Acked Write 1 Done Done Sent false
10230 3661 Acked Read 5 Done Done Sent false
10231 3662 Acked Write 3 Done Done Sent false
10232 3663 Acked Read 5 Done Done Sent false
10233 3664 Acked Read 3 Done Done Sent false
10234 3665 Acked Read 7 Done Done Sent false
10235 3666 Acked Read 10 Done Done Sent false
10236 3667 Acked Read 5 Done Done Sent false
10237 3668 Acked Read 3 Done Done Sent false
10238 3669 Acked Write 7 Done Done Sent false
10239 3670 Acked Read 10 Done Done Sent false
10240 3671 Acked Read 2 Done Done Sent false
10241 3672 Acked Write 6 Done Done Sent false
10242 3673 Acked Write 5 Done Done Sent false
10243 3674 Acked Read 1 Done Done Sent false
10244 3675 Acked Write 2 Done Done Sent false
10245 3676 Acked Write 10 Done Done Sent false
10246 3677 Acked Write 6 Done Done Sent false
10247 3678 Acked Write 9 Done Done Sent false
10248 3679 Acked Read 2 Done Done Sent false
10249 3680 Acked Read 9 Done Done Sent false
10250 3681 Acked Read 3 Done Done Sent false
10251 3682 Acked Read 5 Done Done Sent false
10252 3683 Acked Read 3 Done Done Sent false
10253 3684 Acked Read 2 Done Done Sent false
10254 3685 Acked Write 1 Done Done Sent false
10255 3686 Acked Write 10 Done Done Sent false
10256 3687 Acked Read 3 Done Done Sent false
10257 3688 Acked Flush 0 Done Done Sent false
10258 3689 Acked Write 10 Done Done Sent false
10259 3690 Acked Write 3 Done Done Sent false
10260 3691 Acked Write 3 Done Done Sent false
10261 3692 Acked Write 4 Done Done Sent false
10262 3693 Acked Read 6 Done Done Sent false
10263 3694 Acked Flush 0 Done Done Sent false
10264 3695 Acked Read 5 Done Done Sent false
10265 3696 Acked Write 6 Done Done Sent false
10266 3697 Acked Read 9 Done Done Sent false
10267 3698 Acked Read 10 Done Done Sent false
10268 3699 Acked Write 5 Done Done Sent false
10269 3700 Acked Read 1 Done Done Sent false
10270 3701 Acked Write 1 Done Done Sent false
10271 3702 Acked Read 5 Done Done Sent false
10272 3703 Acked Write 5 Done Done Sent false
10273 3704 Acked Read 8 Done Done Sent false
10274 3705 Acked Write 9 Done Done Sent false
10275 3706 Acked Write 1 Done Done Sent false
10276 3707 Acked Write 1 Done Done Sent false
10277 3708 Acked Read 10 Done Done Sent false
10278 3709 Acked Write 6 Done Done Sent false
10279 3710 Acked Write 1 Done Done Sent false
10280 3711 Acked Write 6 Done Done Sent false
10281 3712 Acked Read 2 Done Done Sent false
10282 3713 Acked Read 8 Done Done Sent false
10283 3714 Acked Read 9 Done Done Sent false
10284 3715 Acked Read 9 Done Done Sent false
10285 STATES DS:0 DS:1 DS:2 TOTAL
10286 Sent 0 0 301 301
10287 Done 301 301 0 602
10288 Skipped 0 0 0 0
10289 Error 0 0 0 0
10290 Last Flush: Some(JobId(3694)) Some(JobId(3694)) Some(JobId(3414))
10291 Downstairs last five completed: 3414 3413 3412 3411 3410
10292 Upstairs last five completed: 3715 3714 3713 3712 3711
10293 CLIENT: Up:0 ds:301 act:2
102942025-03-17T22:18:30.201ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
102952025-03-17T22:18:30.807ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
102962025-03-17T22:18:31.212ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
102972025-03-17T22:18:31.212ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
102982025-03-17T22:18:31.212ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
102992025-03-17T22:18:31.212ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
103002025-03-17T22:18:31.212ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
103012025-03-17T22:18:31.809ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
103022025-03-17T22:18:32.810ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
10303 [8830] got start action
10304 [8830] I am going to restart
10305 Starting downstairs at port 8830
10306 Make output file at "/tmp/dsc/downstairs-8830.txt"
10307 [8830][2] reports Starting
10308 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1215)
10309 [8830][2] reports Running
10310 ----------------------------------------------------------------
10311 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10312 States: Active Active Offline
10313 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10314 3415 Acked Write 10 Done Done Sent false
10315 3416 Acked Write 1 Done Done Sent false
10316 3417 Acked Read 6 Done Done Sent false
10317 3418 Acked Write 10 Done Done Sent false
10318 3419 Acked Write 9 Done Done Sent false
10319 3420 Acked Read 5 Done Done Sent false
10320 3421 Acked Write 9 Done Done Sent false
10321 3422 Acked Read 8 Done Done Sent false
10322 3423 Acked Flush 0 Done Done Sent false
10323 3424 Acked Read 4 Done Done Sent false
10324 3425 Acked Write 2 Done Done Sent false
10325 3426 Acked Read 5 Done Done Sent false
10326 3427 Acked Read 10 Done Done Sent false
10327 3428 Acked Flush 0 Done Done Sent false
10328 3429 Acked Read 6 Done Done Sent false
10329 3430 Acked Read 3 Done Done Sent false
10330 3431 Acked Read 2 Done Done Sent false
10331 3432 Acked Read 10 Done Done Sent false
10332 3433 Acked Read 10 Done Done Sent false
10333 3434 Acked Read 9 Done Done Sent false
10334 3435 Acked Read 9 Done Done Sent false
10335 3436 Acked Read 4 Done Done Sent false
10336 3437 Acked Write 1 Done Done Sent false
10337 3438 Acked Read 8 Done Done Sent false
10338 3439 Acked Write 10 Done Done Sent false
10339 3440 Acked Read 10 Done Done Sent false
10340 3441 Acked Read 7 Done Done Sent false
10341 3442 Acked Read 7 Done Done Sent false
10342 3443 Acked Read 4 Done Done Sent false
10343 3444 Acked Read 4 Done Done Sent false
10344 3445 Acked Write 10 Done Done Sent false
10345 3446 Acked Read 6 Done Done Sent false
10346 3447 Acked Write 6 Done Done Sent false
10347 3448 Acked Read 3 Done Done Sent false
10348 3449 Acked Read 5 Done Done Sent false
10349 3450 Acked Flush 0 Done Done Sent false
10350 3451 Acked Write 10 Done Done Sent false
10351 3452 Acked Write 7 Done Done Sent false
10352 3453 Acked Read 1 Done Done Sent false
10353 3454 Acked Read 9 Done Done Sent false
10354 3455 Acked Write 1 Done Done Sent false
10355 3456 Acked Read 9 Done Done Sent false
10356 3457 Acked Write 10 Done Done Sent false
10357 3458 Acked Read 2 Done Done Sent false
10358 3459 Acked Read 9 Done Done Sent false
10359 3460 Acked Read 4 Done Done Sent false
10360 3461 Acked Read 2 Done Done Sent false
10361 3462 Acked Write 8 Done Done Sent false
10362 3463 Acked Write 5 Done Done Sent false
10363 3464 Acked Flush 0 Done Done Sent false
10364 3465 Acked Read 1 Done Done Sent false
10365 3466 Acked Read 9 Done Done Sent false
10366 3467 Acked Read 3 Done Done Sent false
10367 3468 Acked Read 9 Done Done Sent false
10368 3469 Acked Write 7 Done Done Sent false
10369 3470 Acked Write 9 Done Done Sent false
10370 3471 Acked Write 4 Done Done Sent false
10371 3472 Acked Flush 0 Done Done Sent false
10372 3473 Acked Read 10 Done Done Sent false
10373 3474 Acked Read 9 Done Done Sent false
10374 3475 Acked Read 9 Done Done Sent false
10375 3476 Acked Read 2 Done Done Sent false
10376 3477 Acked Write 10 Done Done Sent false
10377 3478 Acked Write 1 Done Done Sent false
10378 3479 Acked Read 3 Done Done Sent false
10379 3480 Acked Read 2 Done Done Sent false
10380 3481 Acked Write 2 Done Done Sent false
10381 3482 Acked Read 9 Done Done Sent false
10382 3483 Acked Read 1 Done Done Sent false
10383 3484 Acked Write 5 Done Done Sent false
10384 3485 Acked Read 5 Done Done Sent false
10385 3486 Acked Write 6 Done Done Sent false
10386 3487 Acked Read 1 Done Done Sent false
10387 3488 Acked Read 3 Done Done Sent false
10388 3489 Acked Read 8 Done Done Sent false
10389 3490 Acked Write 1 Done Done Sent false
10390 3491 Acked Read 5 Done Done Sent false
10391 3492 Acked Write 6 Done Done Sent false
10392 3493 Acked Write 3 Done Done Sent false
10393 3494 Acked Write 2 Done Done Sent false
10394 3495 Acked Write 8 Done Done Sent false
10395 3496 Acked Write 6 Done Done Sent false
10396 3497 Acked Write 4 Done Done Sent false
10397 3498 Acked Flush 0 Done Done Sent false
10398 3499 Acked Write 10 Done Done Sent false
10399 3500 Acked Write 3 Done Done Sent false
10400 3501 Acked Write 1 Done Done Sent false
10401 3502 Acked Read 1 Done Done Sent false
10402 3503 Acked Read 2 Done Done Sent false
10403 3504 Acked Write 8 Done Done Sent false
10404 3505 Acked Read 4 Done Done Sent false
10405 3506 Acked Read 1 Done Done Sent false
10406 3507 Acked Read 9 Done Done Sent false
10407 3508 Acked Read 7 Done Done Sent false
10408 3509 Acked Flush 0 Done Done Sent false
10409 3510 Acked Read 6 Done Done Sent false
10410 3511 Acked Read 2 Done Done Sent false
10411 3512 Acked Write 2 Done Done Sent false
10412 3513 Acked Read 1 Done Done Sent false
10413 3514 Acked Write 1 Done Done Sent false
10414 3515 Acked Write 9 Done Done Sent false
10415 3516 Acked Write 9 Done Done Sent false
10416 3517 Acked Write 3 Done Done Sent false
10417 3518 Acked Read 1 Done Done Sent false
10418 3519 Acked Write 8 Done Done Sent false
10419 3520 Acked Flush 0 Done Done Sent false
10420 3521 Acked Read 5 Done Done Sent false
10421 3522 Acked Write 4 Done Done Sent false
10422 3523 Acked Write 10 Done Done Sent false
10423 3524 Acked Read 5 Done Done Sent false
10424 3525 Acked Write 1 Done Done Sent false
10425 3526 Acked Read 3 Done Done Sent false
10426 3527 Acked Flush 0 Done Done Sent false
10427 3528 Acked Read 7 Done Done Sent false
10428 3529 Acked Flush 0 Done Done Sent false
10429 3530 Acked Read 9 Done Done Sent false
10430 3531 Acked Write 2 Done Done Sent false
10431 3532 Acked Read 10 Done Done Sent false
10432 3533 Acked Read 2 Done Done Sent false
10433 3534 Acked Write 8 Done Done Sent false
10434 3535 Acked Write 4 Done Done Sent false
10435 3536 Acked Write 6 Done Done Sent false
10436 3537 Acked Write 2 Done Done Sent false
10437 3538 Acked Read 2 Done Done Sent false
10438 3539 Acked Write 1 Done Done Sent false
10439 3540 Acked Read 3 Done Done Sent false
10440 3541 Acked Write 1 Done Done Sent false
10441 3542 Acked Read 5 Done Done Sent false
10442 3543 Acked Read 1 Done Done Sent false
10443 3544 Acked Read 1 Done Done Sent false
10444 3545 Acked Read 9 Done Done Sent false
10445 3546 Acked Read 10 Done Done Sent false
10446 3547 Acked Read 8 Done Done Sent false
10447 3548 Acked Write 7 Done Done Sent false
10448 3549 Acked Read 9 Done Done Sent false
10449 3550 Acked Write 2 Done Done Sent false
10450 3551 Acked Write 2 Done Done Sent false
10451 3552 Acked Read 8 Done Done Sent false
10452 3553 Acked Read 7 Done Done Sent false
10453 3554 Acked Write 9 Done Done Sent false
10454 3555 Acked Flush 0 Done Done Sent false
10455 3556 Acked Flush 0 Done Done Sent false
10456 3557 Acked Read 3 Done Done Sent false
10457 3558 Acked Read 8 Done Done Sent false
10458 3559 Acked Read 3 Done Done Sent false
10459 3560 Acked Read 1 Done Done Sent false
10460 3561 Acked Write 6 Done Done Sent false
10461 3562 Acked Read 2 Done Done Sent false
10462 3563 Acked Read 9 Done Done Sent false
10463 3564 Acked Read 5 Done Done Sent false
10464 3565 Acked Write 2 Done Done Sent false
10465 3566 Acked Read 1 Done Done Sent false
10466 3567 Acked Write 2 Done Done Sent false
10467 3568 Acked Read 1 Done Done Sent false
10468 3569 Acked Write 5 Done Done Sent false
10469 3570 Acked Read 10 Done Done Sent false
10470 3571 Acked Write 8 Done Done Sent false
10471 3572 Acked Read 5 Done Done Sent false
10472 3573 Acked Write 5 Done Done Sent false
10473 3574 Acked Write 1 Done Done Sent false
10474 3575 Acked Write 3 Done Done Sent false
10475 3576 Acked Write 6 Done Done Sent false
10476 3577 Acked Write 3 Done Done Sent false
10477 3578 Acked Read 4 Done Done Sent false
10478 3579 Acked Read 1 Done Done Sent false
10479 3580 Acked Read 6 Done Done Sent false
10480 3581 Acked Read 5 Done Done Sent false
10481 3582 Acked Write 7 Done Done Sent false
10482 3583 Acked Read 3 Done Done Sent false
10483 3584 Acked Read 2 Done Done Sent false
10484 3585 Acked Write 7 Done Done Sent false
10485 3586 Acked Write 6 Done Done Sent false
10486 3587 Acked Write 5 Done Done Sent false
10487 3588 Acked Read 7 Done Done Sent false
10488 3589 Acked Write 6 Done Done Sent false
10489 3590 Acked Flush 0 Done Done Sent false
10490 3591 Acked Read 8 Done Done Sent false
10491 3592 Acked Read 6 Done Done Sent false
10492 3593 Acked Read 8 Done Done Sent false
10493 3594 Acked Write 9 Done Done Sent false
10494 3595 Acked Write 10 Done Done Sent false
10495 3596 Acked Read 2 Done Done Sent false
10496 3597 Acked Flush 0 Done Done Sent false
10497 3598 Acked Read 5 Done Done Sent false
10498 3599 Acked Read 2 Done Done Sent false
10499 3600 Acked Write 2 Done Done Sent false
10500 3601 Acked Write 5 Done Done Sent false
10501 3602 Acked Read 7 Done Done Sent false
10502 3603 Acked Read 9 Done Done Sent false
10503 3604 Acked Write 10 Done Done Sent false
10504 3605 Acked Write 6 Done Done Sent false
10505 3606 Acked Write 3 Done Done Sent false
10506 3607 Acked Write 1 Done Done Sent false
10507 3608 Acked Write 8 Done Done Sent false
10508 3609 Acked Write 3 Done Done Sent false
10509 3610 Acked Read 9 Done Done Sent false
10510 3611 Acked Read 8 Done Done Sent false
10511 3612 Acked Write 5 Done Done Sent false
10512 3613 Acked Read 7 Done Done Sent false
10513 3614 Acked Read 1 Done Done Sent false
10514 3615 Acked Read 8 Done Done Sent false
10515 3616 Acked Read 4 Done Done Sent false
10516 3617 Acked Write 8 Done Done Sent false
10517 3618 Acked Write 2 Done Done Sent false
10518 3619 Acked Read 8 Done Done Sent false
10519 3620 Acked Write 3 Done Done Sent false
10520 3621 Acked Write 2 Done Done Sent false
10521 3622 Acked Read 2 Done Done Sent false
10522 3623 Acked Write 9 Done Done Sent false
10523 3624 Acked Read 7 Done Done Sent false
10524 3625 Acked Write 3 Done Done Sent false
10525 3626 Acked Read 5 Done Done Sent false
10526 3627 Acked Read 9 Done Done Sent false
10527 3628 Acked Write 6 Done Done Sent false
10528 3629 Acked Flush 0 Done Done Sent false
10529 3630 Acked Write 9 Done Done Sent false
10530 3631 Acked Write 8 Done Done Sent false
10531 3632 Acked Write 6 Done Done Sent false
10532 3633 Acked Write 2 Done Done Sent false
10533 3634 Acked Flush 0 Done Done Sent false
10534 3635 Acked Write 6 Done Done Sent false
10535 3636 Acked Read 4 Done Done Sent false
10536 3637 Acked Write 8 Done Done Sent false
10537 3638 Acked Write 1 Done Done Sent false
10538 3639 Acked Flush 0 Done Done Sent false
10539 3640 Acked Write 4 Done Done Sent false
10540 3641 Acked Read 1 Done Done Sent false
10541 3642 Acked Write 1 Done Done Sent false
10542 3643 Acked Write 9 Done Done Sent false
10543 3644 Acked Write 10 Done Done Sent false
10544 3645 Acked Read 10 Done Done Sent false
10545 3646 Acked Flush 0 Done Done Sent false
10546 3647 Acked Write 7 Done Done Sent false
10547 3648 Acked Read 1 Done Done Sent false
10548 3649 Acked Read 7 Done Done Sent false
10549 3650 Acked Read 9 Done Done Sent false
10550 3651 Acked Read 2 Done Done Sent false
10551 3652 Acked Read 10 Done Done Sent false
10552 3653 Acked Write 3 Done Done Sent false
10553 3654 Acked Write 9 Done Done Sent false
10554 3655 Acked Write 5 Done Done Sent false
10555 3656 Acked Read 6 Done Done Sent false
10556 3657 Acked Flush 0 Done Done Sent false
10557 3658 Acked Read 7 Done Done Sent false
10558 3659 Acked Read 9 Done Done Sent false
10559 3660 Acked Write 1 Done Done Sent false
10560 3661 Acked Read 5 Done Done Sent false
10561 3662 Acked Write 3 Done Done Sent false
10562 3663 Acked Read 5 Done Done Sent false
10563 3664 Acked Read 3 Done Done Sent false
10564 3665 Acked Read 7 Done Done Sent false
10565 3666 Acked Read 10 Done Done Sent false
10566 3667 Acked Read 5 Done Done Sent false
10567 3668 Acked Read 3 Done Done Sent false
10568 3669 Acked Write 7 Done Done Sent false
10569 3670 Acked Read 10 Done Done Sent false
10570 3671 Acked Read 2 Done Done Sent false
10571 3672 Acked Write 6 Done Done Sent false
10572 3673 Acked Write 5 Done Done Sent false
10573 3674 Acked Read 1 Done Done Sent false
10574 3675 Acked Write 2 Done Done Sent false
10575 3676 Acked Write 10 Done Done Sent false
10576 3677 Acked Write 6 Done Done Sent false
10577 3678 Acked Write 9 Done Done Sent false
10578 3679 Acked Read 2 Done Done Sent false
10579 3680 Acked Read 9 Done Done Sent false
10580 3681 Acked Read 3 Done Done Sent false
10581 3682 Acked Read 5 Done Done Sent false
10582 3683 Acked Read 3 Done Done Sent false
10583 3684 Acked Read 2 Done Done Sent false
10584 3685 Acked Write 1 Done Done Sent false
10585 3686 Acked Write 10 Done Done Sent false
10586 3687 Acked Read 3 Done Done Sent false
10587 3688 Acked Flush 0 Done Done Sent false
10588 3689 Acked Write 10 Done Done Sent false
10589 3690 Acked Write 3 Done Done Sent false
10590 3691 Acked Write 3 Done Done Sent false
10591 3692 Acked Write 4 Done Done Sent false
10592 3693 Acked Read 6 Done Done Sent false
10593 3694 Acked Flush 0 Done Done Sent false
10594 3695 Acked Read 5 Done Done Sent false
10595 3696 Acked Write 6 Done Done Sent false
10596 3697 Acked Read 9 Done Done Sent false
10597 3698 Acked Read 10 Done Done Sent false
10598 3699 Acked Write 5 Done Done Sent false
10599 3700 Acked Read 1 Done Done Sent false
10600 3701 Acked Write 1 Done Done Sent false
10601 3702 Acked Read 5 Done Done Sent false
10602 3703 Acked Write 5 Done Done Sent false
10603 3704 Acked Read 8 Done Done Sent false
10604 3705 Acked Write 9 Done Done Sent false
10605 3706 Acked Write 1 Done Done Sent false
10606 3707 Acked Write 1 Done Done Sent false
10607 3708 Acked Read 10 Done Done Sent false
10608 3709 Acked Write 6 Done Done Sent false
10609 3710 Acked Write 1 Done Done Sent false
10610 3711 Acked Write 6 Done Done Sent false
10611 3712 Acked Read 2 Done Done Sent false
10612 3713 Acked Read 8 Done Done Sent false
10613 3714 Acked Read 9 Done Done Sent false
10614 3715 Acked Read 9 Done Done Sent false
10615 STATES DS:0 DS:1 DS:2 TOTAL
10616 Sent 0 0 301 301
10617 Done 301 301 0 602
10618 Skipped 0 0 0 0
10619 Error 0 0 0 0
10620 Last Flush: Some(JobId(3694)) Some(JobId(3694)) Some(JobId(3414))
10621 Downstairs last five completed: 3414 3413 3412 3411 3410
10622 Upstairs last five completed: 3715 3714 3713 3712 3711
10623 CLIENT: Up:0 ds:301 act:2
106242025-03-17T22:18:33.961ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
106252025-03-17T22:18:35.201ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
106262025-03-17T22:18:35.815ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
106272025-03-17T22:18:36.816ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
10628 ----------------------------------------------------------------
10629 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10630 States: Active Active Offline
10631 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10632 3415 Acked Write 10 Done Done Sent false
10633 3416 Acked Write 1 Done Done Sent false
10634 3417 Acked Read 6 Done Done Sent false
10635 3418 Acked Write 10 Done Done Sent false
10636 3419 Acked Write 9 Done Done Sent false
10637 3420 Acked Read 5 Done Done Sent false
10638 3421 Acked Write 9 Done Done Sent false
10639 3422 Acked Read 8 Done Done Sent false
10640 3423 Acked Flush 0 Done Done Sent false
10641 3424 Acked Read 4 Done Done Sent false
10642 3425 Acked Write 2 Done Done Sent false
10643 3426 Acked Read 5 Done Done Sent false
10644 3427 Acked Read 10 Done Done Sent false
10645 3428 Acked Flush 0 Done Done Sent false
10646 3429 Acked Read 6 Done Done Sent false
10647 3430 Acked Read 3 Done Done Sent false
10648 3431 Acked Read 2 Done Done Sent false
10649 3432 Acked Read 10 Done Done Sent false
10650 3433 Acked Read 10 Done Done Sent false
10651 3434 Acked Read 9 Done Done Sent false
10652 3435 Acked Read 9 Done Done Sent false
10653 3436 Acked Read 4 Done Done Sent false
10654 3437 Acked Write 1 Done Done Sent false
10655 3438 Acked Read 8 Done Done Sent false
10656 3439 Acked Write 10 Done Done Sent false
10657 3440 Acked Read 10 Done Done Sent false
10658 3441 Acked Read 7 Done Done Sent false
10659 3442 Acked Read 7 Done Done Sent false
10660 3443 Acked Read 4 Done Done Sent false
10661 3444 Acked Read 4 Done Done Sent false
10662 3445 Acked Write 10 Done Done Sent false
10663 3446 Acked Read 6 Done Done Sent false
10664 3447 Acked Write 6 Done Done Sent false
10665 3448 Acked Read 3 Done Done Sent false
10666 3449 Acked Read 5 Done Done Sent false
10667 3450 Acked Flush 0 Done Done Sent false
10668 3451 Acked Write 10 Done Done Sent false
10669 3452 Acked Write 7 Done Done Sent false
10670 3453 Acked Read 1 Done Done Sent false
10671 3454 Acked Read 9 Done Done Sent false
10672 3455 Acked Write 1 Done Done Sent false
10673 3456 Acked Read 9 Done Done Sent false
10674 3457 Acked Write 10 Done Done Sent false
10675 3458 Acked Read 2 Done Done Sent false
10676 3459 Acked Read 9 Done Done Sent false
10677 3460 Acked Read 4 Done Done Sent false
10678 3461 Acked Read 2 Done Done Sent false
10679 3462 Acked Write 8 Done Done Sent false
10680 3463 Acked Write 5 Done Done Sent false
10681 3464 Acked Flush 0 Done Done Sent false
10682 3465 Acked Read 1 Done Done Sent false
10683 3466 Acked Read 9 Done Done Sent false
10684 3467 Acked Read 3 Done Done Sent false
10685 3468 Acked Read 9 Done Done Sent false
10686 3469 Acked Write 7 Done Done Sent false
10687 3470 Acked Write 9 Done Done Sent false
10688 3471 Acked Write 4 Done Done Sent false
10689 3472 Acked Flush 0 Done Done Sent false
10690 3473 Acked Read 10 Done Done Sent false
10691 3474 Acked Read 9 Done Done Sent false
10692 3475 Acked Read 9 Done Done Sent false
10693 3476 Acked Read 2 Done Done Sent false
10694 3477 Acked Write 10 Done Done Sent false
10695 3478 Acked Write 1 Done Done Sent false
10696 3479 Acked Read 3 Done Done Sent false
10697 3480 Acked Read 2 Done Done Sent false
10698 3481 Acked Write 2 Done Done Sent false
10699 3482 Acked Read 9 Done Done Sent false
10700 3483 Acked Read 1 Done Done Sent false
10701 3484 Acked Write 5 Done Done Sent false
10702 3485 Acked Read 5 Done Done Sent false
10703 3486 Acked Write 6 Done Done Sent false
10704 3487 Acked Read 1 Done Done Sent false
10705 3488 Acked Read 3 Done Done Sent false
10706 3489 Acked Read 8 Done Done Sent false
10707 3490 Acked Write 1 Done Done Sent false
10708 3491 Acked Read 5 Done Done Sent false
10709 3492 Acked Write 6 Done Done Sent false
10710 3493 Acked Write 3 Done Done Sent false
10711 3494 Acked Write 2 Done Done Sent false
10712 3495 Acked Write 8 Done Done Sent false
10713 3496 Acked Write 6 Done Done Sent false
10714 3497 Acked Write 4 Done Done Sent false
10715 3498 Acked Flush 0 Done Done Sent false
10716 3499 Acked Write 10 Done Done Sent false
10717 3500 Acked Write 3 Done Done Sent false
10718 3501 Acked Write 1 Done Done Sent false
10719 3502 Acked Read 1 Done Done Sent false
10720 3503 Acked Read 2 Done Done Sent false
10721 3504 Acked Write 8 Done Done Sent false
10722 3505 Acked Read 4 Done Done Sent false
10723 3506 Acked Read 1 Done Done Sent false
10724 3507 Acked Read 9 Done Done Sent false
10725 3508 Acked Read 7 Done Done Sent false
10726 3509 Acked Flush 0 Done Done Sent false
10727 3510 Acked Read 6 Done Done Sent false
10728 3511 Acked Read 2 Done Done Sent false
10729 3512 Acked Write 2 Done Done Sent false
10730 3513 Acked Read 1 Done Done Sent false
10731 3514 Acked Write 1 Done Done Sent false
10732 3515 Acked Write 9 Done Done Sent false
10733 3516 Acked Write 9 Done Done Sent false
10734 3517 Acked Write 3 Done Done Sent false
10735 3518 Acked Read 1 Done Done Sent false
10736 3519 Acked Write 8 Done Done Sent false
10737 3520 Acked Flush 0 Done Done Sent false
10738 3521 Acked Read 5 Done Done Sent false
10739 3522 Acked Write 4 Done Done Sent false
10740 3523 Acked Write 10 Done Done Sent false
10741 3524 Acked Read 5 Done Done Sent false
10742 3525 Acked Write 1 Done Done Sent false
10743 3526 Acked Read 3 Done Done Sent false
10744 3527 Acked Flush 0 Done Done Sent false
10745 3528 Acked Read 7 Done Done Sent false
10746 3529 Acked Flush 0 Done Done Sent false
10747 3530 Acked Read 9 Done Done Sent false
10748 3531 Acked Write 2 Done Done Sent false
10749 3532 Acked Read 10 Done Done Sent false
10750 3533 Acked Read 2 Done Done Sent false
10751 3534 Acked Write 8 Done Done Sent false
10752 3535 Acked Write 4 Done Done Sent false
10753 3536 Acked Write 6 Done Done Sent false
10754 3537 Acked Write 2 Done Done Sent false
10755 3538 Acked Read 2 Done Done Sent false
10756 3539 Acked Write 1 Done Done Sent false
10757 3540 Acked Read 3 Done Done Sent false
10758 3541 Acked Write 1 Done Done Sent false
10759 3542 Acked Read 5 Done Done Sent false
10760 3543 Acked Read 1 Done Done Sent false
10761 3544 Acked Read 1 Done Done Sent false
10762 3545 Acked Read 9 Done Done Sent false
10763 3546 Acked Read 10 Done Done Sent false
10764 3547 Acked Read 8 Done Done Sent false
10765 3548 Acked Write 7 Done Done Sent false
10766 3549 Acked Read 9 Done Done Sent false
10767 3550 Acked Write 2 Done Done Sent false
10768 3551 Acked Write 2 Done Done Sent false
10769 3552 Acked Read 8 Done Done Sent false
10770 3553 Acked Read 7 Done Done Sent false
10771 3554 Acked Write 9 Done Done Sent false
10772 3555 Acked Flush 0 Done Done Sent false
10773 3556 Acked Flush 0 Done Done Sent false
10774 3557 Acked Read 3 Done Done Sent false
10775 3558 Acked Read 8 Done Done Sent false
10776 3559 Acked Read 3 Done Done Sent false
10777 3560 Acked Read 1 Done Done Sent false
10778 3561 Acked Write 6 Done Done Sent false
10779 3562 Acked Read 2 Done Done Sent false
10780 3563 Acked Read 9 Done Done Sent false
10781 3564 Acked Read 5 Done Done Sent false
10782 3565 Acked Write 2 Done Done Sent false
10783 3566 Acked Read 1 Done Done Sent false
10784 3567 Acked Write 2 Done Done Sent false
10785 3568 Acked Read 1 Done Done Sent false
10786 3569 Acked Write 5 Done Done Sent false
10787 3570 Acked Read 10 Done Done Sent false
10788 3571 Acked Write 8 Done Done Sent false
10789 3572 Acked Read 5 Done Done Sent false
10790 3573 Acked Write 5 Done Done Sent false
10791 3574 Acked Write 1 Done Done Sent false
10792 3575 Acked Write 3 Done Done Sent false
10793 3576 Acked Write 6 Done Done Sent false
10794 3577 Acked Write 3 Done Done Sent false
10795 3578 Acked Read 4 Done Done Sent false
10796 3579 Acked Read 1 Done Done Sent false
10797 3580 Acked Read 6 Done Done Sent false
10798 3581 Acked Read 5 Done Done Sent false
10799 3582 Acked Write 7 Done Done Sent false
10800 3583 Acked Read 3 Done Done Sent false
10801 3584 Acked Read 2 Done Done Sent false
10802 3585 Acked Write 7 Done Done Sent false
10803 3586 Acked Write 6 Done Done Sent false
10804 3587 Acked Write 5 Done Done Sent false
10805 3588 Acked Read 7 Done Done Sent false
10806 3589 Acked Write 6 Done Done Sent false
10807 3590 Acked Flush 0 Done Done Sent false
10808 3591 Acked Read 8 Done Done Sent false
10809 3592 Acked Read 6 Done Done Sent false
10810 3593 Acked Read 8 Done Done Sent false
10811 3594 Acked Write 9 Done Done Sent false
10812 3595 Acked Write 10 Done Done Sent false
10813 3596 Acked Read 2 Done Done Sent false
10814 3597 Acked Flush 0 Done Done Sent false
10815 3598 Acked Read 5 Done Done Sent false
10816 3599 Acked Read 2 Done Done Sent false
10817 3600 Acked Write 2 Done Done Sent false
10818 3601 Acked Write 5 Done Done Sent false
10819 3602 Acked Read 7 Done Done Sent false
10820 3603 Acked Read 9 Done Done Sent false
10821 3604 Acked Write 10 Done Done Sent false
10822 3605 Acked Write 6 Done Done Sent false
10823 3606 Acked Write 3 Done Done Sent false
10824 3607 Acked Write 1 Done Done Sent false
10825 3608 Acked Write 8 Done Done Sent false
10826 3609 Acked Write 3 Done Done Sent false
10827 3610 Acked Read 9 Done Done Sent false
10828 3611 Acked Read 8 Done Done Sent false
10829 3612 Acked Write 5 Done Done Sent false
10830 3613 Acked Read 7 Done Done Sent false
10831 3614 Acked Read 1 Done Done Sent false
10832 3615 Acked Read 8 Done Done Sent false
10833 3616 Acked Read 4 Done Done Sent false
10834 3617 Acked Write 8 Done Done Sent false
10835 3618 Acked Write 2 Done Done Sent false
10836 3619 Acked Read 8 Done Done Sent false
10837 3620 Acked Write 3 Done Done Sent false
10838 3621 Acked Write 2 Done Done Sent false
10839 3622 Acked Read 2 Done Done Sent false
10840 3623 Acked Write 9 Done Done Sent false
10841 3624 Acked Read 7 Done Done Sent false
10842 3625 Acked Write 3 Done Done Sent false
10843 3626 Acked Read 5 Done Done Sent false
10844 3627 Acked Read 9 Done Done Sent false
10845 3628 Acked Write 6 Done Done Sent false
10846 3629 Acked Flush 0 Done Done Sent false
10847 3630 Acked Write 9 Done Done Sent false
10848 3631 Acked Write 8 Done Done Sent false
10849 3632 Acked Write 6 Done Done Sent false
10850 3633 Acked Write 2 Done Done Sent false
10851 3634 Acked Flush 0 Done Done Sent false
10852 3635 Acked Write 6 Done Done Sent false
10853 3636 Acked Read 4 Done Done Sent false
10854 3637 Acked Write 8 Done Done Sent false
10855 3638 Acked Write 1 Done Done Sent false
10856 3639 Acked Flush 0 Done Done Sent false
10857 3640 Acked Write 4 Done Done Sent false
10858 3641 Acked Read 1 Done Done Sent false
10859 3642 Acked Write 1 Done Done Sent false
10860 3643 Acked Write 9 Done Done Sent false
10861 3644 Acked Write 10 Done Done Sent false
10862 3645 Acked Read 10 Done Done Sent false
10863 3646 Acked Flush 0 Done Done Sent false
10864 3647 Acked Write 7 Done Done Sent false
10865 3648 Acked Read 1 Done Done Sent false
10866 3649 Acked Read 7 Done Done Sent false
10867 3650 Acked Read 9 Done Done Sent false
10868 3651 Acked Read 2 Done Done Sent false
10869 3652 Acked Read 10 Done Done Sent false
10870 3653 Acked Write 3 Done Done Sent false
10871 3654 Acked Write 9 Done Done Sent false
10872 3655 Acked Write 5 Done Done Sent false
10873 3656 Acked Read 6 Done Done Sent false
10874 3657 Acked Flush 0 Done Done Sent false
10875 3658 Acked Read 7 Done Done Sent false
10876 3659 Acked Read 9 Done Done Sent false
10877 3660 Acked Write 1 Done Done Sent false
10878 3661 Acked Read 5 Done Done Sent false
10879 3662 Acked Write 3 Done Done Sent false
10880 3663 Acked Read 5 Done Done Sent false
10881 3664 Acked Read 3 Done Done Sent false
10882 3665 Acked Read 7 Done Done Sent false
10883 3666 Acked Read 10 Done Done Sent false
10884 3667 Acked Read 5 Done Done Sent false
10885 3668 Acked Read 3 Done Done Sent false
10886 3669 Acked Write 7 Done Done Sent false
10887 3670 Acked Read 10 Done Done Sent false
10888 3671 Acked Read 2 Done Done Sent false
10889 3672 Acked Write 6 Done Done Sent false
10890 3673 Acked Write 5 Done Done Sent false
10891 3674 Acked Read 1 Done Done Sent false
10892 3675 Acked Write 2 Done Done Sent false
10893 3676 Acked Write 10 Done Done Sent false
10894 3677 Acked Write 6 Done Done Sent false
10895 3678 Acked Write 9 Done Done Sent false
10896 3679 Acked Read 2 Done Done Sent false
10897 3680 Acked Read 9 Done Done Sent false
10898 3681 Acked Read 3 Done Done Sent false
10899 3682 Acked Read 5 Done Done Sent false
10900 3683 Acked Read 3 Done Done Sent false
10901 3684 Acked Read 2 Done Done Sent false
10902 3685 Acked Write 1 Done Done Sent false
10903 3686 Acked Write 10 Done Done Sent false
10904 3687 Acked Read 3 Done Done Sent false
10905 3688 Acked Flush 0 Done Done Sent false
10906 3689 Acked Write 10 Done Done Sent false
10907 3690 Acked Write 3 Done Done Sent false
10908 3691 Acked Write 3 Done Done Sent false
10909 3692 Acked Write 4 Done Done Sent false
10910 3693 Acked Read 6 Done Done Sent false
10911 3694 Acked Flush 0 Done Done Sent false
10912 3695 Acked Read 5 Done Done Sent false
10913 3696 Acked Write 6 Done Done Sent false
10914 3697 Acked Read 9 Done Done Sent false
10915 3698 Acked Read 10 Done Done Sent false
10916 3699 Acked Write 5 Done Done Sent false
10917 3700 Acked Read 1 Done Done Sent false
10918 3701 Acked Write 1 Done Done Sent false
10919 3702 Acked Read 5 Done Done Sent false
10920 3703 Acked Write 5 Done Done Sent false
10921 3704 Acked Read 8 Done Done Sent false
10922 3705 Acked Write 9 Done Done Sent false
10923 3706 Acked Write 1 Done Done Sent false
10924 3707 Acked Write 1 Done Done Sent false
10925 3708 Acked Read 10 Done Done Sent false
10926 3709 Acked Write 6 Done Done Sent false
10927 3710 Acked Write 1 Done Done Sent false
10928 3711 Acked Write 6 Done Done Sent false
10929 3712 Acked Read 2 Done Done Sent false
10930 3713 Acked Read 8 Done Done Sent false
10931 3714 Acked Read 9 Done Done Sent false
10932 3715 Acked Read 9 Done Done Sent false
10933 STATES DS:0 DS:1 DS:2 TOTAL
10934 Sent 0 0 301 301
10935 Done 301 301 0 602
10936 Skipped 0 0 0 0
10937 Error 0 0 0 0
10938 Last Flush: Some(JobId(3694)) Some(JobId(3694)) Some(JobId(3414))
10939 Downstairs last five completed: 3414 3413 3412 3411 3410
10940 Upstairs last five completed: 3715 3714 3713 3712 3711
10941 CLIENT: Up:0 ds:301 act:2
109422025-03-17T22:18:37.964ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
109432025-03-17T22:18:38.819ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
109442025-03-17T22:18:40.202ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
109452025-03-17T22:18:40.822ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
109462025-03-17T22:18:41.214ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
109472025-03-17T22:18:41.214ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:35121) = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
109482025-03-17T22:18:41.215ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
109492025-03-17T22:18:41.216ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
109502025-03-17T22:18:41.216ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
109512025-03-17T22:18:41.216ZINFOcrucible: send last flush ID to this DS: Some(JobId(3414)) = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
109522025-03-17T22:18:41.216ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
109532025-03-17T22:18:41.216ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(3414)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
109542025-03-17T22:18:41.216ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(3414)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
10955 ----------------------------------------------------------------
10956 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:21
10957 States: Active Active Active
10958 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
10959 3695 Acked Read 5 Done Done Done true
10960 3696 Acked Write 6 Done Done Done true
10961 3697 Acked Read 9 Done Done Done true
10962 3698 Acked Read 10 Done Done Done true
10963 3699 Acked Write 5 Done Done Done true
10964 3700 Acked Read 1 Done Done Done true
10965 3701 Acked Write 1 Done Done Done true
10966 3702 Acked Read 5 Done Done Done true
10967 3703 Acked Write 5 Done Done Done true
10968 3704 Acked Read 8 Done Done Done true
10969 3705 Acked Write 9 Done Done Done true
10970 3706 Acked Write 1 Done Done Done true
10971 3707 Acked Write 1 Done Done Done true
10972 3708 Acked Read 10 Done Done Done true
10973 3709 Acked Write 6 Done Done Done true
10974 3710 Acked Write 1 Done Done Done true
10975 3711 Acked Write 6 Done Done Done true
10976 3712 Acked Read 2 Done Done Done true
10977 3713 Acked Read 8 Done Done Done true
10978 3714 Acked Read 9 Done Done Done true
10979 3715 Acked Read 9 Done Done Done true
10980 STATES DS:0 DS:1 DS:2 TOTAL
10981 Sent 0 0 0 0
10982 Done 21 21 21 63
10983 Skipped 0 0 0 0
10984 Error 0 0 0 0
10985 Last Flush: Some(JobId(3694)) Some(JobId(3694)) Some(JobId(3694))
10986 Downstairs last five completed: 3694 3693 3692 3691 3690
10987 Upstairs last five completed: 3715 3714 3713 3712 3711
10988 CLIENT: Up:0 ds:21 act:3
10989 ----------------------------------------------------------------
10990 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
10991 Downstairs last five completed: 3716 3715 3714 3713 3712
10992 Upstairs last five completed: 3716 3715 3714 3713 3712
10993 CLIENT: Up:0 ds:0 act:3
10994 Replay: All jobs finished
10995 Main task has work to do, go find it
10996 got dsc Stop(2)
10997 stop 2
10998 Mar 17 22:18:45.466 INFO request completed, latency_us: 360, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 61c3a493-2921-40ff-9a9b-fd3eac56a910, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
10999 [8830] Got stop action so:false kr:false
11000 Mar 17 22:18:45.467 INFO request completed, latency_us: 137, response_code: 200, uri: /state/cid/2, method: GET, req_id: f80167b6-630a-4542-a8bc-c234c55e1d4b, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
110012025-03-17T22:18:45.467ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
110022025-03-17T22:18:45.467ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
110032025-03-17T22:18:45.467ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
110042025-03-17T22:18:45.467ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
11005 [8830] Exited with: ExitStatus(unix_wait_status(9))
11006 [8830][2] reports Exit
11007 Mar 17 22:18:49.469 INFO request completed, latency_us: 217, response_code: 200, uri: /state/cid/2, method: GET, req_id: 0867e2b0-5a24-40be-a034-78320852bf7c, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
11008 001/300 Read block 26 len 12288
11009 002/300 Write block 4192 len 4096 data: 2
11010 003/300 Read block 411 len 16384
11011 004/300 Write block 1179 len 28672 data: 3 3 3 3 3 3 3
11012 005/300 Read block 3540 len 36864
11013 006/300 Read block 1131 len 20480
11014 007/300 Write block 3441 len 32768 data: 3 2 3 4 4 4 3 3
11015 008/300 Write block 1454 len 40960 data: 2 2 3 4 4 5 4 4 3 3
11016 009/300 Read block 933 len 12288
11017 010/300 Read block 4332 len 16384
11018 011/300 Write block 401 len 32768 data: 3 2 2 2 2 2 2 2
11019 012/300 Read block 1553 len 4096
11020 013/300 Write block 3817 len 12288 data: 3 3 3
11021 014/300 Write block 1488 len 20480 data: 3 4 3 2 3
11022 015/300 Write block 1485 len 24576 data: 4 3 3 4 5 4
11023 016/300 Read block 2008 len 4096
11024 017/300 Write block 959 len 4096 data: 4
11025 018/300 Read block 2319 len 24576
11026 019/300 Write block 3907 len 4096 data: 2
11027 020/300 Write block 2967 len 12288 data: 4 4 4
11028 021/300 Write block 800 len 32768 data: 3 3 2 2 2 2 2 2
11029 022/300 Write block 200 len 16384 data: 3 3 3 3
11030 023/300 Flush
11031 024/300 Write block 3182 len 12288 data: 3 3 2
11032 025/300 Read block 4037 len 36864
11033 026/300 Read block 1125 len 20480
11034 027/300 Write block 4744 len 36864 data: 3 3 3 3 3 3 2 3 3
11035 028/300 Read block 1217 len 4096
11036 029/300 Read block 2521 len 36864
11037 030/300 Write block 3416 len 12288 data: 3 3 2
11038 031/300 Flush
11039 032/300 Write block 140 len 24576 data: 2 2 2 2 2 2
11040 033/300 Read block 4407 len 24576
11041 034/300 Write block 4857 len 32768 data: 3 3 5 6 6 6 6 5
11042 035/300 Read block 2520 len 16384
11043 036/300 Write block 4448 len 28672 data: 3 4 4 4 4 3 3
11044 037/300 Write block 2732 len 4096 data: 2
11045 038/300 Read block 3235 len 20480
11046 039/300 Read block 1814 len 8192
11047 040/300 Write block 4810 len 8192 data: 5 4
11048 041/300 Read block 1081 len 4096
11049 042/300 Write block 3824 len 24576 data: 3 3 3 3 4 4
11050 043/300 Write block 3579 len 24576 data: 2 2 2 2 2 3
11051 044/300 Read block 655 len 8192
11052 045/300 Read block 3769 len 28672
11053 046/300 Read block 4513 len 24576
11054 047/300 Write block 544 len 40960 data: 4 2 2 2 2 2 2 2 2 3
11055 048/300 Read block 4738 len 32768
11056 049/300 Write block 1413 len 4096 data: 4
11057 050/300 Write block 4077 len 4096 data: 2
11058 051/300 Flush
11059 052/300 Read block 4405 len 12288
11060 053/300 Read block 2037 len 12288
11061 054/300 Write block 3683 len 24576 data: 2 2 2 2 2 2
11062 055/300 Read block 4122 len 12288
11063 056/300 Read block 3773 len 12288
11064 057/300 Read block 280 len 40960
11065 058/300 Read block 3449 len 24576
11066 059/300 Read block 4982 len 28672
11067 060/300 Read block 2511 len 32768
11068 061/300 Read block 619 len 36864
11069 062/300 Read block 1191 len 16384
11070 063/300 Read block 43 len 16384
11071 064/300 Flush
11072 065/300 Write block 2096 len 4096 data: 2
11073 066/300 Write block 3167 len 32768 data: 4 4 3 3 3 2 3 3
11074 067/300 Read block 909 len 4096
11075 068/300 Write block 1152 len 12288 data: 5 5 3
11076 069/300 Read block 2706 len 32768
11077 070/300 Read block 1266 len 4096
11078 071/300 Flush
11079 072/300 Read block 832 len 40960
11080 073/300 Read block 3335 len 28672
11081 074/300 Read block 2450 len 28672
11082 075/300 Flush
11083 076/300 Write block 2604 len 8192 data: 5 4
11084 077/300 Read block 1923 len 12288
11085 078/300 Read block 4791 len 8192
11086 079/300 Read block 1598 len 36864
11087 080/300 Write block 3494 len 32768 data: 3 4 3 3 2 2 2 2
11088 081/300 Flush
11089 082/300 Read block 339 len 36864
11090 083/300 Write block 1182 len 28672 data: 4 4 4 4 2 2 3
11091 084/300 Write block 3894 len 36864 data: 2 3 4 5 5 5 5 5 4
11092 085/300 Write block 2953 len 40960 data: 4 4 3 5 6 6 6 6 5 4
11093 086/300 Write block 2427 len 36864 data: 3 2 3 3 2 2 2 3 4
11094 087/300 Read block 173 len 16384
11095 088/300 Write block 1356 len 20480 data: 4 5 5 5 5
11096 089/300 Read block 3214 len 8192
11097 090/300 Read block 1774 len 32768
11098 091/300 Flush
11099 092/300 Write block 445 len 28672 data: 4 4 4 3 3 3 3
11100 093/300 Write block 2922 len 40960 data: 3 2 2 2 4 4 4 4 7 6
11101 094/300 Flush
11102 095/300 Read block 2731 len 20480
11103 096/300 Read block 4199 len 16384
11104 097/300 Write block 2986 len 40960 data: 2 3 3 3 3 3 3 4 3 3
11105 098/300 Write block 4454 len 12288 data: 4 3 3
11106 099/300 Write block 3263 len 12288 data: 3 3 3
11107 100/300 Read block 1405 len 8192
11108 101/300 Write block 2548 len 28672 data: 4 4 4 2 2 2 2
11109 102/300 Read block 1801 len 12288
11110 103/300 Read block 4590 len 24576
11111 104/300 Read block 3064 len 32768
11112 105/300 Write block 946 len 8192 data: 3 3
11113 106/300 Write block 4720 len 16384 data: 4 5 5 6
11114 107/300 Write block 1689 len 16384 data: 3 3 3 3
11115 108/300 Read block 4070 len 8192
11116 109/300 Read block 2287 len 16384
11117 110/300 Write block 4542 len 36864 data: 2 2 2 3 2 4 3 4 3
11118 111/300 Read block 3602 len 4096
11119 112/300 Read block 1441 len 36864
11120 113/300 Read block 4378 len 36864
11121 114/300 Write block 410 len 40960 data: 2 3 3 3 3 4 4 4 4 3
11122 115/300 Read block 1928 len 40960
11123 116/300 Read block 1496 len 20480
11124 117/300 Flush
11125 118/300 Flush
11126 119/300 Read block 3223 len 12288
11127 120/300 Write block 4837 len 8192 data: 3 3
11128 121/300 Write block 218 len 8192 data: 3 2
11129 122/300 Read block 851 len 16384
11130 123/300 Flush
11131 124/300 Flush
11132 125/300 Write block 1126 len 24576 data: 2 3 3 2 2 3
11133 126/300 Read block 2718 len 24576
11134 127/300 Read block 4184 len 20480
11135 128/300 Read block 1189 len 32768
11136 129/300 Read block 2548 len 12288
11137 130/300 Flush
11138 131/300 Read block 3724 len 24576
11139 132/300 Read block 3487 len 20480
11140 133/300 Write block 531 len 4096 data: 2
11141 134/300 Write block 4553 len 28672 data: 3 3 3 3 2 2 2
11142 135/300 Read block 1428 len 4096
11143 136/300 Read block 1870 len 24576
11144 137/300 Read block 2686 len 32768
11145 138/300 Read block 2449 len 36864
11146 139/300 Read block 3783 len 12288
11147 140/300 Read block 4639 len 12288
11148 141/300 Read block 3081 len 40960
11149 142/300 Read block 4454 len 28672
11150 143/300 Flush
11151 144/300 Write block 548 len 28672 data: 3 3 3 3 3 4 3
11152 145/300 Write block 2641 len 36864 data: 2 2 2 2 2 2 2 2 2
11153 146/300 Read block 3111 len 40960
11154 147/300 Write block 2543 len 4096 data: 4
11155 148/300 Read block 1778 len 36864
11156 149/300 Read block 2685 len 24576
11157 150/300 Write block 2770 len 8192 data: 5 5
11158 151/300 Read block 4223 len 16384
11159 152/300 Write block 906 len 24576 data: 3 3 2 2 2 2
11160 153/300 Read block 1465 len 36864
11161 154/300 Read block 1613 len 4096
11162 155/300 Read block 1756 len 4096
11163 156/300 Write block 1102 len 20480 data: 5 4 4 3 3
11164 157/300 Write block 3169 len 40960 data: 4 4 4 3 4 4 3 3 3 3
11165 158/300 Read block 1395 len 8192
11166 159/300 Read block 513 len 24576
11167 160/300 Read block 2423 len 32768
11168 161/300 Flush
11169 162/300 Write block 4438 len 36864 data: 4 4 4 4 4 4 4 4 2
11170 163/300 Write block 885 len 24576 data: 3 3 4 4 4 4
11171 164/300 Flush
11172 165/300 Write block 3858 len 36864 data: 3 4 4 4 4 3 3 3 2
11173 166/300 Read block 234 len 16384
11174 167/300 Write block 4775 len 40960 data: 4 4 4 4 5 5 5 5 5 4
11175 168/300 Write block 2268 len 36864 data: 2 2 2 2 2 2 2 2 2
11176 169/300 Write block 1478 len 40960 data: 5 4 4 5 4 4 4 5 4 4
11177 170/300 Write block 928 len 12288 data: 4 3 5
11178 171/300 Read block 2716 len 20480
11179 172/300 Write block 665 len 36864 data: 2 2 2 2 2 3 3 2 2
11180 173/300 Write block 4269 len 28672 data: 2 2 4 4 4 3 3
11181 174/300 Read block 4838 len 40960
11182 175/300 Read block 826 len 32768
11183 176/300 Read block 137 len 20480
11184 177/300 Read block 33 len 16384
11185 178/300 Flush
11186 179/300 Read block 3870 len 28672
11187 180/300 Write block 4717 len 12288 data: 4 4 4
11188 181/300 Read block 833 len 12288
11189 182/300 Read block 1087 len 36864
11190 183/300 Read block 1157 len 20480
11191 184/300 Write block 3417 len 36864 data: 4 3 2 2 2 4 4 4 4
11192 185/300 Read block 3667 len 40960
11193 186/300 Read block 528 len 40960
11194 187/300 Write block 1777 len 28672 data: 3 3 2 2 2 2 2
11195 188/300 Write block 2943 len 24576 data: 4 3 2 4 4 4
11196 189/300 Read block 2486 len 20480
11197 190/300 Read block 1838 len 40960
11198 191/300 Flush
11199 192/300 Flush
11200 193/300 Write block 2598 len 20480 data: 4 5 4 3 3
11201 194/300 Write block 3581 len 12288 data: 3 3 3
11202 195/300 Write block 2773 len 8192 data: 3 3
11203 196/300 Write block 926 len 36864 data: 4 4 5 4 6 5 5 5 4
11204 197/300 Write block 2900 len 12288 data: 3 2 2
11205 198/300 Write block 2592 len 8192 data: 4 4
11206 199/300 Write block 2728 len 36864 data: 2 2 2 2 3 2 2 2 2
11207 200/300 Write block 3514 len 28672 data: 4 2 2 2 2 2 2
11208 201/300 Read block 107 len 12288
11209 202/300 Read block 4423 len 36864
11210 203/300 Read block 853 len 4096
11211 204/300 Write block 3236 len 20480 data: 4 4 5 5 4
11212 205/300 Read block 1923 len 12288
11213 206/300 Read block 259 len 28672
11214 207/300 Read block 4234 len 16384
11215 208/300 Read block 4518 len 20480
11216 209/300 Read block 1574 len 28672
11217 210/300 Flush
11218 211/300 Write block 4464 len 40960 data: 5 5 5 4 5 6 7 7 6 5
11219 212/300 Flush
11220 213/300 Flush
11221 214/300 Write block 1430 len 32768 data: 2 2 2 3 4 3 3 3
11222 215/300 Read block 2186 len 16384
11223 216/300 Read block 3057 len 8192
11224 217/300 Write block 3397 len 28672 data: 3 3 3 2 2 3 4
11225 218/300 Write block 3427 len 4096 data: 3
11226 219/300 Flush
11227 220/300 Write block 4147 len 12288 data: 4 4 5
11228 221/300 Read block 2158 len 32768
11229 222/300 Read block 2247 len 8192
11230 223/300 Read block 4625 len 12288
11231 224/300 Write block 2658 len 40960 data: 3 4 3 3 2 2 3 3 3 5
11232 225/300 Write block 2824 len 28672 data: 6 6 6 6 4 4 5
11233 226/300 Write block 4701 len 24576 data: 2 2 3 3 3 4
11234 227/300 Read block 2225 len 12288
11235 228/300 Write block 1295 len 28672 data: 3 3 3 3 3 3 3
11236 229/300 Write block 4819 len 24576 data: 2 2 3 4 3 3
11237 230/300 Read block 343 len 24576
11238 231/300 Read block 3351 len 36864
11239 232/300 Flush
11240 233/300 Read block 3338 len 24576
11241 234/300 Read block 2377 len 24576
11242 235/300 Read block 3824 len 36864
11243 236/300 Read block 4992 len 4096
11244 237/300 Read block 3694 len 20480
11245 238/300 Write block 489 len 24576 data: 2 3 3 3 4 4
11246 239/300 Write block 3112 len 16384 data: 3 2 3 3
11247 240/300 Write block 1993 len 16384 data: 2 2 2 4
11248 241/300 Write block 4377 len 32768 data: 4 3 2 2 2 2 3 3
11249 242/300 Read block 2290 len 40960
11250 243/300 Write block 1673 len 8192 data: 5 5
11251 244/300 Read block 3087 len 32768
11252 245/300 Read block 1231 len 8192
11253 246/300 Read block 1990 len 24576
11254 247/300 Read block 3452 len 12288
11255 248/300 Read block 961 len 20480
11256 249/300 Read block 4460 len 28672
11257 250/300 Flush
11258 251/300 Read block 671 len 8192
11259 252/300 Read block 3566 len 28672
11260 253/300 Read block 2510 len 32768
11261 254/300 Read block 3536 len 28672
11262 255/300 Flush
11263 256/300 Read block 2345 len 8192
11264 257/300 Read block 3100 len 32768
11265 258/300 Read block 1945 len 8192
11266 259/300 Read block 380 len 24576
11267 260/300 Read block 934 len 20480
11268 261/300 Write block 1694 len 4096 data: 4
11269 262/300 Flush
11270 263/300 Write block 2070 len 28672 data: 3 4 6 6 7 5 4
11271 264/300 Write block 4328 len 24576 data: 3 2 3 3 3 3
11272 265/300 Read block 2979 len 36864
11273 266/300 Read block 3964 len 24576
11274 267/300 Read block 566 len 24576
11275 268/300 Read block 1128 len 32768
11276 269/300 Read block 90 len 16384
11277 270/300 Read block 2443 len 24576
11278 271/300 Read block 2822 len 16384
11279 272/300 Read block 4103 len 20480
11280 273/300 Flush
11281 274/300 Read block 1507 len 36864
11282 275/300 Flush
11283 276/300 Read block 3606 len 20480
11284 277/300 Write block 2432 len 20480 data: 3 3 4 5 4
11285 278/300 Read block 4013 len 8192
11286 279/300 Write block 4502 len 40960 data: 3 3 2 2 2 3 3 3 3 3
11287 280/300 Write block 3371 len 12288 data: 2 2 2
11288 281/300 Read block 3915 len 8192
11289 282/300 Read block 1665 len 20480
11290 283/300 Read block 4983 len 32768
11291 284/300 Flush
11292 285/300 Read block 290 len 36864
11293 286/300 Read block 4751 len 20480
11294 287/300 Read block 1529 len 28672
11295 288/300 Flush
11296 289/300 Read block 39 len 20480
11297 290/300 Read block 145 len 32768
11298 291/300 Write block 1219 len 24576 data: 5 4 5 5 4 4
11299 292/300 Read block 1248 len 40960
11300 293/300 Write block 4031 len 8192 data: 2 2
11301 294/300 Read block 3844 len 24576
11302 295/300 Read block 511 len 32768
11303 296/300 Write block 1850 len 24576 data: 3 3 3 3 2 2
11304 297/300 Read block 2951 len 36864
11305 298/300 Write block 2407 len 28672 data: 3 3 3 3 3 3 3
11306 299/300 Read block 3411 len 8192
11307 300/300 Read block 2299 len 36864
11308 301/300 Write block 3795 len 20480 data: 3 4 4 4 6
11309 Main task has work to do, go find it
11310 got dsc Start(2)
11311 start 2
11312 Mar 17 22:18:49.727 INFO request completed, latency_us: 221, response_code: 204, uri: /start/cid/2, method: POST, req_id: 6bd521d1-4bfb-459a-b8dc-1f152dde6c7c, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
11313 Replay: started 2, returned:Ok(())
11314 ----------------------------------------------------------------
11315 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11316 States: Active Active Offline
11317 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11318 3717 Acked Read 3 Done Done Sent false
11319 3718 Acked Write 1 Done Done Sent false
11320 3719 Acked Read 4 Done Done Sent false
11321 3720 Acked Write 7 Done Done Sent false
11322 3721 Acked Read 9 Done Done Sent false
11323 3722 Acked Read 5 Done Done Sent false
11324 3723 Acked Write 8 Done Done Sent false
11325 3724 Acked Write 10 Done Done Sent false
11326 3725 Acked Read 3 Done Done Sent false
11327 3726 Acked Read 4 Done Done Sent false
11328 3727 Acked Write 8 Done Done Sent false
11329 3728 Acked Read 1 Done Done Sent false
11330 3729 Acked Write 3 Done Done Sent false
11331 3730 Acked Write 5 Done Done Sent false
11332 3731 Acked Write 6 Done Done Sent false
11333 3732 Acked Read 1 Done Done Sent false
11334 3733 Acked Write 1 Done Done Sent false
11335 3734 Acked Read 6 Done Done Sent false
11336 3735 Acked Write 1 Done Done Sent false
11337 3736 Acked Write 3 Done Done Sent false
11338 3737 Acked Write 8 Done Done Sent false
11339 3738 Acked Write 4 Done Done Sent false
11340 3739 Acked Flush 0 Done Done Sent false
11341 3740 Acked Write 3 Done Done Sent false
11342 3741 Acked Read 9 Done Done Sent false
11343 3742 Acked Read 5 Done Done Sent false
11344 3743 Acked Write 9 Done Done Sent false
11345 3744 Acked Read 1 Done Done Sent false
11346 3745 Acked Read 9 Done Done Sent false
11347 3746 Acked Write 3 Done Done Sent false
11348 3747 Acked Flush 0 Done Done Sent false
11349 3748 Acked Write 6 Done Done Sent false
11350 3749 Acked Read 6 Done Done Sent false
11351 3750 Acked Write 8 Done Done Sent false
11352 3751 Acked Read 4 Done Done Sent false
11353 3752 Acked Write 7 Done Done Sent false
11354 3753 Acked Write 1 Done Done Sent false
11355 3754 Acked Read 5 Done Done Sent false
11356 3755 Acked Read 2 Done Done Sent false
11357 3756 Acked Write 2 Done Done Sent false
11358 3757 Acked Read 1 Done Done Sent false
11359 3758 Acked Write 6 Done Done Sent false
11360 3759 Acked Write 6 Done Done Sent false
11361 3760 Acked Read 2 Done Done Sent false
11362 3761 Acked Read 7 Done Done Sent false
11363 3762 Acked Read 6 Done Done Sent false
11364 3763 Acked Write 10 Done Done Sent false
11365 3764 Acked Read 8 Done Done Sent false
11366 3765 Acked Write 1 Done Done Sent false
11367 3766 Acked Write 1 Done Done Sent false
11368 3767 Acked Flush 0 Done Done Sent false
11369 3768 Acked Read 3 Done Done Sent false
11370 3769 Acked Read 3 Done Done Sent false
11371 3770 Acked Write 6 Done Done Sent false
11372 3771 Acked Read 3 Done Done Sent false
11373 3772 Acked Read 3 Done Done Sent false
11374 3773 Acked Read 10 Done Done Sent false
11375 3774 Acked Read 6 Done Done Sent false
11376 3775 Acked Read 7 Done Done Sent false
11377 3776 Acked Read 8 Done Done Sent false
11378 3777 Acked Read 9 Done Done Sent false
11379 3778 Acked Read 4 Done Done Sent false
11380 3779 Acked Read 4 Done Done Sent false
11381 3780 Acked Flush 0 Done Done Sent false
11382 3781 Acked Write 1 Done Done Sent false
11383 3782 Acked Write 8 Done Done Sent false
11384 3783 Acked Read 1 Done Done Sent false
11385 3784 Acked Write 3 Done Done Sent false
11386 3785 Acked Read 8 Done Done Sent false
11387 3786 Acked Read 1 Done Done Sent false
11388 3787 Acked Flush 0 Done Done Sent false
11389 3788 Acked Read 10 Done Done Sent false
11390 3789 Acked Read 7 Done Done Sent false
11391 3790 Acked Read 7 Done Done Sent false
11392 3791 Acked Flush 0 Done Done Sent false
11393 3792 Acked Write 2 Done Done Sent false
11394 3793 Acked Read 3 Done Done Sent false
11395 3794 Acked Read 2 Done Done Sent false
11396 3795 Acked Read 9 Done Done Sent false
11397 3796 Acked Write 8 Done Done Sent false
11398 3797 Acked Flush 0 Done Done Sent false
11399 3798 Acked Read 9 Done Done Sent false
11400 3799 Acked Write 7 Done Done Sent false
11401 3800 Acked Write 9 Done Done Sent false
11402 3801 Acked Write 10 Done Done Sent false
11403 3802 Acked Write 9 Done Done Sent false
11404 3803 Acked Read 4 Done Done Sent false
11405 3804 Acked Write 5 Done Done Sent false
11406 3805 Acked Read 2 Done Done Sent false
11407 3806 Acked Read 8 Done Done Sent false
11408 3807 Acked Flush 0 Done Done Sent false
11409 3808 Acked Write 7 Done Done Sent false
11410 3809 Acked Write 10 Done Done Sent false
11411 3810 Acked Flush 0 Done Done Sent false
11412 3811 Acked Read 5 Done Done Sent false
11413 3812 Acked Read 4 Done Done Sent false
11414 3813 Acked Write 10 Done Done Sent false
11415 3814 Acked Write 3 Done Done Sent false
11416 3815 Acked Write 3 Done Done Sent false
11417 3816 Acked Read 2 Done Done Sent false
11418 3817 Acked Write 7 Done Done Sent false
11419 3818 Acked Read 3 Done Done Sent false
11420 3819 Acked Read 6 Done Done Sent false
11421 3820 Acked Read 8 Done Done Sent false
11422 3821 Acked Write 2 Done Done Sent false
11423 3822 Acked Write 4 Done Done Sent false
11424 3823 Acked Write 4 Done Done Sent false
11425 3824 Acked Read 2 Done Done Sent false
11426 3825 Acked Read 4 Done Done Sent false
11427 3826 Acked Write 9 Done Done Sent false
11428 3827 Acked Read 1 Done Done Sent false
11429 3828 Acked Read 9 Done Done Sent false
11430 3829 Acked Read 9 Done Done Sent false
11431 3830 Acked Write 10 Done Done Sent false
11432 3831 Acked Read 10 Done Done Sent false
11433 3832 Acked Read 5 Done Done Sent false
11434 3833 Acked Flush 0 Done Done Sent false
11435 3834 Acked Flush 0 Done Done Sent false
11436 3835 Acked Read 3 Done Done Sent false
11437 3836 Acked Write 2 Done Done Sent false
11438 3837 Acked Write 2 Done Done Sent false
11439 3838 Acked Read 4 Done Done Sent false
11440 3839 Acked Flush 0 Done Done Sent false
11441 3840 Acked Flush 0 Done Done Sent false
11442 3841 Acked Write 6 Done Done Sent false
11443 3842 Acked Read 6 Done Done Sent false
11444 3843 Acked Read 5 Done Done Sent false
11445 3844 Acked Read 8 Done Done Sent false
11446 3845 Acked Read 3 Done Done Sent false
11447 3846 Acked Flush 0 Done Done Sent false
11448 3847 Acked Read 6 Done Done Sent false
11449 3848 Acked Read 5 Done Done Sent false
11450 3849 Acked Write 1 Done Done Sent false
11451 3850 Acked Write 7 Done Done Sent false
11452 3851 Acked Read 1 Done Done Sent false
11453 3852 Acked Read 6 Done Done Sent false
11454 3853 Acked Read 8 Done Done Sent false
11455 3854 Acked Read 9 Done Done Sent false
11456 3855 Acked Read 3 Done Done Sent false
11457 3856 Acked Read 3 Done Done Sent false
11458 3857 Acked Read 10 Done Done Sent false
11459 3858 Acked Read 7 Done Done Sent false
11460 3859 Acked Flush 0 Done Done Sent false
11461 3860 Acked Write 7 Done Done Sent false
11462 3861 Acked Write 9 Done Done Sent false
11463 3862 Acked Read 10 Done Done Sent false
11464 3863 Acked Write 1 Done Done Sent false
11465 3864 Acked Read 9 Done Done Sent false
11466 3865 Acked Read 6 Done Done Sent false
11467 3866 Acked Write 2 Done Done Sent false
11468 3867 Acked Read 4 Done Done Sent false
11469 3868 Acked Write 6 Done Done Sent false
11470 3869 Acked Read 9 Done Done Sent false
11471 3870 Acked Read 1 Done Done Sent false
11472 3871 Acked Read 1 Done Done Sent false
11473 3872 Acked Write 5 Done Done Sent false
11474 3873 Acked Write 10 Done Done Sent false
11475 3874 Acked Read 2 Done Done Sent false
11476 3875 Acked Read 6 Done Done Sent false
11477 3876 Acked Read 8 Done Done Sent false
11478 3877 Acked Flush 0 Done Done Sent false
11479 3878 Acked Write 9 Done Done Sent false
11480 3879 Acked Write 6 Done Done Sent false
11481 3880 Acked Flush 0 Done Done Sent false
11482 3881 Acked Write 9 Done Done Sent false
11483 3882 Acked Read 4 Done Done Sent false
11484 3883 Acked Write 10 Done Done Sent false
11485 3884 Acked Write 9 Done Done Sent false
11486 3885 Acked Write 10 Done Done Sent false
11487 3886 Acked Write 3 Done Done Sent false
11488 3887 Acked Read 5 Done Done Sent false
11489 3888 Acked Write 9 Done Done Sent false
11490 3889 Acked Write 7 Done Done Sent false
11491 3890 Acked Read 10 Done Done Sent false
11492 3891 Acked Read 8 Done Done Sent false
11493 3892 Acked Read 5 Done Done Sent false
11494 3893 Acked Read 4 Done Done Sent false
11495 3894 Acked Flush 0 Done Done Sent false
11496 3895 Acked Read 7 Done Done Sent false
11497 3896 Acked Write 3 Done Done Sent false
11498 3897 Acked Read 3 Done Done Sent false
11499 3898 Acked Read 9 Done Done Sent false
11500 3899 Acked Read 5 Done Done Sent false
11501 3900 Acked Write 9 Done Done Sent false
11502 3901 Acked Read 10 Done Done Sent false
11503 3902 Acked Read 10 Done Done Sent false
11504 3903 Acked Write 7 Done Done Sent false
11505 3904 Acked Write 6 Done Done Sent false
11506 3905 Acked Read 5 Done Done Sent false
11507 3906 Acked Read 10 Done Done Sent false
11508 3907 Acked Flush 0 Done Done Sent false
11509 3908 Acked Flush 0 Done Done Sent false
11510 3909 Acked Write 5 Done Done Sent false
11511 3910 Acked Write 3 Done Done Sent false
11512 3911 Acked Write 2 Done Done Sent false
11513 3912 Acked Write 9 Done Done Sent false
11514 3913 Acked Write 3 Done Done Sent false
11515 3914 Acked Write 2 Done Done Sent false
11516 3915 Acked Write 9 Done Done Sent false
11517 3916 Acked Write 7 Done Done Sent false
11518 3917 Acked Read 3 Done Done Sent false
11519 3918 Acked Read 9 Done Done Sent false
11520 3919 Acked Read 1 Done Done Sent false
11521 3920 Acked Write 5 Done Done Sent false
11522 3921 Acked Read 3 Done Done Sent false
11523 3922 Acked Read 7 Done Done Sent false
11524 3923 Acked Read 4 Done Done Sent false
11525 3924 Acked Read 5 Done Done Sent false
11526 3925 Acked Read 7 Done Done Sent false
11527 3926 Acked Flush 0 Done Done Sent false
11528 3927 Acked Write 10 Done Done Sent false
11529 3928 Acked Flush 0 Done Done Sent false
11530 3929 Acked Flush 0 Done Done Sent false
11531 3930 Acked Write 8 Done Done Sent false
11532 3931 Acked Read 4 Done Done Sent false
11533 3932 Acked Read 2 Done Done Sent false
11534 3933 Acked Write 7 Done Done Sent false
11535 3934 Acked Write 1 Done Done Sent false
11536 3935 Acked Flush 0 Done Done Sent false
11537 3936 Acked Write 3 Done Done Sent false
11538 3937 Acked Read 8 Done Done Sent false
11539 3938 Acked Read 2 Done Done Sent false
11540 3939 Acked Read 3 Done Done Sent false
11541 3940 Acked Write 10 Done Done Sent false
11542 3941 Acked Write 7 Done Done Sent false
11543 3942 Acked Write 6 Done Done Sent false
11544 3943 Acked Read 3 Done Done Sent false
11545 3944 Acked Write 7 Done Done Sent false
11546 3945 Acked Write 6 Done Done Sent false
11547 3946 Acked Read 6 Done Done Sent false
11548 3947 Acked Read 9 Done Done Sent false
11549 3948 Acked Flush 0 Done Done Sent false
11550 3949 Acked Read 6 Done Done Sent false
11551 3950 Acked Read 6 Done Done Sent false
11552 3951 Acked Read 9 Done Done Sent false
11553 3952 Acked Read 1 Done Done Sent false
11554 3953 Acked Read 5 Done Done Sent false
11555 3954 Acked Write 6 Done Done Sent false
11556 3955 Acked Write 4 Done Done Sent false
11557 3956 Acked Write 4 Done Done Sent false
11558 3957 Acked Write 8 Done Done Sent false
11559 3958 Acked Read 10 Done Done Sent false
11560 3959 Acked Write 2 Done Done Sent false
11561 3960 Acked Read 8 Done Done Sent false
11562 3961 Acked Read 2 Done Done Sent false
11563 3962 Acked Read 6 Done Done Sent false
11564 3963 Acked Read 3 Done Done Sent false
11565 3964 Acked Read 5 Done Done Sent false
11566 3965 Acked Read 7 Done Done Sent false
11567 3966 Acked Flush 0 Done Done Sent false
11568 3967 Acked Read 2 Done Done Sent false
11569 3968 Acked Read 7 Done Done Sent false
11570 3969 Acked Read 8 Done Done Sent false
11571 3970 Acked Read 7 Done Done Sent false
11572 3971 Acked Flush 0 Done Done Sent false
11573 3972 Acked Read 2 Done Done Sent false
11574 3973 Acked Read 8 Done Done Sent false
11575 3974 Acked Read 2 Done Done Sent false
11576 3975 Acked Read 6 Done Done Sent false
11577 3976 Acked Read 5 Done Done Sent false
11578 3977 Acked Write 1 Done Done Sent false
11579 3978 Acked Flush 0 Done Done Sent false
11580 3979 Acked Write 7 Done Done Sent false
11581 3980 Acked Write 6 Done Done Sent false
11582 3981 Acked Read 9 Done Done Sent false
11583 3982 Acked Read 6 Done Done Sent false
11584 3983 Acked Read 6 Done Done Sent false
11585 3984 Acked Read 8 Done Done Sent false
11586 3985 Acked Read 4 Done Done Sent false
11587 3986 Acked Read 6 Done Done Sent false
11588 3987 Acked Read 4 Done Done Sent false
11589 3988 Acked Read 5 Done Done Sent false
11590 3989 Acked Flush 0 Done Done Sent false
11591 3990 Acked Read 9 Done Done Sent false
11592 3991 Acked Flush 0 Done Done Sent false
11593 3992 Acked Read 5 Done Done Sent false
11594 3993 Acked Write 5 Done Done Sent false
11595 3994 Acked Read 2 Done Done Sent false
11596 3995 Acked Write 10 Done Done Sent false
11597 3996 Acked Write 3 Done Done Sent false
11598 3997 Acked Read 2 Done Done Sent false
11599 3998 Acked Read 5 Done Done Sent false
11600 3999 Acked Read 8 Done Done Sent false
11601 4000 Acked Flush 0 Done Done Sent false
11602 4001 Acked Read 9 Done Done Sent false
11603 4002 Acked Read 5 Done Done Sent false
11604 4003 Acked Read 7 Done Done Sent false
11605 4004 Acked Flush 0 Done Done Sent false
11606 4005 Acked Read 5 Done Done Sent false
11607 4006 Acked Read 8 Done Done Sent false
11608 4007 Acked Write 6 Done Done Sent false
11609 4008 Acked Read 10 Done Done Sent false
11610 4009 Acked Write 2 Done Done Sent false
11611 4010 Acked Read 6 Done Done Sent false
11612 4011 Acked Read 8 Done Done Sent false
11613 4012 Acked Write 6 Done Done Sent false
11614 4013 Acked Read 9 Done Done Sent false
11615 4014 Acked Write 7 Done Done Sent false
11616 4015 Acked Read 2 Done Done Sent false
11617 4016 Acked Read 9 Done Done Sent false
11618 4017 Acked Write 5 Done Done Sent false
11619 STATES DS:0 DS:1 DS:2 TOTAL
11620 Sent 0 0 301 301
11621 Done 301 301 0 602
11622 Skipped 0 0 0 0
11623 Error 0 0 0 0
11624 Last Flush: Some(JobId(4004)) Some(JobId(4004)) Some(JobId(3716))
11625 Downstairs last five completed: 3716 3715 3714 3713 3712
11626 Upstairs last five completed: 4017 4016 4015 4014 4013
11627 CLIENT: Up:0 ds:301 act:2
116282025-03-17T22:18:50.229ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
116292025-03-17T22:18:50.836ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
11630 [8830] got start action
11631 [8830] I am going to restart
11632 Starting downstairs at port 8830
11633 Make output file at "/tmp/dsc/downstairs-8830.txt"
11634 [8830][2] reports Starting
11635 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1216)
11636 [8830][2] reports Running
116372025-03-17T22:18:51.839ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
116382025-03-17T22:18:52.841ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
11639 ----------------------------------------------------------------
11640 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11641 States: Active Active Offline
11642 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
11643 3717 Acked Read 3 Done Done Sent false
11644 3718 Acked Write 1 Done Done Sent false
11645 3719 Acked Read 4 Done Done Sent false
11646 3720 Acked Write 7 Done Done Sent false
11647 3721 Acked Read 9 Done Done Sent false
11648 3722 Acked Read 5 Done Done Sent false
11649 3723 Acked Write 8 Done Done Sent false
11650 3724 Acked Write 10 Done Done Sent false
11651 3725 Acked Read 3 Done Done Sent false
11652 3726 Acked Read 4 Done Done Sent false
11653 3727 Acked Write 8 Done Done Sent false
11654 3728 Acked Read 1 Done Done Sent false
11655 3729 Acked Write 3 Done Done Sent false
11656 3730 Acked Write 5 Done Done Sent false
11657 3731 Acked Write 6 Done Done Sent false
11658 3732 Acked Read 1 Done Done Sent false
11659 3733 Acked Write 1 Done Done Sent false
11660 3734 Acked Read 6 Done Done Sent false
11661 3735 Acked Write 1 Done Done Sent false
11662 3736 Acked Write 3 Done Done Sent false
11663 3737 Acked Write 8 Done Done Sent false
11664 3738 Acked Write 4 Done Done Sent false
11665 3739 Acked Flush 0 Done Done Sent false
11666 3740 Acked Write 3 Done Done Sent false
11667 3741 Acked Read 9 Done Done Sent false
11668 3742 Acked Read 5 Done Done Sent false
11669 3743 Acked Write 9 Done Done Sent false
11670 3744 Acked Read 1 Done Done Sent false
11671 3745 Acked Read 9 Done Done Sent false
11672 3746 Acked Write 3 Done Done Sent false
11673 3747 Acked Flush 0 Done Done Sent false
11674 3748 Acked Write 6 Done Done Sent false
11675 3749 Acked Read 6 Done Done Sent false
11676 3750 Acked Write 8 Done Done Sent false
11677 3751 Acked Read 4 Done Done Sent false
11678 3752 Acked Write 7 Done Done Sent false
11679 3753 Acked Write 1 Done Done Sent false
11680 3754 Acked Read 5 Done Done Sent false
11681 3755 Acked Read 2 Done Done Sent false
11682 3756 Acked Write 2 Done Done Sent false
11683 3757 Acked Read 1 Done Done Sent false
11684 3758 Acked Write 6 Done Done Sent false
11685 3759 Acked Write 6 Done Done Sent false
11686 3760 Acked Read 2 Done Done Sent false
11687 3761 Acked Read 7 Done Done Sent false
11688 3762 Acked Read 6 Done Done Sent false
11689 3763 Acked Write 10 Done Done Sent false
11690 3764 Acked Read 8 Done Done Sent false
11691 3765 Acked Write 1 Done Done Sent false
11692 3766 Acked Write 1 Done Done Sent false
11693 3767 Acked Flush 0 Done Done Sent false
11694 3768 Acked Read 3 Done Done Sent false
11695 3769 Acked Read 3 Done Done Sent false
11696 3770 Acked Write 6 Done Done Sent false
11697 3771 Acked Read 3 Done Done Sent false
11698 3772 Acked Read 3 Done Done Sent false
11699 3773 Acked Read 10 Done Done Sent false
11700 3774 Acked Read 6 Done Done Sent false
11701 3775 Acked Read 7 Done Done Sent false
11702 3776 Acked Read 8 Done Done Sent false
11703 3777 Acked Read 9 Done Done Sent false
11704 3778 Acked Read 4 Done Done Sent false
11705 3779 Acked Read 4 Done Done Sent false
11706 3780 Acked Flush 0 Done Done Sent false
11707 3781 Acked Write 1 Done Done Sent false
11708 3782 Acked Write 8 Done Done Sent false
11709 3783 Acked Read 1 Done Done Sent false
11710 3784 Acked Write 3 Done Done Sent false
11711 3785 Acked Read 8 Done Done Sent false
11712 3786 Acked Read 1 Done Done Sent false
11713 3787 Acked Flush 0 Done Done Sent false
11714 3788 Acked Read 10 Done Done Sent false
11715 3789 Acked Read 7 Done Done Sent false
11716 3790 Acked Read 7 Done Done Sent false
11717 3791 Acked Flush 0 Done Done Sent false
11718 3792 Acked Write 2 Done Done Sent false
11719 3793 Acked Read 3 Done Done Sent false
11720 3794 Acked Read 2 Done Done Sent false
11721 3795 Acked Read 9 Done Done Sent false
11722 3796 Acked Write 8 Done Done Sent false
11723 3797 Acked Flush 0 Done Done Sent false
11724 3798 Acked Read 9 Done Done Sent false
11725 3799 Acked Write 7 Done Done Sent false
11726 3800 Acked Write 9 Done Done Sent false
11727 3801 Acked Write 10 Done Done Sent false
11728 3802 Acked Write 9 Done Done Sent false
11729 3803 Acked Read 4 Done Done Sent false
11730 3804 Acked Write 5 Done Done Sent false
11731 3805 Acked Read 2 Done Done Sent false
11732 3806 Acked Read 8 Done Done Sent false
11733 3807 Acked Flush 0 Done Done Sent false
11734 3808 Acked Write 7 Done Done Sent false
11735 3809 Acked Write 10 Done Done Sent false
11736 3810 Acked Flush 0 Done Done Sent false
11737 3811 Acked Read 5 Done Done Sent false
11738 3812 Acked Read 4 Done Done Sent false
11739 3813 Acked Write 10 Done Done Sent false
11740 3814 Acked Write 3 Done Done Sent false
11741 3815 Acked Write 3 Done Done Sent false
11742 3816 Acked Read 2 Done Done Sent false
11743 3817 Acked Write 7 Done Done Sent false
11744 3818 Acked Read 3 Done Done Sent false
11745 3819 Acked Read 6 Done Done Sent false
11746 3820 Acked Read 8 Done Done Sent false
11747 3821 Acked Write 2 Done Done Sent false
11748 3822 Acked Write 4 Done Done Sent false
11749 3823 Acked Write 4 Done Done Sent false
11750 3824 Acked Read 2 Done Done Sent false
11751 3825 Acked Read 4 Done Done Sent false
11752 3826 Acked Write 9 Done Done Sent false
11753 3827 Acked Read 1 Done Done Sent false
11754 3828 Acked Read 9 Done Done Sent false
11755 3829 Acked Read 9 Done Done Sent false
11756 3830 Acked Write 10 Done Done Sent false
11757 3831 Acked Read 10 Done Done Sent false
11758 3832 Acked Read 5 Done Done Sent false
11759 3833 Acked Flush 0 Done Done Sent false
11760 3834 Acked Flush 0 Done Done Sent false
11761 3835 Acked Read 3 Done Done Sent false
11762 3836 Acked Write 2 Done Done Sent false
11763 3837 Acked Write 2 Done Done Sent false
11764 3838 Acked Read 4 Done Done Sent false
11765 3839 Acked Flush 0 Done Done Sent false
11766 3840 Acked Flush 0 Done Done Sent false
11767 3841 Acked Write 6 Done Done Sent false
11768 3842 Acked Read 6 Done Done Sent false
11769 3843 Acked Read 5 Done Done Sent false
11770 3844 Acked Read 8 Done Done Sent false
11771 3845 Acked Read 3 Done Done Sent false
11772 3846 Acked Flush 0 Done Done Sent false
11773 3847 Acked Read 6 Done Done Sent false
11774 3848 Acked Read 5 Done Done Sent false
11775 3849 Acked Write 1 Done Done Sent false
11776 3850 Acked Write 7 Done Done Sent false
11777 3851 Acked Read 1 Done Done Sent false
11778 3852 Acked Read 6 Done Done Sent false
11779 3853 Acked Read 8 Done Done Sent false
11780 3854 Acked Read 9 Done Done Sent false
11781 3855 Acked Read 3 Done Done Sent false
11782 3856 Acked Read 3 Done Done Sent false
11783 3857 Acked Read 10 Done Done Sent false
11784 3858 Acked Read 7 Done Done Sent false
11785 3859 Acked Flush 0 Done Done Sent false
11786 3860 Acked Write 7 Done Done Sent false
11787 3861 Acked Write 9 Done Done Sent false
11788 3862 Acked Read 10 Done Done Sent false
11789 3863 Acked Write 1 Done Done Sent false
11790 3864 Acked Read 9 Done Done Sent false
11791 3865 Acked Read 6 Done Done Sent false
11792 3866 Acked Write 2 Done Done Sent false
11793 3867 Acked Read 4 Done Done Sent false
11794 3868 Acked Write 6 Done Done Sent false
11795 3869 Acked Read 9 Done Done Sent false
11796 3870 Acked Read 1 Done Done Sent false
11797 3871 Acked Read 1 Done Done Sent false
11798 3872 Acked Write 5 Done Done Sent false
11799 3873 Acked Write 10 Done Done Sent false
11800 3874 Acked Read 2 Done Done Sent false
11801 3875 Acked Read 6 Done Done Sent false
11802 3876 Acked Read 8 Done Done Sent false
11803 3877 Acked Flush 0 Done Done Sent false
11804 3878 Acked Write 9 Done Done Sent false
11805 3879 Acked Write 6 Done Done Sent false
11806 3880 Acked Flush 0 Done Done Sent false
11807 3881 Acked Write 9 Done Done Sent false
11808 3882 Acked Read 4 Done Done Sent false
11809 3883 Acked Write 10 Done Done Sent false
11810 3884 Acked Write 9 Done Done Sent false
11811 3885 Acked Write 10 Done Done Sent false
11812 3886 Acked Write 3 Done Done Sent false
11813 3887 Acked Read 5 Done Done Sent false
11814 3888 Acked Write 9 Done Done Sent false
11815 3889 Acked Write 7 Done Done Sent false
11816 3890 Acked Read 10 Done Done Sent false
11817 3891 Acked Read 8 Done Done Sent false
11818 3892 Acked Read 5 Done Done Sent false
11819 3893 Acked Read 4 Done Done Sent false
11820 3894 Acked Flush 0 Done Done Sent false
11821 3895 Acked Read 7 Done Done Sent false
11822 3896 Acked Write 3 Done Done Sent false
11823 3897 Acked Read 3 Done Done Sent false
11824 3898 Acked Read 9 Done Done Sent false
11825 3899 Acked Read 5 Done Done Sent false
11826 3900 Acked Write 9 Done Done Sent false
11827 3901 Acked Read 10 Done Done Sent false
11828 3902 Acked Read 10 Done Done Sent false
11829 3903 Acked Write 7 Done Done Sent false
11830 3904 Acked Write 6 Done Done Sent false
11831 3905 Acked Read 5 Done Done Sent false
11832 3906 Acked Read 10 Done Done Sent false
11833 3907 Acked Flush 0 Done Done Sent false
11834 3908 Acked Flush 0 Done Done Sent false
11835 3909 Acked Write 5 Done Done Sent false
11836 3910 Acked Write 3 Done Done Sent false
11837 3911 Acked Write 2 Done Done Sent false
11838 3912 Acked Write 9 Done Done Sent false
11839 3913 Acked Write 3 Done Done Sent false
11840 3914 Acked Write 2 Done Done Sent false
11841 3915 Acked Write 9 Done Done Sent false
11842 3916 Acked Write 7 Done Done Sent false
11843 3917 Acked Read 3 Done Done Sent false
11844 3918 Acked Read 9 Done Done Sent false
11845 3919 Acked Read 1 Done Done Sent false
11846 3920 Acked Write 5 Done Done Sent false
11847 3921 Acked Read 3 Done Done Sent false
11848 3922 Acked Read 7 Done Done Sent false
11849 3923 Acked Read 4 Done Done Sent false
11850 3924 Acked Read 5 Done Done Sent false
11851 3925 Acked Read 7 Done Done Sent false
11852 3926 Acked Flush 0 Done Done Sent false
11853 3927 Acked Write 10 Done Done Sent false
11854 3928 Acked Flush 0 Done Done Sent false
11855 3929 Acked Flush 0 Done Done Sent false
11856 3930 Acked Write 8 Done Done Sent false
11857 3931 Acked Read 4 Done Done Sent false
11858 3932 Acked Read 2 Done Done Sent false
11859 3933 Acked Write 7 Done Done Sent false
11860 3934 Acked Write 1 Done Done Sent false
11861 3935 Acked Flush 0 Done Done Sent false
11862 3936 Acked Write 3 Done Done Sent false
11863 3937 Acked Read 8 Done Done Sent false
11864 3938 Acked Read 2 Done Done Sent false
11865 3939 Acked Read 3 Done Done Sent false
11866 3940 Acked Write 10 Done Done Sent false
11867 3941 Acked Write 7 Done Done Sent false
11868 3942 Acked Write 6 Done Done Sent false
11869 3943 Acked Read 3 Done Done Sent false
11870 3944 Acked Write 7 Done Done Sent false
11871 3945 Acked Write 6 Done Done Sent false
11872 3946 Acked Read 6 Done Done Sent false
11873 3947 Acked Read 9 Done Done Sent false
11874 3948 Acked Flush 0 Done Done Sent false
11875 3949 Acked Read 6 Done Done Sent false
11876 3950 Acked Read 6 Done Done Sent false
11877 3951 Acked Read 9 Done Done Sent false
11878 3952 Acked Read 1 Done Done Sent false
11879 3953 Acked Read 5 Done Done Sent false
11880 3954 Acked Write 6 Done Done Sent false
11881 3955 Acked Write 4 Done Done Sent false
11882 3956 Acked Write 4 Done Done Sent false
11883 3957 Acked Write 8 Done Done Sent false
11884 3958 Acked Read 10 Done Done Sent false
11885 3959 Acked Write 2 Done Done Sent false
11886 3960 Acked Read 8 Done Done Sent false
11887 3961 Acked Read 2 Done Done Sent false
11888 3962 Acked Read 6 Done Done Sent false
11889 3963 Acked Read 3 Done Done Sent false
11890 3964 Acked Read 5 Done Done Sent false
11891 3965 Acked Read 7 Done Done Sent false
11892 3966 Acked Flush 0 Done Done Sent false
11893 3967 Acked Read 2 Done Done Sent false
11894 3968 Acked Read 7 Done Done Sent false
11895 3969 Acked Read 8 Done Done Sent false
11896 3970 Acked Read 7 Done Done Sent false
11897 3971 Acked Flush 0 Done Done Sent false
11898 3972 Acked Read 2 Done Done Sent false
11899 3973 Acked Read 8 Done Done Sent false
11900 3974 Acked Read 2 Done Done Sent false
11901 3975 Acked Read 6 Done Done Sent false
11902 3976 Acked Read 5 Done Done Sent false
11903 3977 Acked Write 1 Done Done Sent false
11904 3978 Acked Flush 0 Done Done Sent false
11905 3979 Acked Write 7 Done Done Sent false
11906 3980 Acked Write 6 Done Done Sent false
11907 3981 Acked Read 9 Done Done Sent false
11908 3982 Acked Read 6 Done Done Sent false
11909 3983 Acked Read 6 Done Done Sent false
11910 3984 Acked Read 8 Done Done Sent false
11911 3985 Acked Read 4 Done Done Sent false
11912 3986 Acked Read 6 Done Done Sent false
11913 3987 Acked Read 4 Done Done Sent false
11914 3988 Acked Read 5 Done Done Sent false
11915 3989 Acked Flush 0 Done Done Sent false
11916 3990 Acked Read 9 Done Done Sent false
11917 3991 Acked Flush 0 Done Done Sent false
11918 3992 Acked Read 5 Done Done Sent false
11919 3993 Acked Write 5 Done Done Sent false
11920 3994 Acked Read 2 Done Done Sent false
11921 3995 Acked Write 10 Done Done Sent false
11922 3996 Acked Write 3 Done Done Sent false
11923 3997 Acked Read 2 Done Done Sent false
11924 3998 Acked Read 5 Done Done Sent false
11925 3999 Acked Read 8 Done Done Sent false
11926 4000 Acked Flush 0 Done Done Sent false
11927 4001 Acked Read 9 Done Done Sent false
11928 4002 Acked Read 5 Done Done Sent false
11929 4003 Acked Read 7 Done Done Sent false
11930 4004 Acked Flush 0 Done Done Sent false
11931 4005 Acked Read 5 Done Done Sent false
11932 4006 Acked Read 8 Done Done Sent false
11933 4007 Acked Write 6 Done Done Sent false
11934 4008 Acked Read 10 Done Done Sent false
11935 4009 Acked Write 2 Done Done Sent false
11936 4010 Acked Read 6 Done Done Sent false
11937 4011 Acked Read 8 Done Done Sent false
11938 4012 Acked Write 6 Done Done Sent false
11939 4013 Acked Read 9 Done Done Sent false
11940 4014 Acked Write 7 Done Done Sent false
11941 4015 Acked Read 2 Done Done Sent false
11942 4016 Acked Read 9 Done Done Sent false
11943 4017 Acked Write 5 Done Done Sent false
11944 STATES DS:0 DS:1 DS:2 TOTAL
11945 Sent 0 0 301 301
11946 Done 301 301 0 602
11947 Skipped 0 0 0 0
11948 Error 0 0 0 0
11949 Last Flush: Some(JobId(4004)) Some(JobId(4004)) Some(JobId(3716))
11950 Downstairs last five completed: 3716 3715 3714 3713 3712
11951 Upstairs last five completed: 4017 4016 4015 4014 4013
11952 CLIENT: Up:0 ds:301 act:2
119532025-03-17T22:18:54.232ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
119542025-03-17T22:18:55.206ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
119552025-03-17T22:18:55.468ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
119562025-03-17T22:18:55.469ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:57055) = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
119572025-03-17T22:18:55.470ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
119582025-03-17T22:18:55.470ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
119592025-03-17T22:18:55.470ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
119602025-03-17T22:18:55.470ZINFOcrucible: send last flush ID to this DS: Some(JobId(3716)) = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
119612025-03-17T22:18:55.470ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
119622025-03-17T22:18:55.470ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(3716)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
119632025-03-17T22:18:55.471ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(3716)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
11964 ----------------------------------------------------------------
11965 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
11966 Downstairs last five completed: 4018 4017 4016 4015 4014
11967 Upstairs last five completed: 4018 4017 4016 4015 4014
11968 CLIENT: Up:0 ds:0 act:3
11969 Replay: All jobs finished
11970 Main task has work to do, go find it
11971 got dsc Stop(1)
11972 stop 1
11973 Mar 17 22:18:57.733 INFO[8820] Got stop action so:false kr:false
11974 request completed, latency_us: 373, response_code: 204, uri: /stop/cid/1, method: POST, req_id: c39d6a26-919c-4e15-928f-ea778344947f, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
119752025-03-17T22:18:57.734ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
119762025-03-17T22:18:57.734ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
119772025-03-17T22:18:57.734ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
11978 Mar 17 22:18:57.734 INFO request completed, latency_us: 162, response_code: 200, uri: /state/cid/1, method: GET, {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:18:57.734811288Z","hostname":"w-01JPK2S810SFAA9787XEG8GFX5","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"0f8310d7-099e-4ead-8484-91ddc162d307"}
11979 req_id: 87af24e1-1efd-495b-a1ab-22b81b4acaa2, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
11980 [8820] Exited with: ExitStatus(unix_wait_status(9))
11981 [8820][1] reports Exit
11982 Mar 17 22:19:01.737 INFO request completed, latency_us: 214, response_code: 200, uri: /state/cid/1, method: GET, req_id: 67ed5a0c-14b6-42a3-b40d-0e7b5797c443, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
11983 001/300 Read block 2733 len 8192
11984 002/300 Write block 130 len 16384 data: 4 4 3 2
11985 003/300 Write block 703 len 32768 data: 5 4 4 4 4 4 3 3
11986 004/300 Read block 1095 len 4096
11987 005/300 Write block 2959 len 40960 data: 7 7 6 5 4 4 4 4 5 5
11988 006/300 Read block 2101 len 4096
11989 007/300 Write block 4430 len 20480 data: 3 2 2 3 3
11990 008/300 Flush
11991 009/300 Write block 1409 len 36864 data: 2 4 4 4 5 4 4 4 4
11992 010/300 Write block 386 len 12288 data: 2 2 2
11993 011/300 Write block 4106 len 16384 data: 4 5 4 3
11994 012/300 Read block 2863 len 4096
11995 013/300 Read block 4867 len 20480
11996 014/300 Flush
11997 015/300 Write block 1986 len 20480 data: 4 3 3 5 5
11998 016/300 Read block 919 len 24576
11999 017/300 Read block 3396 len 36864
12000 018/300 Write block 2117 len 36864 data: 6 7 7 6 6 4 4 3 2
12001 019/300 Write block 385 len 20480 data: 2 3 3 3 2
12002 020/300 Write block 20 len 40960 data: 4 4 4 3 3 3 4 4 5 3
12003 021/300 Read block 501 len 12288
12004 022/300 Write block 2934 len 8192 data: 7 5
12005 023/300 Write block 724 len 20480 data: 2 2 2 2 2
12006 024/300 Read block 4258 len 32768
12007 025/300 Write block 2439 len 36864 data: 6 7 6 5 4 3 2 2 2
12008 026/300 Write block 4746 len 28672 data: 4 4 4 4 3 4 4
12009 027/300 Write block 2159 len 28672 data: 2 2 2 2 2 2 3
12010 028/300 Write block 1724 len 12288 data: 2 2 2
12011 029/300 Read block 2133 len 32768
12012 030/300 Flush
12013 031/300 Flush
12014 032/300 Write block 4239 len 4096 data: 3
12015 033/300 Read block 2433 len 16384
12016 034/300 Read block 3578 len 8192
12017 035/300 Read block 2015 len 16384
12018 036/300 Read block 2753 len 28672
12019 037/300 Read block 2971 len 24576
12020 038/300 Flush
12021 039/300 Read block 1895 len 28672
12022 040/300 Read block 2512 len 24576
12023 041/300 Read block 728 len 36864
12024 042/300 Write block 4946 len 20480 data: 2 2 2 3 4
12025 043/300 Read block 993 len 40960
12026 044/300 Read block 1881 len 4096
12027 045/300 Read block 313 len 28672
12028 046/300 Write block 631 len 20480 data: 5 6 5 5 3
12029 047/300 Read block 2631 len 12288
12030 048/300 Read block 4405 len 36864
12031 049/300 Write block 2139 len 20480 data: 4 3 2 3 3
12032 050/300 Read block 497 len 8192
12033 051/300 Flush
12034 052/300 Read block 4652 len 28672
12035 053/300 Read block 1478 len 28672
12036 054/300 Read block 172 len 12288
12037 055/300 Read block 1925 len 12288
12038 056/300 Read block 1707 len 24576
12039 057/300 Write block 1421 len 4096 data: 3
12040 058/300 Flush
12041 059/300 Read block 4834 len 8192
12042 060/300 Write block 4870 len 16384 data: 3 3 3 5
12043 061/300 Write block 1358 len 20480 data: 6 6 6 5 5
12044 062/300 Read block 561 len 28672
12045 063/300 Flush
12046 064/300 Write block 2046 len 40960 data: 3 3 4 4 4 4 4 4 5 4
12047 065/300 Read block 1759 len 28672
12048 066/300 Read block 4446 len 24576
12049 067/300 Read block 3914 len 4096
12050 068/300 Write block 398 len 4096 data: 3
12051 069/300 Write block 1882 len 36864 data: 3 2 2 2 2 2 2 2 2
12052 070/300 Flush
12053 071/300 Write block 3240 len 40960 data: 5 4 4 3 3 2 2 3 3 3
12054 072/300 Read block 646 len 8192
12055 073/300 Read block 1402 len 8192
12056 074/300 Write block 1886 len 12288 data: 3 3 3
12057 075/300 Write block 4615 len 4096 data: 4
12058 076/300 Write block 460 len 28672 data: 3 3 3 2 3 3 3
12059 077/300 Read block 66 len 36864
12060 078/300 Write block 1827 len 32768 data: 3 3 3 3 4 4 4 4
12061 079/300 Write block 1570 len 32768 data: 5 4 4 4 3 3 3 3
12062 080/300 Write block 2863 len 8192 data: 3 3
12063 081/300 Read block 1094 len 24576
12064 082/300 Read block 1655 len 16384
12065 083/300 Write block 4514 len 40960 data: 3 3 4 3 3 3 2 3 3 3
12066 084/300 Read block 147 len 40960
12067 085/300 Read block 2778 len 20480
12068 086/300 Write block 4352 len 4096 data: 2
12069 087/300 Write block 431 len 16384 data: 2 2 2 2
12070 088/300 Read block 70 len 20480
12071 089/300 Write block 4984 len 12288 data: 3 3 2
12072 090/300 Write block 756 len 16384 data: 2 3 2 2
12073 091/300 Write block 1527 len 28672 data: 6 5 5 5 5 5 6
12074 092/300 Read block 408 len 12288
12075 093/300 Read block 2228 len 20480
12076 094/300 Read block 2507 len 24576
12077 095/300 Flush
12078 096/300 Read block 1952 len 40960
12079 097/300 Read block 4785 len 8192
12080 098/300 Write block 1128 len 4096 data: 4
12081 099/300 Write block 2805 len 4096 data: 5
12082 100/300 Read block 944 len 20480
12083 101/300 Read block 4587 len 36864
12084 102/300 Flush
12085 103/300 Read block 290 len 24576
12086 104/300 Write block 1188 len 8192 data: 4 3
12087 105/300 Read block 4954 len 24576
12088 106/300 Flush
12089 107/300 Read block 1753 len 24576
12090 108/300 Write block 1134 len 28672 data: 3 3 3 2 2 2 2
12091 109/300 Read block 1024 len 12288
12092 110/300 Write block 1880 len 24576 data: 2 2 4 3 3 3
12093 111/300 Write block 1283 len 4096 data: 4
12094 112/300 Read block 3295 len 16384
12095 113/300 Write block 2486 len 28672 data: 3 3 3 3 3 3 2
12096 114/300 Read block 3475 len 20480
12097 115/300 Read block 1718 len 8192
12098 116/300 Read block 168 len 4096
12099 117/300 Read block 3202 len 32768
12100 118/300 Read block 3657 len 24576
12101 119/300 Read block 3973 len 4096
12102 120/300 Flush
12103 121/300 Write block 1112 len 40960 data: 2 3 3 3 3 3 3 3 2 2
12104 122/300 Write block 3894 len 12288 data: 3 4 5
12105 123/300 Write block 4985 len 8192 data: 4 3
12106 124/300 Write block 765 len 40960 data: 2 2 3 3 3 3 3 3 3 3
12107 125/300 Write block 2594 len 16384 data: 4 3 4 4
12108 126/300 Write block 3280 len 8192 data: 3 3
12109 127/300 Write block 4878 len 12288 data: 4 4 3
12110 128/300 Flush
12111 129/300 Flush
12112 130/300 Read block 278 len 28672
12113 131/300 Read block 2512 len 32768
12114 132/300 Write block 3221 len 28672 data: 3 3 3 4 3 3 3
12115 133/300 Read block 2727 len 28672
12116 134/300 Read block 1412 len 8192
12117 135/300 Write block 3725 len 8192 data: 3 3
12118 136/300 Read block 4476 len 8192
12119 137/300 Read block 386 len 24576
12120 138/300 Read block 4661 len 4096
12121 139/300 Read block 4350 len 32768
12122 140/300 Read block 2192 len 8192
12123 141/300 Flush
12124 142/300 Write block 3213 len 36864 data: 2 2 3 3 3 3 3 3 4
12125 143/300 Read block 3810 len 36864
12126 144/300 Write block 1881 len 8192 data: 3 5
12127 145/300 Read block 4781 len 4096
12128 146/300 Read block 2701 len 40960
12129 147/300 Write block 3826 len 36864 data: 4 4 5 5 4 3 3 3 3
12130 148/300 Read block 2343 len 28672
12131 149/300 Read block 755 len 40960
12132 150/300 Write block 4397 len 20480 data: 3 4 4 4 2
12133 151/300 Write block 3745 len 20480 data: 2 2 3 2 3
12134 152/300 Read block 3130 len 12288
12135 153/300 Write block 1032 len 12288 data: 4 5 6
12136 154/300 Write block 2909 len 28672 data: 4 3 3 3 3 3 3
12137 155/300 Write block 2846 len 36864 data: 6 5 4 3 4 3 3 3 3
12138 156/300 Write block 4360 len 32768 data: 2 2 2 2 2 2 3 4
12139 157/300 Read block 2913 len 8192
12140 158/300 Read block 3521 len 36864
12141 159/300 Flush
12142 160/300 Write block 1361 len 28672 data: 6 6 5 4 4 4 3
12143 161/300 Read block 2572 len 32768
12144 162/300 Write block 3642 len 4096 data: 5
12145 163/300 Write block 7 len 36864 data: 3 3 2 3 3 4 3 3 3
12146 164/300 Write block 4883 len 20480 data: 5 6 5 6 4
12147 165/300 Read block 3665 len 24576
12148 166/300 Write block 3331 len 32768 data: 4 3 3 3 3 3 2 3
12149 167/300 Write block 104 len 8192 data: 2 2
12150 168/300 Read block 189 len 16384
12151 169/300 Read block 2517 len 24576
12152 170/300 Write block 3133 len 16384 data: 5 5 5 4
12153 171/300 Read block 3705 len 12288
12154 172/300 Read block 1928 len 20480
12155 173/300 Read block 2514 len 20480
12156 174/300 Flush
12157 175/300 Read block 2328 len 4096
12158 176/300 Read block 4089 len 36864
12159 177/300 Read block 3364 len 8192
12160 178/300 Write block 4526 len 32768 data: 2 2 2 2 3 3 3 3
12161 179/300 Read block 4310 len 20480
12162 180/300 Flush
12163 181/300 Write block 1519 len 32768 data: 2 2 2 3 3 4 5 5
12164 182/300 Read block 1806 len 16384
12165 183/300 Read block 2721 len 20480
12166 184/300 Read block 946 len 4096
12167 185/300 Write block 3984 len 4096 data: 2
12168 186/300 Write block 1851 len 32768 data: 4 4 4 3 3 2 3 4
12169 187/300 Read block 56 len 12288
12170 188/300 Read block 491 len 32768
12171 189/300 Flush
12172 190/300 Read block 3736 len 16384
12173 191/300 Read block 416 len 8192
12174 192/300 Read block 3764 len 28672
12175 193/300 Flush
12176 194/300 Write block 3318 len 12288 data: 2 3 3
12177 195/300 Write block 1522 len 40960 data: 4 4 5 6 6 7 6 6 6 6
12178 196/300 Read block 189 len 32768
12179 197/300 Read block 4032 len 32768
12180 198/300 Write block 3729 len 24576 data: 4 6 5 4 4 4
12181 199/300 Read block 915 len 20480
12182 200/300 Write block 398 len 12288 data: 4 3 3
12183 201/300 Read block 3577 len 8192
12184 202/300 Write block 2020 len 28672 data: 3 3 3 3 3 3 2
12185 203/300 Read block 1137 len 24576
12186 204/300 Read block 4474 len 12288
12187 205/300 Write block 337 len 4096 data: 6
12188 206/300 Flush
12189 207/300 Write block 465 len 40960 data: 4 4 3 3 3 2 2 2 2 2
12190 208/300 Write block 1379 len 16384 data: 2 2 2 2
12191 209/300 Read block 3600 len 12288
12192 210/300 Read block 1787 len 32768
12193 211/300 Write block 258 len 40960 data: 4 4 4 4 4 2 3 3 3 2
12194 212/300 Read block 1767 len 16384
12195 213/300 Flush
12196 214/300 Read block 1837 len 12288
12197 215/300 Write block 2528 len 32768 data: 2 2 3 3 3 3 3 2
12198 216/300 Write block 1354 len 28672 data: 3 3 5 6 7 7 7
12199 217/300 Read block 1943 len 8192
12200 218/300 Write block 1020 len 40960 data: 3 3 2 2 2 2 2 3 3 4
12201 219/300 Read block 4628 len 12288
12202 220/300 Read block 1933 len 16384
12203 221/300 Read block 2410 len 20480
12204 222/300 Read block 3985 len 28672
12205 223/300 Read block 2561 len 16384
12206 224/300 Write block 3142 len 36864 data: 3 4 4 4 3 4 4 4 4
12207 225/300 Write block 4472 len 4096 data: 7
12208 226/300 Write block 2928 len 20480 data: 5 5 8 7 6
12209 227/300 Read block 680 len 16384
12210 228/300 Write block 605 len 4096 data: 5
12211 229/300 Write block 4710 len 16384 data: 4 4 3 2
12212 230/300 Write block 798 len 40960 data: 4 3 4 4 3 3 3 3 3 3
12213 231/300 Write block 2755 len 36864 data: 2 3 4 4 4 4 3 3 2
12214 232/300 Read block 3399 len 20480
12215 233/300 Write block 328 len 40960 data: 2 2 4 4 4 6 6 7 7 7
12216 234/300 Flush
12217 235/300 Read block 4931 len 24576
12218 236/300 Write block 3631 len 16384 data: 2 2 2 2
12219 237/300 Read block 826 len 28672
12220 238/300 Read block 4750 len 4096
12221 239/300 Write block 3491 len 32768 data: 4 5 3 4 5 4 4 3
12222 240/300 Write block 4313 len 28672 data: 2 2 2 2 2 2 2
12223 241/300 Read block 3523 len 8192
12224 242/300 Write block 51 len 40960 data: 4 3 2 2 2 3 3 3 4 5
12225 243/300 Read block 2022 len 12288
12226 244/300 Write block 3802 len 24576 data: 6 5 4 4 5 4
12227 245/300 Flush
12228 246/300 Write block 1991 len 28672 data: 5 6 3 3 3 5 4
12229 247/300 Read block 654 len 40960
12230 248/300 Read block 3456 len 16384
12231 249/300 Read block 78 len 24576
12232 250/300 Read block 4303 len 36864
12233 251/300 Write block 4147 len 16384 data: 5 5 6 4
12234 252/300 Read block 4579 len 12288
12235 253/300 Flush
12236 254/300 Write block 3699 len 4096 data: 3
12237 255/300 Read block 4323 len 40960
12238 256/300 Read block 2765 len 28672
12239 257/300 Read block 733 len 8192
12240 258/300 Read block 912 len 4096
12241 259/300 Write block 1598 len 8192 data: 5 6
12242 260/300 Read block 2448 len 32768
12243 261/300 Write block 4776 len 36864 data: 5 5 5 6 6 6 6 6 5
12244 262/300 Read block 4702 len 24576
12245 263/300 Read block 1816 len 36864
12246 264/300 Read block 1992 len 16384
12247 265/300 Write block 4039 len 16384 data: 3 3 3 3
12248 266/300 Read block 888 len 8192
12249 267/300 Write block 4402 len 12288 data: 2 3 4
12250 268/300 Write block 2508 len 32768 data: 2 2 2 2 2 2 2 3
12251 269/300 Read block 984 len 4096
12252 270/300 Read block 1263 len 40960
12253 271/300 Read block 4612 len 20480
12254 272/300 Flush
12255 273/300 Flush
12256 274/300 Write block 2176 len 12288 data: 3 4 5
12257 275/300 Read block 1413 len 32768
12258 276/300 Write block 3335 len 8192 data: 4 4
12259 277/300 Flush
12260 278/300 Read block 344 len 20480
12261 279/300 Read block 4127 len 20480
12262 280/300 Read block 478 len 28672
12263 281/300 Write block 3418 len 36864 data: 4 3 3 3 5 5 5 5 4
12264 282/300 Write block 1837 len 36864 data: 3 3 2 2 2 3 3 3 4
12265 283/300 Write block 23 len 40960 data: 4 4 4 5 5 6 4 3 4 4
12266 284/300 Read block 4056 len 4096
12267 285/300 Flush
12268 286/300 Read block 2545 len 4096
12269 287/300 Read block 4552 len 32768
12270 288/300 Read block 3599 len 28672
12271 289/300 Read block 1095 len 16384
12272 290/300 Read block 4525 len 16384
12273 291/300 Write block 4899 len 28672 data: 2 4 4 4 5 5 5
12274 292/300 Read block 4027 len 24576
12275 293/300 Read block 3058 len 16384
12276 294/300 Read block 4267 len 12288
12277 295/300 Read block 672 len 8192
12278 296/300 Read block 1341 len 16384
12279 297/300 Read block 2718 len 12288
12280 298/300 Read block 3838 len 8192
12281 299/300 Write block 4107 len 12288 data: 6 5 4
12282 300/300 Read block 2200 len 36864
12283 301/300 Write block 31 len 24576 data: 5 5 4 4 4 4
12284 Main task has work to do, go find it
12285 got dsc Start(1)
12286 start 1
12287 Mar 17 22:19:01.981 INFO request completed, latency_us: 269, response_code: 204, uri: /start/cid/1, method: POST, req_id: fbe2fc0c-be08-42b3-8ba0-81e785a4baaf, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
12288 Replay: started 1, returned:Ok(())
12289 ----------------------------------------------------------------
12290 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12291 States: Active Offline Active
12292 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12293 4019 Acked Read 2 Done Sent Done false
12294 4020 Acked Write 4 Done Sent Done false
12295 4021 Acked Write 8 Done Sent Done false
12296 4022 Acked Read 1 Done Sent Done false
12297 4023 Acked Write 10 Done Sent Done false
12298 4024 Acked Read 1 Done Sent Done false
12299 4025 Acked Write 5 Done Sent Done false
12300 4026 Acked Flush 0 Done Sent Done false
12301 4027 Acked Write 9 Done Sent Done false
12302 4028 Acked Write 3 Done Sent Done false
12303 4029 Acked Write 4 Done Sent Done false
12304 4030 Acked Read 1 Done Sent Done false
12305 4031 Acked Read 5 Done Sent Done false
12306 4032 Acked Flush 0 Done Sent Done false
12307 4033 Acked Write 5 Done Sent Done false
12308 4034 Acked Read 6 Done Sent Done false
12309 4035 Acked Read 9 Done Sent Done false
12310 4036 Acked Write 9 Done Sent Done false
12311 4037 Acked Write 5 Done Sent Done false
12312 4038 Acked Write 10 Done Sent Done false
12313 4039 Acked Read 3 Done Sent Done false
12314 4040 Acked Write 2 Done Sent Done false
12315 4041 Acked Write 5 Done Sent Done false
12316 4042 Acked Read 8 Done Sent Done false
12317 4043 Acked Write 9 Done Sent Done false
12318 4044 Acked Write 7 Done Sent Done false
12319 4045 Acked Write 7 Done Sent Done false
12320 4046 Acked Write 3 Done Sent Done false
12321 4047 Acked Read 8 Done Sent Done false
12322 4048 Acked Flush 0 Done Sent Done false
12323 4049 Acked Flush 0 Done Sent Done false
12324 4050 Acked Write 1 Done Sent Done false
12325 4051 Acked Read 4 Done Sent Done false
12326 4052 Acked Read 2 Done Sent Done false
12327 4053 Acked Read 4 Done Sent Done false
12328 4054 Acked Read 7 Done Sent Done false
12329 4055 Acked Read 6 Done Sent Done false
12330 4056 Acked Flush 0 Done Sent Done false
12331 4057 Acked Read 7 Done Sent Done false
12332 4058 Acked Read 6 Done Sent Done false
12333 4059 Acked Read 9 Done Sent Done false
12334 4060 Acked Write 5 Done Sent Done false
12335 4061 Acked Read 10 Done Sent Done false
12336 4062 Acked Read 1 Done Sent Done false
12337 4063 Acked Read 7 Done Sent Done false
12338 4064 Acked Write 5 Done Sent Done false
12339 4065 Acked Read 3 Done Sent Done false
12340 4066 Acked Read 9 Done Sent Done false
12341 4067 Acked Write 5 Done Sent Done false
12342 4068 Acked Read 2 Done Sent Done false
12343 4069 Acked Flush 0 Done Sent Done false
12344 4070 Acked Read 7 Done Sent Done false
12345 4071 Acked Read 7 Done Sent Done false
12346 4072 Acked Read 3 Done Sent Done false
12347 4073 Acked Read 3 Done Sent Done false
12348 4074 Acked Read 6 Done Sent Done false
12349 4075 Acked Write 1 Done Sent Done false
12350 4076 Acked Flush 0 Done Sent Done false
12351 4077 Acked Read 2 Done Sent Done false
12352 4078 Acked Write 4 Done Sent Done false
12353 4079 Acked Write 5 Done Sent Done false
12354 4080 Acked Read 7 Done Sent Done false
12355 4081 Acked Flush 0 Done Sent Done false
12356 4082 Acked Write 10 Done Sent Done false
12357 4083 Acked Read 7 Done Sent Done false
12358 4084 Acked Read 6 Done Sent Done false
12359 4085 Acked Read 1 Done Sent Done false
12360 4086 Acked Write 1 Done Sent Done false
12361 4087 Acked Write 9 Done Sent Done false
12362 4088 Acked Flush 0 Done Sent Done false
12363 4089 Acked Write 10 Done Sent Done false
12364 4090 Acked Read 2 Done Sent Done false
12365 4091 Acked Read 2 Done Sent Done false
12366 4092 Acked Write 3 Done Sent Done false
12367 4093 Acked Write 1 Done Sent Done false
12368 4094 Acked Write 7 Done Sent Done false
12369 4095 Acked Read 9 Done Sent Done false
12370 4096 Acked Write 8 Done Sent Done false
12371 4097 Acked Write 8 Done Sent Done false
12372 4098 Acked Write 2 Done Sent Done false
12373 4099 Acked Read 6 Done Sent Done false
12374 4100 Acked Read 4 Done Sent Done false
12375 4101 Acked Write 10 Done Sent Done false
12376 4102 Acked Read 10 Done Sent Done false
12377 4103 Acked Read 5 Done Sent Done false
12378 4104 Acked Write 1 Done Sent Done false
12379 4105 Acked Write 4 Done Sent Done false
12380 4106 Acked Read 5 Done Sent Done false
12381 4107 Acked Write 3 Done Sent Done false
12382 4108 Acked Write 4 Done Sent Done false
12383 4109 Acked Write 7 Done Sent Done false
12384 4110 Acked Read 3 Done Sent Done false
12385 4111 Acked Read 5 Done Sent Done false
12386 4112 Acked Read 6 Done Sent Done false
12387 4113 Acked Flush 0 Done Sent Done false
12388 4114 Acked Read 10 Done Sent Done false
12389 4115 Acked Read 2 Done Sent Done false
12390 4116 Acked Write 1 Done Sent Done false
12391 4117 Acked Write 1 Done Sent Done false
12392 4118 Acked Read 5 Done Sent Done false
12393 4119 Acked Read 9 Done Sent Done false
12394 4120 Acked Flush 0 Done Sent Done false
12395 4121 Acked Read 6 Done Sent Done false
12396 4122 Acked Write 2 Done Sent Done false
12397 4123 Acked Read 6 Done Sent Done false
12398 4124 Acked Flush 0 Done Sent Done false
12399 4125 Acked Read 6 Done Sent Done false
12400 4126 Acked Write 7 Done Sent Done false
12401 4127 Acked Read 3 Done Sent Done false
12402 4128 Acked Write 6 Done Sent Done false
12403 4129 Acked Write 1 Done Sent Done false
12404 4130 Acked Read 4 Done Sent Done false
12405 4131 Acked Write 7 Done Sent Done false
12406 4132 Acked Read 5 Done Sent Done false
12407 4133 Acked Read 2 Done Sent Done false
12408 4134 Acked Read 1 Done Sent Done false
12409 4135 Acked Read 8 Done Sent Done false
12410 4136 Acked Read 6 Done Sent Done false
12411 4137 Acked Read 1 Done Sent Done false
12412 4138 Acked Flush 0 Done Sent Done false
12413 4139 Acked Write 10 Done Sent Done false
12414 4140 Acked Write 3 Done Sent Done false
12415 4141 Acked Write 2 Done Sent Done false
12416 4142 Acked Write 10 Done Sent Done false
12417 4143 Acked Write 4 Done Sent Done false
12418 4144 Acked Write 2 Done Sent Done false
12419 4145 Acked Write 3 Done Sent Done false
12420 4146 Acked Flush 0 Done Sent Done false
12421 4147 Acked Flush 0 Done Sent Done false
12422 4148 Acked Read 7 Done Sent Done false
12423 4149 Acked Read 8 Done Sent Done false
12424 4150 Acked Write 7 Done Sent Done false
12425 4151 Acked Read 7 Done Sent Done false
12426 4152 Acked Read 2 Done Sent Done false
12427 4153 Acked Write 2 Done Sent Done false
12428 4154 Acked Read 2 Done Sent Done false
12429 4155 Acked Read 6 Done Sent Done false
12430 4156 Acked Read 1 Done Sent Done false
12431 4157 Acked Read 8 Done Sent Done false
12432 4158 Acked Read 2 Done Sent Done false
12433 4159 Acked Flush 0 Done Sent Done false
12434 4160 Acked Write 9 Done Sent Done false
12435 4161 Acked Read 9 Done Sent Done false
12436 4162 Acked Write 2 Done Sent Done false
12437 4163 Acked Read 1 Done Sent Done false
12438 4164 Acked Read 10 Done Sent Done false
12439 4165 Acked Write 9 Done Sent Done false
12440 4166 Acked Read 7 Done Sent Done false
12441 4167 Acked Read 10 Done Sent Done false
12442 4168 Acked Write 5 Done Sent Done false
12443 4169 Acked Write 5 Done Sent Done false
12444 4170 Acked Read 3 Done Sent Done false
12445 4171 Acked Write 3 Done Sent Done false
12446 4172 Acked Write 7 Done Sent Done false
12447 4173 Acked Write 9 Done Sent Done false
12448 4174 Acked Write 8 Done Sent Done false
12449 4175 Acked Read 2 Done Sent Done false
12450 4176 Acked Read 9 Done Sent Done false
12451 4177 Acked Flush 0 Done Sent Done false
12452 4178 Acked Write 7 Done Sent Done false
12453 4179 Acked Read 8 Done Sent Done false
12454 4180 Acked Write 1 Done Sent Done false
12455 4181 Acked Write 9 Done Sent Done false
12456 4182 Acked Write 5 Done Sent Done false
12457 4183 Acked Read 6 Done Sent Done false
12458 4184 Acked Write 8 Done Sent Done false
12459 4185 Acked Write 2 Done Sent Done false
12460 4186 Acked Read 4 Done Sent Done false
12461 4187 Acked Read 6 Done Sent Done false
12462 4188 Acked Write 4 Done Sent Done false
12463 4189 Acked Read 3 Done Sent Done false
12464 4190 Acked Read 5 Done Sent Done false
12465 4191 Acked Read 5 Done Sent Done false
12466 4192 Acked Flush 0 Done Sent Done false
12467 4193 Acked Read 1 Done Sent Done false
12468 4194 Acked Read 9 Done Sent Done false
12469 4195 Acked Read 2 Done Sent Done false
12470 4196 Acked Write 8 Done Sent Done false
12471 4197 Acked Read 5 Done Sent Done false
12472 4198 Acked Flush 0 Done Sent Done false
12473 4199 Acked Write 8 Done Sent Done false
12474 4200 Acked Read 4 Done Sent Done false
12475 4201 Acked Read 5 Done Sent Done false
12476 4202 Acked Read 1 Done Sent Done false
12477 4203 Acked Write 1 Done Sent Done false
12478 4204 Acked Write 8 Done Sent Done false
12479 4205 Acked Read 3 Done Sent Done false
12480 4206 Acked Read 8 Done Sent Done false
12481 4207 Acked Flush 0 Done Sent Done false
12482 4208 Acked Read 4 Done Sent Done false
12483 4209 Acked Read 2 Done Sent Done false
12484 4210 Acked Read 7 Done Sent Done false
12485 4211 Acked Flush 0 Done Sent Done false
12486 4212 Acked Write 3 Done Sent Done false
12487 4213 Acked Write 10 Done Sent Done false
12488 4214 Acked Read 8 Done Sent Done false
12489 4215 Acked Read 8 Done Sent Done false
12490 4216 Acked Write 6 Done Sent Done false
12491 4217 Acked Read 5 Done Sent Done false
12492 4218 Acked Write 3 Done Sent Done false
12493 4219 Acked Read 2 Done Sent Done false
12494 4220 Acked Write 7 Done Sent Done false
12495 4221 Acked Read 6 Done Sent Done false
12496 4222 Acked Read 3 Done Sent Done false
12497 4223 Acked Write 1 Done Sent Done false
12498 4224 Acked Flush 0 Done Sent Done false
12499 4225 Acked Write 10 Done Sent Done false
12500 4226 Acked Write 4 Done Sent Done false
12501 4227 Acked Read 3 Done Sent Done false
12502 4228 Acked Read 8 Done Sent Done false
12503 4229 Acked Write 10 Done Sent Done false
12504 4230 Acked Read 4 Done Sent Done false
12505 4231 Acked Flush 0 Done Sent Done false
12506 4232 Acked Read 3 Done Sent Done false
12507 4233 Acked Write 8 Done Sent Done false
12508 4234 Acked Write 7 Done Sent Done false
12509 4235 Acked Read 2 Done Sent Done false
12510 4236 Acked Write 10 Done Sent Done false
12511 4237 Acked Read 3 Done Sent Done false
12512 4238 Acked Read 4 Done Sent Done false
12513 4239 Acked Read 5 Done Sent Done false
12514 4240 Acked Read 7 Done Sent Done false
12515 4241 Acked Read 4 Done Sent Done false
12516 4242 Acked Write 9 Done Sent Done false
12517 4243 Acked Write 1 Done Sent Done false
12518 4244 Acked Write 5 Done Sent Done false
12519 4245 Acked Read 4 Done Sent Done false
12520 4246 Acked Write 1 Done Sent Done false
12521 4247 Acked Write 4 Done Sent Done false
12522 4248 Acked Write 10 Done Sent Done false
12523 4249 Acked Write 9 Done Sent Done false
12524 4250 Acked Read 5 Done Sent Done false
12525 4251 Acked Write 10 Done Sent Done false
12526 4252 Acked Flush 0 Done Sent Done false
12527 4253 Acked Read 6 Done Sent Done false
12528 4254 Acked Write 4 Done Sent Done false
12529 4255 Acked Read 7 Done Sent Done false
12530 4256 Acked Read 1 Done Sent Done false
12531 4257 Acked Write 8 Done Sent Done false
12532 4258 Acked Write 7 Done Sent Done false
12533 4259 Acked Read 2 Done Sent Done false
12534 4260 Acked Write 10 Done Sent Done false
12535 4261 Acked Read 3 Done Sent Done false
12536 4262 Acked Write 6 Done Sent Done false
12537 4263 Acked Flush 0 Done Sent Done false
12538 4264 Acked Write 7 Done Sent Done false
12539 4265 Acked Read 10 Done Sent Done false
12540 4266 Acked Read 4 Done Sent Done false
12541 4267 Acked Read 6 Done Sent Done false
12542 4268 Acked Read 9 Done Sent Done false
12543 4269 Acked Write 4 Done Sent Done false
12544 4270 Acked Read 3 Done Sent Done false
12545 4271 Acked Flush 0 Done Sent Done false
12546 4272 Acked Write 1 Done Sent Done false
12547 4273 Acked Read 10 Done Sent Done false
12548 4274 Acked Read 7 Done Sent Done false
12549 4275 Acked Read 2 Done Sent Done false
12550 4276 Acked Read 1 Done Sent Done false
12551 4277 Acked Write 2 Done Sent Done false
12552 4278 Acked Read 8 Done Sent Done false
12553 4279 Acked Write 9 Done Sent Done false
12554 4280 Acked Read 6 Done Sent Done false
12555 4281 Acked Read 9 Done Sent Done false
12556 4282 Acked Read 4 Done Sent Done false
12557 4283 Acked Write 4 Done Sent Done false
12558 4284 Acked Read 2 Done Sent Done false
12559 4285 Acked Write 3 Done Sent Done false
12560 4286 Acked Write 8 Done Sent Done false
12561 4287 Acked Read 1 Done Sent Done false
12562 4288 Acked Read 10 Done Sent Done false
12563 4289 Acked Read 5 Done Sent Done false
12564 4290 Acked Flush 0 Done Sent Done false
12565 4291 Acked Flush 0 Done Sent Done false
12566 4292 Acked Write 3 Done Sent Done false
12567 4293 Acked Read 8 Done Sent Done false
12568 4294 Acked Write 2 Done Sent Done false
12569 4295 Acked Flush 0 Done Sent Done false
12570 4296 Acked Read 5 Done Sent Done false
12571 4297 Acked Read 5 Done Sent Done false
12572 4298 Acked Read 7 Done Sent Done false
12573 4299 Acked Write 9 Done Sent Done false
12574 4300 Acked Write 9 Done Sent Done false
12575 4301 Acked Write 10 Done Sent Done false
12576 4302 Acked Read 1 Done Sent Done false
12577 4303 Acked Flush 0 Done Sent Done false
12578 4304 Acked Read 1 Done Sent Done false
12579 4305 Acked Read 8 Done Sent Done false
12580 4306 Acked Read 7 Done Sent Done false
12581 4307 Acked Read 4 Done Sent Done false
12582 4308 Acked Read 4 Done Sent Done false
12583 4309 Acked Write 7 Done Sent Done false
12584 4310 Acked Read 6 Done Sent Done false
12585 4311 Acked Read 4 Done Sent Done false
12586 4312 Acked Read 3 Done Sent Done false
12587 4313 Acked Read 2 Done Sent Done false
12588 4314 Acked Read 4 Done Sent Done false
12589 4315 Acked Read 3 Done Sent Done false
12590 4316 Acked Read 2 Done Sent Done false
12591 4317 Acked Write 3 Done Sent Done false
12592 4318 Acked Read 9 Done Sent Done false
12593 4319 Acked Write 6 Done Sent Done false
12594 STATES DS:0 DS:1 DS:2 TOTAL
12595 Sent 0 301 0 301
12596 Done 301 0 301 602
12597 Skipped 0 0 0 0
12598 Error 0 0 0 0
12599 Last Flush: Some(JobId(4303)) Some(JobId(4018)) Some(JobId(4303))
12600 Downstairs last five completed: 4018 4017 4016 4015 4014
12601 Upstairs last five completed: 4319 4318 4317 4316 4315
12602 CLIENT: Up:0 ds:301 act:2
126032025-03-17T22:19:02.856ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
12604 [8820] got start action
12605 [8820] I am going to restart
12606 Starting downstairs at port 8820
12607 Make output file at "/tmp/dsc/downstairs-8820.txt"
12608 [8820][1] reports Starting
12609 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1217)
12610 [8820][1] reports Running
126112025-03-17T22:19:03.857ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
126122025-03-17T22:19:04.357ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
126132025-03-17T22:19:05.207ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
126142025-03-17T22:19:05.972ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
12615 ----------------------------------------------------------------
12616 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12617 States: Active Offline Active
12618 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
12619 4019 Acked Read 2 Done Sent Done false
12620 4020 Acked Write 4 Done Sent Done false
12621 4021 Acked Write 8 Done Sent Done false
12622 4022 Acked Read 1 Done Sent Done false
12623 4023 Acked Write 10 Done Sent Done false
12624 4024 Acked Read 1 Done Sent Done false
12625 4025 Acked Write 5 Done Sent Done false
12626 4026 Acked Flush 0 Done Sent Done false
12627 4027 Acked Write 9 Done Sent Done false
12628 4028 Acked Write 3 Done Sent Done false
12629 4029 Acked Write 4 Done Sent Done false
12630 4030 Acked Read 1 Done Sent Done false
12631 4031 Acked Read 5 Done Sent Done false
12632 4032 Acked Flush 0 Done Sent Done false
12633 4033 Acked Write 5 Done Sent Done false
12634 4034 Acked Read 6 Done Sent Done false
12635 4035 Acked Read 9 Done Sent Done false
12636 4036 Acked Write 9 Done Sent Done false
12637 4037 Acked Write 5 Done Sent Done false
12638 4038 Acked Write 10 Done Sent Done false
12639 4039 Acked Read 3 Done Sent Done false
12640 4040 Acked Write 2 Done Sent Done false
12641 4041 Acked Write 5 Done Sent Done false
12642 4042 Acked Read 8 Done Sent Done false
12643 4043 Acked Write 9 Done Sent Done false
12644 4044 Acked Write 7 Done Sent Done false
12645 4045 Acked Write 7 Done Sent Done false
12646 4046 Acked Write 3 Done Sent Done false
12647 4047 Acked Read 8 Done Sent Done false
12648 4048 Acked Flush 0 Done Sent Done false
12649 4049 Acked Flush 0 Done Sent Done false
12650 4050 Acked Write 1 Done Sent Done false
12651 4051 Acked Read 4 Done Sent Done false
12652 4052 Acked Read 2 Done Sent Done false
12653 4053 Acked Read 4 Done Sent Done false
12654 4054 Acked Read 7 Done Sent Done false
12655 4055 Acked Read 6 Done Sent Done false
12656 4056 Acked Flush 0 Done Sent Done false
12657 4057 Acked Read 7 Done Sent Done false
12658 4058 Acked Read 6 Done Sent Done false
12659 4059 Acked Read 9 Done Sent Done false
12660 4060 Acked Write 5 Done Sent Done false
12661 4061 Acked Read 10 Done Sent Done false
12662 4062 Acked Read 1 Done Sent Done false
12663 4063 Acked Read 7 Done Sent Done false
12664 4064 Acked Write 5 Done Sent Done false
12665 4065 Acked Read 3 Done Sent Done false
12666 4066 Acked Read 9 Done Sent Done false
12667 4067 Acked Write 5 Done Sent Done false
12668 4068 Acked Read 2 Done Sent Done false
12669 4069 Acked Flush 0 Done Sent Done false
12670 4070 Acked Read 7 Done Sent Done false
12671 4071 Acked Read 7 Done Sent Done false
12672 4072 Acked Read 3 Done Sent Done false
12673 4073 Acked Read 3 Done Sent Done false
12674 4074 Acked Read 6 Done Sent Done false
12675 4075 Acked Write 1 Done Sent Done false
12676 4076 Acked Flush 0 Done Sent Done false
12677 4077 Acked Read 2 Done Sent Done false
12678 4078 Acked Write 4 Done Sent Done false
12679 4079 Acked Write 5 Done Sent Done false
12680 4080 Acked Read 7 Done Sent Done false
12681 4081 Acked Flush 0 Done Sent Done false
12682 4082 Acked Write 10 Done Sent Done false
12683 4083 Acked Read 7 Done Sent Done false
12684 4084 Acked Read 6 Done Sent Done false
12685 4085 Acked Read 1 Done Sent Done false
12686 4086 Acked Write 1 Done Sent Done false
12687 4087 Acked Write 9 Done Sent Done false
12688 4088 Acked Flush 0 Done Sent Done false
12689 4089 Acked Write 10 Done Sent Done false
12690 4090 Acked Read 2 Done Sent Done false
12691 4091 Acked Read 2 Done Sent Done false
12692 4092 Acked Write 3 Done Sent Done false
12693 4093 Acked Write 1 Done Sent Done false
12694 4094 Acked Write 7 Done Sent Done false
12695 4095 Acked Read 9 Done Sent Done false
12696 4096 Acked Write 8 Done Sent Done false
12697 4097 Acked Write 8 Done Sent Done false
12698 4098 Acked Write 2 Done Sent Done false
12699 4099 Acked Read 6 Done Sent Done false
12700 4100 Acked Read 4 Done Sent Done false
12701 4101 Acked Write 10 Done Sent Done false
12702 4102 Acked Read 10 Done Sent Done false
12703 4103 Acked Read 5 Done Sent Done false
12704 4104 Acked Write 1 Done Sent Done false
12705 4105 Acked Write 4 Done Sent Done false
12706 4106 Acked Read 5 Done Sent Done false
12707 4107 Acked Write 3 Done Sent Done false
12708 4108 Acked Write 4 Done Sent Done false
12709 4109 Acked Write 7 Done Sent Done false
12710 4110 Acked Read 3 Done Sent Done false
12711 4111 Acked Read 5 Done Sent Done false
12712 4112 Acked Read 6 Done Sent Done false
12713 4113 Acked Flush 0 Done Sent Done false
12714 4114 Acked Read 10 Done Sent Done false
12715 4115 Acked Read 2 Done Sent Done false
12716 4116 Acked Write 1 Done Sent Done false
12717 4117 Acked Write 1 Done Sent Done false
12718 4118 Acked Read 5 Done Sent Done false
12719 4119 Acked Read 9 Done Sent Done false
12720 4120 Acked Flush 0 Done Sent Done false
12721 4121 Acked Read 6 Done Sent Done false
12722 4122 Acked Write 2 Done Sent Done false
12723 4123 Acked Read 6 Done Sent Done false
12724 4124 Acked Flush 0 Done Sent Done false
12725 4125 Acked Read 6 Done Sent Done false
12726 4126 Acked Write 7 Done Sent Done false
12727 4127 Acked Read 3 Done Sent Done false
12728 4128 Acked Write 6 Done Sent Done false
12729 4129 Acked Write 1 Done Sent Done false
12730 4130 Acked Read 4 Done Sent Done false
12731 4131 Acked Write 7 Done Sent Done false
12732 4132 Acked Read 5 Done Sent Done false
12733 4133 Acked Read 2 Done Sent Done false
12734 4134 Acked Read 1 Done Sent Done false
12735 4135 Acked Read 8 Done Sent Done false
12736 4136 Acked Read 6 Done Sent Done false
12737 4137 Acked Read 1 Done Sent Done false
12738 4138 Acked Flush 0 Done Sent Done false
12739 4139 Acked Write 10 Done Sent Done false
12740 4140 Acked Write 3 Done Sent Done false
12741 4141 Acked Write 2 Done Sent Done false
12742 4142 Acked Write 10 Done Sent Done false
12743 4143 Acked Write 4 Done Sent Done false
12744 4144 Acked Write 2 Done Sent Done false
12745 4145 Acked Write 3 Done Sent Done false
12746 4146 Acked Flush 0 Done Sent Done false
12747 4147 Acked Flush 0 Done Sent Done false
12748 4148 Acked Read 7 Done Sent Done false
12749 4149 Acked Read 8 Done Sent Done false
12750 4150 Acked Write 7 Done Sent Done false
12751 4151 Acked Read 7 Done Sent Done false
12752 4152 Acked Read 2 Done Sent Done false
12753 4153 Acked Write 2 Done Sent Done false
12754 4154 Acked Read 2 Done Sent Done false
12755 4155 Acked Read 6 Done Sent Done false
12756 4156 Acked Read 1 Done Sent Done false
12757 4157 Acked Read 8 Done Sent Done false
12758 4158 Acked Read 2 Done Sent Done false
12759 4159 Acked Flush 0 Done Sent Done false
12760 4160 Acked Write 9 Done Sent Done false
12761 4161 Acked Read 9 Done Sent Done false
12762 4162 Acked Write 2 Done Sent Done false
12763 4163 Acked Read 1 Done Sent Done false
12764 4164 Acked Read 10 Done Sent Done false
12765 4165 Acked Write 9 Done Sent Done false
12766 4166 Acked Read 7 Done Sent Done false
12767 4167 Acked Read 10 Done Sent Done false
12768 4168 Acked Write 5 Done Sent Done false
12769 4169 Acked Write 5 Done Sent Done false
12770 4170 Acked Read 3 Done Sent Done false
12771 4171 Acked Write 3 Done Sent Done false
12772 4172 Acked Write 7 Done Sent Done false
12773 4173 Acked Write 9 Done Sent Done false
12774 4174 Acked Write 8 Done Sent Done false
12775 4175 Acked Read 2 Done Sent Done false
12776 4176 Acked Read 9 Done Sent Done false
12777 4177 Acked Flush 0 Done Sent Done false
12778 4178 Acked Write 7 Done Sent Done false
12779 4179 Acked Read 8 Done Sent Done false
12780 4180 Acked Write 1 Done Sent Done false
12781 4181 Acked Write 9 Done Sent Done false
12782 4182 Acked Write 5 Done Sent Done false
12783 4183 Acked Read 6 Done Sent Done false
12784 4184 Acked Write 8 Done Sent Done false
12785 4185 Acked Write 2 Done Sent Done false
12786 4186 Acked Read 4 Done Sent Done false
12787 4187 Acked Read 6 Done Sent Done false
12788 4188 Acked Write 4 Done Sent Done false
12789 4189 Acked Read 3 Done Sent Done false
12790 4190 Acked Read 5 Done Sent Done false
12791 4191 Acked Read 5 Done Sent Done false
12792 4192 Acked Flush 0 Done Sent Done false
12793 4193 Acked Read 1 Done Sent Done false
12794 4194 Acked Read 9 Done Sent Done false
12795 4195 Acked Read 2 Done Sent Done false
12796 4196 Acked Write 8 Done Sent Done false
12797 4197 Acked Read 5 Done Sent Done false
12798 4198 Acked Flush 0 Done Sent Done false
12799 4199 Acked Write 8 Done Sent Done false
12800 4200 Acked Read 4 Done Sent Done false
12801 4201 Acked Read 5 Done Sent Done false
12802 4202 Acked Read 1 Done Sent Done false
12803 4203 Acked Write 1 Done Sent Done false
12804 4204 Acked Write 8 Done Sent Done false
12805 4205 Acked Read 3 Done Sent Done false
12806 4206 Acked Read 8 Done Sent Done false
12807 4207 Acked Flush 0 Done Sent Done false
12808 4208 Acked Read 4 Done Sent Done false
12809 4209 Acked Read 2 Done Sent Done false
12810 4210 Acked Read 7 Done Sent Done false
12811 4211 Acked Flush 0 Done Sent Done false
12812 4212 Acked Write 3 Done Sent Done false
12813 4213 Acked Write 10 Done Sent Done false
12814 4214 Acked Read 8 Done Sent Done false
12815 4215 Acked Read 8 Done Sent Done false
12816 4216 Acked Write 6 Done Sent Done false
12817 4217 Acked Read 5 Done Sent Done false
12818 4218 Acked Write 3 Done Sent Done false
12819 4219 Acked Read 2 Done Sent Done false
12820 4220 Acked Write 7 Done Sent Done false
12821 4221 Acked Read 6 Done Sent Done false
12822 4222 Acked Read 3 Done Sent Done false
12823 4223 Acked Write 1 Done Sent Done false
12824 4224 Acked Flush 0 Done Sent Done false
12825 4225 Acked Write 10 Done Sent Done false
12826 4226 Acked Write 4 Done Sent Done false
12827 4227 Acked Read 3 Done Sent Done false
12828 4228 Acked Read 8 Done Sent Done false
12829 4229 Acked Write 10 Done Sent Done false
12830 4230 Acked Read 4 Done Sent Done false
12831 4231 Acked Flush 0 Done Sent Done false
12832 4232 Acked Read 3 Done Sent Done false
12833 4233 Acked Write 8 Done Sent Done false
12834 4234 Acked Write 7 Done Sent Done false
12835 4235 Acked Read 2 Done Sent Done false
12836 4236 Acked Write 10 Done Sent Done false
12837 4237 Acked Read 3 Done Sent Done false
12838 4238 Acked Read 4 Done Sent Done false
12839 4239 Acked Read 5 Done Sent Done false
12840 4240 Acked Read 7 Done Sent Done false
12841 4241 Acked Read 4 Done Sent Done false
12842 4242 Acked Write 9 Done Sent Done false
12843 4243 Acked Write 1 Done Sent Done false
12844 4244 Acked Write 5 Done Sent Done false
12845 4245 Acked Read 4 Done Sent Done false
12846 4246 Acked Write 1 Done Sent Done false
12847 4247 Acked Write 4 Done Sent Done false
12848 4248 Acked Write 10 Done Sent Done false
12849 4249 Acked Write 9 Done Sent Done false
12850 4250 Acked Read 5 Done Sent Done false
12851 4251 Acked Write 10 Done Sent Done false
12852 4252 Acked Flush 0 Done Sent Done false
12853 4253 Acked Read 6 Done Sent Done false
12854 4254 Acked Write 4 Done Sent Done false
12855 4255 Acked Read 7 Done Sent Done false
12856 4256 Acked Read 1 Done Sent Done false
12857 4257 Acked Write 8 Done Sent Done false
12858 4258 Acked Write 7 Done Sent Done false
12859 4259 Acked Read 2 Done Sent Done false
12860 4260 Acked Write 10 Done Sent Done false
12861 4261 Acked Read 3 Done Sent Done false
12862 4262 Acked Write 6 Done Sent Done false
12863 4263 Acked Flush 0 Done Sent Done false
12864 4264 Acked Write 7 Done Sent Done false
12865 4265 Acked Read 10 Done Sent Done false
12866 4266 Acked Read 4 Done Sent Done false
12867 4267 Acked Read 6 Done Sent Done false
12868 4268 Acked Read 9 Done Sent Done false
12869 4269 Acked Write 4 Done Sent Done false
12870 4270 Acked Read 3 Done Sent Done false
12871 4271 Acked Flush 0 Done Sent Done false
12872 4272 Acked Write 1 Done Sent Done false
12873 4273 Acked Read 10 Done Sent Done false
12874 4274 Acked Read 7 Done Sent Done false
12875 4275 Acked Read 2 Done Sent Done false
12876 4276 Acked Read 1 Done Sent Done false
12877 4277 Acked Write 2 Done Sent Done false
12878 4278 Acked Read 8 Done Sent Done false
12879 4279 Acked Write 9 Done Sent Done false
12880 4280 Acked Read 6 Done Sent Done false
12881 4281 Acked Read 9 Done Sent Done false
12882 4282 Acked Read 4 Done Sent Done false
12883 4283 Acked Write 4 Done Sent Done false
12884 4284 Acked Read 2 Done Sent Done false
12885 4285 Acked Write 3 Done Sent Done false
12886 4286 Acked Write 8 Done Sent Done false
12887 4287 Acked Read 1 Done Sent Done false
12888 4288 Acked Read 10 Done Sent Done false
12889 4289 Acked Read 5 Done Sent Done false
12890 4290 Acked Flush 0 Done Sent Done false
12891 4291 Acked Flush 0 Done Sent Done false
12892 4292 Acked Write 3 Done Sent Done false
12893 4293 Acked Read 8 Done Sent Done false
12894 4294 Acked Write 2 Done Sent Done false
12895 4295 Acked Flush 0 Done Sent Done false
12896 4296 Acked Read 5 Done Sent Done false
12897 4297 Acked Read 5 Done Sent Done false
12898 4298 Acked Read 7 Done Sent Done false
12899 4299 Acked Write 9 Done Sent Done false
12900 4300 Acked Write 9 Done Sent Done false
12901 4301 Acked Write 10 Done Sent Done false
12902 4302 Acked Read 1 Done Sent Done false
12903 4303 Acked Flush 0 Done Sent Done false
12904 4304 Acked Read 1 Done Sent Done false
12905 4305 Acked Read 8 Done Sent Done false
12906 4306 Acked Read 7 Done Sent Done false
12907 4307 Acked Read 4 Done Sent Done false
12908 4308 Acked Read 4 Done Sent Done false
12909 4309 Acked Write 7 Done Sent Done false
12910 4310 Acked Read 6 Done Sent Done false
12911 4311 Acked Read 4 Done Sent Done false
12912 4312 Acked Read 3 Done Sent Done false
12913 4313 Acked Read 2 Done Sent Done false
12914 4314 Acked Read 4 Done Sent Done false
12915 4315 Acked Read 3 Done Sent Done false
12916 4316 Acked Read 2 Done Sent Done false
12917 4317 Acked Write 3 Done Sent Done false
12918 4318 Acked Read 9 Done Sent Done false
12919 4319 Acked Write 6 Done Sent Done false
12920 STATES DS:0 DS:1 DS:2 TOTAL
12921 Sent 0 301 0 301
12922 Done 301 0 301 602
12923 Skipped 0 0 0 0
12924 Error 0 0 0 0
12925 Last Flush: Some(JobId(4303)) Some(JobId(4018)) Some(JobId(4303))
12926 Downstairs last five completed: 4018 4017 4016 4015 4014
12927 Upstairs last five completed: 4319 4318 4317 4316 4315
12928 CLIENT: Up:0 ds:301 act:2
129292025-03-17T22:19:06.861ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
129302025-03-17T22:19:07.735ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
129312025-03-17T22:19:07.735ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:49505) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
129322025-03-17T22:19:07.736ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
129332025-03-17T22:19:07.737ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
129342025-03-17T22:19:07.737ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
129352025-03-17T22:19:07.737ZINFOcrucible: send last flush ID to this DS: Some(JobId(4018)) = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
129362025-03-17T22:19:07.737ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
129372025-03-17T22:19:07.737ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(4018)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
129382025-03-17T22:19:07.737ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(4018)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
12939 ----------------------------------------------------------------
12940 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
12941 Downstairs last five completed: 4320 4319 4318 4317 4316
12942 Upstairs last five completed: 4320 4319 4318 4317 4316
12943 CLIENT: Up:0 ds:0 act:3
12944 Replay: All jobs finished
12945 Main task has work to do, go find it
12946 got dsc Stop(1)
12947 stop 1
12948 Mar 17 22:19:09.986 INFO request completed, latency_us[8820] Got stop action so:false kr:false
12949 : 366, response_code: 204, uri: /stop/cid/1, method: POST, req_id: f4ef4a52-ac54-489e-9e24-3c4a62877da6, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
12950 Mar 17 22:19:09.986 INFO request completed, latency_us: 118, response_code: 200, uri: /state/cid/1, method: GET, req_id: 308c9091-29c0-4979-8f6d-053cf9729e23, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
129512025-03-17T22:19:09.986ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
129522025-03-17T22:19:09.987ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
129532025-03-17T22:19:09.987ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
129542025-03-17T22:19:09.987ZINFOcrucible: client task is exiting = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
12955 [8820] Exited with: ExitStatus(unix_wait_status(9))
12956 [8820][1] reports Exit
12957 Mar 17 22:19:13.989 INFO request completed, latency_us: 219, response_code: 200, uri: /state/cid/1, method: GET, req_id: 0dd8e10f-f444-4452-a698-f3a2732b04d8, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
12958 001/300 Read block 1939 len 36864
12959 002/300 Write block 2606 len 20480 data: 6 7 6 6 6
12960 003/300 Read block 1133 len 20480
12961 004/300 Read block 1396 len 4096
12962 005/300 Read block 1999 len 40960
12963 006/300 Read block 1679 len 8192
12964 007/300 Read block 4531 len 40960
12965 008/300 Read block 1255 len 40960
12966 009/300 Read block 3508 len 8192
12967 010/300 Read block 1605 len 8192
12968 011/300 Read block 3790 len 16384
12969 012/300 Read block 1053 len 4096
12970 013/300 Read block 1655 len 36864
12971 014/300 Write block 3376 len 24576 data: 2 2 3 3 3 2
12972 015/300 Read block 1287 len 32768
12973 016/300 Read block 3793 len 16384
12974 017/300 Read block 1306 len 16384
12975 018/300 Read block 2693 len 24576
12976 019/300 Write block 1732 len 20480 data: 3 3 4 4 3
12977 020/300 Write block 2998 len 16384 data: 4 4 3 4
12978 021/300 Read block 3104 len 16384
12979 022/300 Write block 2326 len 28672 data: 5 4 4 3 3 3 3
12980 023/300 Read block 3906 len 16384
12981 024/300 Flush
12982 025/300 Write block 2484 len 8192 data: 3 3
12983 026/300 Write block 3548 len 4096 data: 3
12984 027/300 Read block 903 len 28672
12985 028/300 Read block 3951 len 20480
12986 029/300 Read block 1129 len 16384
12987 030/300 Write block 3077 len 4096 data: 3
12988 031/300 Read block 4339 len 36864
12989 032/300 Write block 4938 len 40960 data: 2 2 2 2 2 3 2 2 3 3
12990 033/300 Read block 3241 len 28672
12991 034/300 Write block 247 len 28672 data: 2 2 2 3 3 2 3
12992 035/300 Read block 459 len 28672
12993 036/300 Write block 316 len 40960 data: 3 3 3 3 2 2 2 2 2 2
12994 037/300 Flush
12995 038/300 Write block 439 len 24576 data: 2 2 2 2 3 3
12996 039/300 Flush
12997 040/300 Read block 1601 len 32768
12998 041/300 Write block 409 len 4096 data: 2
12999 042/300 Read block 4865 len 36864
13000 043/300 Read block 1089 len 16384
13001 044/300 Write block 1919 len 32768 data: 2 2 3 2 2 2 2 3
13002 045/300 Write block 3653 len 36864 data: 4 3 3 3 3 3 3 3 3
13003 046/300 Read block 4703 len 12288
13004 047/300 Read block 67 len 4096
13005 048/300 Flush
13006 049/300 Read block 2881 len 8192
13007 050/300 Write block 4965 len 24576 data: 3 3 2 2 2 2
13008 051/300 Read block 3358 len 36864
13009 052/300 Read block 2440 len 40960
13010 053/300 Read block 907 len 16384
13011 054/300 Write block 1077 len 8192 data: 4 4
13012 055/300 Flush
13013 056/300 Write block 4079 len 12288 data: 3 3 3
13014 057/300 Write block 1616 len 36864 data: 2 2 2 2 2 3 3 3 2
13015 058/300 Write block 4602 len 40960 data: 3 3 5 5 6 6 7 6 6 5
13016 059/300 Write block 643 len 40960 data: 2 3 3 3 3 3 3 3 2 3
13017 060/300 Read block 3968 len 36864
13018 061/300 Read block 2912 len 16384
13019 062/300 Read block 3823 len 36864
13020 063/300 Write block 1253 len 28672 data: 2 3 3 3 3 3 3
13021 064/300 Write block 1074 len 40960 data: 4 5 5 5 5 4 3 2 2 2
13022 065/300 Write block 3554 len 16384 data: 2 2 2 2
13023 066/300 Flush
13024 067/300 Write block 3835 len 20480 data: 3 2 2 2 2
13025 068/300 Read block 2075 len 28672
13026 069/300 Write block 2063 len 28672 data: 5 4 3 3 3 3 3
13027 070/300 Read block 4918 len 12288
13028 071/300 Write block 3166 len 20480 data: 4 5 5 5 5
13029 072/300 Read block 3131 len 4096
13030 073/300 Read block 1241 len 40960
13031 074/300 Flush
13032 075/300 Read block 1307 len 8192
13033 076/300 Read block 4127 len 32768
13034 077/300 Write block 4297 len 20480 data: 3 3 4 4 6
13035 078/300 Read block 1145 len 24576
13036 079/300 Read block 1359 len 20480
13037 080/300 Write block 3389 len 4096 data: 4
13038 081/300 Read block 212 len 28672
13039 082/300 Read block 2024 len 24576
13040 083/300 Write block 4465 len 28672 data: 6 6 5 6 7 8 8
13041 084/300 Read block 2982 len 32768
13042 085/300 Read block 860 len 12288
13043 086/300 Read block 377 len 16384
13044 087/300 Write block 4573 len 12288 data: 3 3 3
13045 088/300 Read block 1344 len 8192
13046 089/300 Write block 547 len 16384 data: 3 4 4 4
13047 090/300 Write block 4915 len 40960 data: 2 2 3 3 3 3 3 4 4 4
13048 091/300 Read block 3583 len 24576
13049 092/300 Read block 3216 len 24576
13050 093/300 Read block 1721 len 40960
13051 094/300 Write block 4168 len 20480 data: 2 2 2 2 2
13052 095/300 Read block 3835 len 16384
13053 096/300 Flush
13054 097/300 Write block 3427 len 32768 data: 4 4 4 4 3 5 5 5
13055 098/300 Read block 2766 len 36864
13056 099/300 Read block 3869 len 8192
13057 100/300 Write block 4208 len 12288 data: 3 3 4
13058 101/300 Read block 650 len 16384
13059 102/300 Write block 4251 len 28672 data: 4 4 4 4 4 3 4
13060 103/300 Read block 1288 len 40960
13061 104/300 Write block 2219 len 28672 data: 6 6 5 5 4 4 4
13062 105/300 Write block 1769 len 20480 data: 3 3 2 3 3
13063 106/300 Write block 2171 len 36864 data: 2 2 2 2 3 4 5 6 4
13064 107/300 Read block 184 len 40960
13065 108/300 Write block 2948 len 12288 data: 5 2 3
13066 109/300 Write block 4633 len 40960 data: 3 3 3 3 3 3 3 2 2 3
13067 110/300 Read block 1297 len 12288
13068 111/300 Write block 4128 len 12288 data: 2 2 2
13069 112/300 Read block 1766 len 20480
13070 113/300 Write block 614 len 16384 data: 2 2 2 2
13071 114/300 Write block 4053 len 24576 data: 3 3 2 5 6 6
13072 115/300 Read block 942 len 8192
13073 116/300 Write block 309 len 12288 data: 3 2 2
13074 117/300 Write block 4895 len 20480 data: 2 2 2 2 3
13075 118/300 Read block 2641 len 4096
13076 119/300 Read block 4759 len 36864
13077 120/300 Write block 865 len 4096 data: 2
13078 121/300 Flush
13079 122/300 Flush
13080 123/300 Write block 62 len 36864 data: 5 5 5 5 4 3 3 3 2
13081 124/300 Read block 4893 len 16384
13082 125/300 Write block 137 len 16384 data: 2 2 2 3
13083 126/300 Read block 4387 len 12288
13084 127/300 Write block 721 len 12288 data: 5 4 3
13085 128/300 Write block 1250 len 36864 data: 2 2 2 3 4 4 4 4 4
13086 129/300 Write block 2445 len 36864 data: 3 3 3 2 2 3 2 2 2
13087 130/300 Write block 3065 len 40960 data: 3 3 3 4 4 4 4 3 4 3
13088 131/300 Read block 4941 len 8192
13089 132/300 Read block 1663 len 36864
13090 133/300 Write block 1745 len 12288 data: 2 2 4
13091 134/300 Read block 2564 len 16384
13092 135/300 Read block 707 len 36864
13093 136/300 Read block 4852 len 32768
13094 137/300 Write block 1632 len 40960 data: 2 2 2 2 2 3 3 3 4 4
13095 138/300 Write block 2685 len 32768 data: 2 4 4 4 4 4 4 4
13096 139/300 Write block 2352 len 36864 data: 2 2 2 3 3 3 5 5 3
13097 140/300 Read block 1892 len 4096
13098 141/300 Write block 2103 len 32768 data: 2 2 3 4 5 4 4 5
13099 142/300 Read block 3441 len 16384
13100 143/300 Write block 32 len 8192 data: 6 5
13101 144/300 Flush
13102 145/300 Read block 1860 len 28672
13103 146/300 Flush
13104 147/300 Flush
13105 148/300 Flush
13106 149/300 Write block 1340 len 12288 data: 3 4 4
13107 150/300 Read block 1403 len 32768
13108 151/300 Write block 4550 len 28672 data: 4 3 3 4 4 4 4
13109 152/300 Write block 2653 len 4096 data: 3
13110 153/300 Read block 1599 len 20480
13111 154/300 Read block 732 len 16384
13112 155/300 Flush
13113 156/300 Read block 3377 len 8192
13114 157/300 Read block 765 len 16384
13115 158/300 Write block 3017 len 12288 data: 7 7 6
13116 159/300 Write block 847 len 24576 data: 2 2 3 3 4 3
13117 160/300 Write block 589 len 32768 data: 6 6 5 5 3 4 4 3
13118 161/300 Write block 1634 len 36864 data: 3 3 3 4 4 4 5 5 4
13119 162/300 Write block 3737 len 24576 data: 2 2 3 3 3 3
13120 163/300 Read block 3131 len 32768
13121 164/300 Read block 3076 len 12288
13122 165/300 Write block 2561 len 16384 data: 2 3 3 3
13123 166/300 Flush
13124 167/300 Write block 3684 len 16384 data: 3 3 3 3
13125 168/300 Read block 2937 len 4096
13126 169/300 Read block 2351 len 40960
13127 170/300 Read block 3015 len 40960
13128 171/300 Write block 2646 len 24576 data: 3 3 3 3 2 2
13129 172/300 Read block 774 len 40960
13130 173/300 Write block 1711 len 40960 data: 2 2 2 2 2 2 2 2 2 2
13131 174/300 Read block 4732 len 32768
13132 175/300 Read block 506 len 4096
13133 176/300 Read block 2037 len 24576
13134 177/300 Write block 697 len 16384 data: 3 4 3 3
13135 178/300 Write block 3323 len 8192 data: 4 4
13136 179/300 Read block 2286 len 20480
13137 180/300 Write block 1252 len 24576 data: 3 4 5 5 5 5
13138 181/300 Write block 4337 len 16384 data: 4 4 3 3
13139 182/300 Read block 3253 len 20480
13140 183/300 Flush
13141 184/300 Read block 2753 len 24576
13142 185/300 Write block 2835 len 24576 data: 3 3 3 4 3 3
13143 186/300 Read block 2160 len 24576
13144 187/300 Read block 313 len 24576
13145 188/300 Write block 2157 len 8192 data: 2 2
13146 189/300 Write block 3129 len 40960 data: 4 5 5 5 6 6 6 5 3 3
13147 190/300 Flush
13148 191/300 Write block 4771 len 36864 data: 3 3 3 4 5 6 6 6 7
13149 192/300 Write block 425 len 4096 data: 3
13150 193/300 Write block 1672 len 20480 data: 6 6 6 4 4
13151 194/300 Read block 3066 len 20480
13152 195/300 Flush
13153 196/300 Read block 4059 len 32768
13154 197/300 Write block 2566 len 4096 data: 4
13155 198/300 Read block 3128 len 20480
13156 199/300 Read block 4003 len 12288
13157 200/300 Read block 714 len 28672
13158 201/300 Write block 4486 len 36864 data: 3 3 3 3 4 3 3 4 2
13159 202/300 Read block 756 len 36864
13160 203/300 Write block 3236 len 20480 data: 5 5 6 6 6
13161 204/300 Write block 2415 len 12288 data: 2 2 3
13162 205/300 Read block 2784 len 40960
13163 206/300 Flush
13164 207/300 Write block 590 len 28672 data: 7 6 6 4 5 5 4
13165 208/300 Read block 2729 len 8192
13166 209/300 Read block 243 len 12288
13167 210/300 Read block 3472 len 12288
13168 211/300 Read block 3366 len 28672
13169 212/300 Read block 365 len 8192
13170 213/300 Write block 258 len 8192 data: 5 5
13171 214/300 Read block 2345 len 16384
13172 215/300 Read block 1920 len 16384
13173 216/300 Read block 2078 len 16384
13174 217/300 Write block 3050 len 24576 data: 4 3 4 3 3 3
13175 218/300 Read block 338 len 32768
13176 219/300 Read block 3227 len 32768
13177 220/300 Flush
13178 221/300 Flush
13179 222/300 Read block 4768 len 32768
13180 223/300 Flush
13181 224/300 Write block 610 len 32768 data: 4 3 2 2 3 3 3 3
13182 225/300 Flush
13183 226/300 Flush
13184 227/300 Read block 3156 len 32768
13185 228/300 Write block 484 len 8192 data: 3 3
13186 229/300 Write block 1986 len 36864 data: 5 4 4 6 6 6 7 4 4
13187 230/300 Write block 342 len 36864 data: 2 2 2 2 3 4 4 3 3
13188 231/300 Write block 1098 len 36864 data: 4 4 4 4 6 5 5 4 4
13189 232/300 Read block 3933 len 20480
13190 233/300 Read block 3633 len 12288
13191 234/300 Read block 2976 len 28672
13192 235/300 Write block 1477 len 12288 data: 5 6 5
13193 236/300 Flush
13194 237/300 Read block 1433 len 20480
13195 238/300 Write block 2206 len 32768 data: 5 6 5 5 5 4 4 4
13196 239/300 Write block 4036 len 24576 data: 2 2 2 4 4 4
13197 240/300 Flush
13198 241/300 Read block 3708 len 8192
13199 242/300 Flush
13200 243/300 Write block 2609 len 32768 data: 7 7 6 4 4 4 5 3
13201 244/300 Write block 316 len 40960 data: 4 4 4 4 3 3 3 3 3 3
13202 245/300 Read block 3832 len 16384
13203 246/300 Read block 1610 len 8192
13204 247/300 Write block 3373 len 28672 data: 3 2 2 3 3 4 4
13205 248/300 Flush
13206 249/300 Read block 4979 len 8192
13207 250/300 Read block 3150 len 20480
13208 251/300 Read block 1978 len 40960
13209 252/300 Read block 1230 len 16384
13210 253/300 Read block 959 len 4096
13211 254/300 Flush
13212 255/300 Write block 4112 len 12288 data: 2 2 2
13213 256/300 Write block 2436 len 20480 data: 5 6 6 7 8
13214 257/300 Read block 1987 len 24576
13215 258/300 Write block 1398 len 28672 data: 4 4 4 4 4 4 3
13216 259/300 Write block 4342 len 20480 data: 4 3 3 2 2
13217 260/300 Read block 764 len 4096
13218 261/300 Write block 4421 len 24576 data: 5 4 4 4 4 4
13219 262/300 Flush
13220 263/300 Read block 2846 len 28672
13221 264/300 Write block 2737 len 36864 data: 2 2 2 2 2 2 2 2 2
13222 265/300 Write block 257 len 24576 data: 4 6 6 5 5 5
13223 266/300 Flush
13224 267/300 Write block 625 len 28672 data: 4 4 4 5 5 6 6
13225 268/300 Flush
13226 269/300 Read block 2975 len 32768
13227 270/300 Write block 3897 len 12288 data: 6 6 6
13228 271/300 Read block 4810 len 36864
13229 272/300 Read block 2181 len 32768
13230 273/300 Write block 4416 len 36864 data: 3 3 3 3 3 6 5 5 5
13231 274/300 Write block 858 len 8192 data: 3 2
13232 275/300 Read block 1073 len 36864
13233 276/300 Read block 4875 len 36864
13234 277/300 Write block 3123 len 32768 data: 2 2 2 2 3 3 5 6
13235 278/300 Write block 4176 len 20480 data: 2 3 3 3 3
13236 279/300 Read block 2682 len 24576
13237 280/300 Write block 4938 len 12288 data: 3 3 3
13238 281/300 Write block 2403 len 40960 data: 2 2 2 3 4 4 4 4 4 4
13239 282/300 Read block 4567 len 8192
13240 283/300 Read block 1450 len 8192
13241 284/300 Write block 1165 len 36864 data: 3 3 3 3 3 3 2 2 2
13242 285/300 Write block 2916 len 28672 data: 3 3 3 3 3 3 4
13243 286/300 Write block 1046 len 32768 data: 2 2 2 3 3 3 2 2
13244 287/300 Write block 2253 len 32768 data: 3 2 2 3 3 3 3 3
13245 288/300 Write block 988 len 20480 data: 4 4 4 4 4
13246 289/300 Read block 4604 len 32768
13247 290/300 Write block 3893 len 12288 data: 3 4 5
13248 291/300 Flush
13249 292/300 Write block 3740 len 40960 data: 4 4 4 2 2 3 3 4 3 4
13250 293/300 Flush
13251 294/300 Flush
13252 295/300 Write block 728 len 36864 data: 3 2 2 2 2 2 2 2 3
13253 296/300 Read block 1216 len 8192
13254 297/300 Write block 1326 len 12288 data: 3 3 3
13255 298/300 Read block 777 len 12288
13256 299/300 Write block 603 len 28672 data: 5 5 6 5 5 5 5
13257 300/300 Write block 4335 len 24576 data: 3 3 5 5 4 4
13258 301/300 Read block 192 len 16384
13259 Main task has work to do, go find it
13260 got dsc Start(1)
13261 start 1
13262 Mar 17 22:19:14.232 INFO request completed, latency_us: 213, response_code: 204, uri: /start/cid/1, method: POST, req_id: 25ba89af-bbae-4780-bb54-de966d38a91d, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
13263 Replay: started 1, returned:Ok(())
13264 ----------------------------------------------------------------
13265 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13266 States: Active Offline Active
13267 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13268 4321 Acked Read 9 Done Sent Done false
13269 4322 Acked Write 5 Done Sent Done false
13270 4323 Acked Read 5 Done Sent Done false
13271 4324 Acked Read 1 Done Sent Done false
13272 4325 Acked Read 10 Done Sent Done false
13273 4326 Acked Read 2 Done Sent Done false
13274 4327 Acked Read 10 Done Sent Done false
13275 4328 Acked Read 10 Done Sent Done false
13276 4329 Acked Read 2 Done Sent Done false
13277 4330 Acked Read 2 Done Sent Done false
13278 4331 Acked Read 4 Done Sent Done false
13279 4332 Acked Read 1 Done Sent Done false
13280 4333 Acked Read 9 Done Sent Done false
13281 4334 Acked Write 6 Done Sent Done false
13282 4335 Acked Read 8 Done Sent Done false
13283 4336 Acked Read 4 Done Sent Done false
13284 4337 Acked Read 4 Done Sent Done false
13285 4338 Acked Read 6 Done Sent Done false
13286 4339 Acked Write 5 Done Sent Done false
13287 4340 Acked Write 4 Done Sent Done false
13288 4341 Acked Read 4 Done Sent Done false
13289 4342 Acked Write 7 Done Sent Done false
13290 4343 Acked Read 4 Done Sent Done false
13291 4344 Acked Flush 0 Done Sent Done false
13292 4345 Acked Write 2 Done Sent Done false
13293 4346 Acked Write 1 Done Sent Done false
13294 4347 Acked Read 7 Done Sent Done false
13295 4348 Acked Read 5 Done Sent Done false
13296 4349 Acked Read 4 Done Sent Done false
13297 4350 Acked Write 1 Done Sent Done false
13298 4351 Acked Read 9 Done Sent Done false
13299 4352 Acked Write 10 Done Sent Done false
13300 4353 Acked Read 7 Done Sent Done false
13301 4354 Acked Write 7 Done Sent Done false
13302 4355 Acked Read 7 Done Sent Done false
13303 4356 Acked Write 10 Done Sent Done false
13304 4357 Acked Flush 0 Done Sent Done false
13305 4358 Acked Write 6 Done Sent Done false
13306 4359 Acked Flush 0 Done Sent Done false
13307 4360 Acked Read 8 Done Sent Done false
13308 4361 Acked Write 1 Done Sent Done false
13309 4362 Acked Read 9 Done Sent Done false
13310 4363 Acked Read 4 Done Sent Done false
13311 4364 Acked Write 8 Done Sent Done false
13312 4365 Acked Write 9 Done Sent Done false
13313 4366 Acked Read 3 Done Sent Done false
13314 4367 Acked Read 1 Done Sent Done false
13315 4368 Acked Flush 0 Done Sent Done false
13316 4369 Acked Read 2 Done Sent Done false
13317 4370 Acked Write 6 Done Sent Done false
13318 4371 Acked Read 9 Done Sent Done false
13319 4372 Acked Read 10 Done Sent Done false
13320 4373 Acked Read 4 Done Sent Done false
13321 4374 Acked Write 2 Done Sent Done false
13322 4375 Acked Flush 0 Done Sent Done false
13323 4376 Acked Write 3 Done Sent Done false
13324 4377 Acked Write 9 Done Sent Done false
13325 4378 Acked Write 10 Done Sent Done false
13326 4379 Acked Write 10 Done Sent Done false
13327 4380 Acked Read 9 Done Sent Done false
13328 4381 Acked Read 4 Done Sent Done false
13329 4382 Acked Read 9 Done Sent Done false
13330 4383 Acked Write 7 Done Sent Done false
13331 4384 Acked Write 10 Done Sent Done false
13332 4385 Acked Write 4 Done Sent Done false
13333 4386 Acked Flush 0 Done Sent Done false
13334 4387 Acked Write 5 Done Sent Done false
13335 4388 Acked Read 7 Done Sent Done false
13336 4389 Acked Write 7 Done Sent Done false
13337 4390 Acked Read 3 Done Sent Done false
13338 4391 Acked Write 5 Done Sent Done false
13339 4392 Acked Read 1 Done Sent Done false
13340 4393 Acked Read 10 Done Sent Done false
13341 4394 Acked Flush 0 Done Sent Done false
13342 4395 Acked Read 2 Done Sent Done false
13343 4396 Acked Read 8 Done Sent Done false
13344 4397 Acked Write 5 Done Sent Done false
13345 4398 Acked Read 6 Done Sent Done false
13346 4399 Acked Read 5 Done Sent Done false
13347 4400 Acked Write 1 Done Sent Done false
13348 4401 Acked Read 7 Done Sent Done false
13349 4402 Acked Read 6 Done Sent Done false
13350 4403 Acked Write 7 Done Sent Done false
13351 4404 Acked Read 8 Done Sent Done false
13352 4405 Acked Read 3 Done Sent Done false
13353 4406 Acked Read 4 Done Sent Done false
13354 4407 Acked Write 3 Done Sent Done false
13355 4408 Acked Read 2 Done Sent Done false
13356 4409 Acked Write 4 Done Sent Done false
13357 4410 Acked Write 10 Done Sent Done false
13358 4411 Acked Read 6 Done Sent Done false
13359 4412 Acked Read 6 Done Sent Done false
13360 4413 Acked Read 10 Done Sent Done false
13361 4414 Acked Write 5 Done Sent Done false
13362 4415 Acked Read 4 Done Sent Done false
13363 4416 Acked Flush 0 Done Sent Done false
13364 4417 Acked Write 8 Done Sent Done false
13365 4418 Acked Read 9 Done Sent Done false
13366 4419 Acked Read 2 Done Sent Done false
13367 4420 Acked Write 3 Done Sent Done false
13368 4421 Acked Read 4 Done Sent Done false
13369 4422 Acked Write 7 Done Sent Done false
13370 4423 Acked Read 10 Done Sent Done false
13371 4424 Acked Write 7 Done Sent Done false
13372 4425 Acked Write 5 Done Sent Done false
13373 4426 Acked Write 9 Done Sent Done false
13374 4427 Acked Read 10 Done Sent Done false
13375 4428 Acked Write 3 Done Sent Done false
13376 4429 Acked Write 10 Done Sent Done false
13377 4430 Acked Read 3 Done Sent Done false
13378 4431 Acked Write 3 Done Sent Done false
13379 4432 Acked Read 5 Done Sent Done false
13380 4433 Acked Write 4 Done Sent Done false
13381 4434 Acked Write 6 Done Sent Done false
13382 4435 Acked Read 2 Done Sent Done false
13383 4436 Acked Write 3 Done Sent Done false
13384 4437 Acked Write 5 Done Sent Done false
13385 4438 Acked Read 1 Done Sent Done false
13386 4439 Acked Read 9 Done Sent Done false
13387 4440 Acked Write 1 Done Sent Done false
13388 4441 Acked Flush 0 Done Sent Done false
13389 4442 Acked Flush 0 Done Sent Done false
13390 4443 Acked Write 9 Done Sent Done false
13391 4444 Acked Read 4 Done Sent Done false
13392 4445 Acked Write 4 Done Sent Done false
13393 4446 Acked Read 3 Done Sent Done false
13394 4447 Acked Write 3 Done Sent Done false
13395 4448 Acked Write 9 Done Sent Done false
13396 4449 Acked Write 9 Done Sent Done false
13397 4450 Acked Write 10 Done Sent Done false
13398 4451 Acked Read 2 Done Sent Done false
13399 4452 Acked Read 9 Done Sent Done false
13400 4453 Acked Write 3 Done Sent Done false
13401 4454 Acked Read 4 Done Sent Done false
13402 4455 Acked Read 9 Done Sent Done false
13403 4456 Acked Read 8 Done Sent Done false
13404 4457 Acked Write 10 Done Sent Done false
13405 4458 Acked Write 8 Done Sent Done false
13406 4459 Acked Write 9 Done Sent Done false
13407 4460 Acked Read 1 Done Sent Done false
13408 4461 Acked Write 8 Done Sent Done false
13409 4462 Acked Read 4 Done Sent Done false
13410 4463 Acked Write 2 Done Sent Done false
13411 4464 Acked Flush 0 Done Sent Done false
13412 4465 Acked Read 7 Done Sent Done false
13413 4466 Acked Flush 0 Done Sent Done false
13414 4467 Acked Flush 0 Done Sent Done false
13415 4468 Acked Flush 0 Done Sent Done false
13416 4469 Acked Write 3 Done Sent Done false
13417 4470 Acked Read 8 Done Sent Done false
13418 4471 Acked Write 7 Done Sent Done false
13419 4472 Acked Write 1 Done Sent Done false
13420 4473 Acked Read 5 Done Sent Done false
13421 4474 Acked Read 4 Done Sent Done false
13422 4475 Acked Flush 0 Done Sent Done false
13423 4476 Acked Read 2 Done Sent Done false
13424 4477 Acked Read 4 Done Sent Done false
13425 4478 Acked Write 3 Done Sent Done false
13426 4479 Acked Write 6 Done Sent Done false
13427 4480 Acked Write 8 Done Sent Done false
13428 4481 Acked Write 9 Done Sent Done false
13429 4482 Acked Write 6 Done Sent Done false
13430 4483 Acked Read 8 Done Sent Done false
13431 4484 Acked Read 3 Done Sent Done false
13432 4485 Acked Write 4 Done Sent Done false
13433 4486 Acked Flush 0 Done Sent Done false
13434 4487 Acked Write 4 Done Sent Done false
13435 4488 Acked Read 1 Done Sent Done false
13436 4489 Acked Read 10 Done Sent Done false
13437 4490 Acked Read 10 Done Sent Done false
13438 4491 Acked Write 6 Done Sent Done false
13439 4492 Acked Read 10 Done Sent Done false
13440 4493 Acked Write 10 Done Sent Done false
13441 4494 Acked Read 8 Done Sent Done false
13442 4495 Acked Read 1 Done Sent Done false
13443 4496 Acked Read 6 Done Sent Done false
13444 4497 Acked Write 4 Done Sent Done false
13445 4498 Acked Write 2 Done Sent Done false
13446 4499 Acked Read 5 Done Sent Done false
13447 4500 Acked Write 6 Done Sent Done false
13448 4501 Acked Write 4 Done Sent Done false
13449 4502 Acked Read 5 Done Sent Done false
13450 4503 Acked Flush 0 Done Sent Done false
13451 4504 Acked Read 6 Done Sent Done false
13452 4505 Acked Write 6 Done Sent Done false
13453 4506 Acked Read 6 Done Sent Done false
13454 4507 Acked Read 6 Done Sent Done false
13455 4508 Acked Write 2 Done Sent Done false
13456 4509 Acked Write 10 Done Sent Done false
13457 4510 Acked Flush 0 Done Sent Done false
13458 4511 Acked Write 9 Done Sent Done false
13459 4512 Acked Write 1 Done Sent Done false
13460 4513 Acked Write 5 Done Sent Done false
13461 4514 Acked Read 5 Done Sent Done false
13462 4515 Acked Flush 0 Done Sent Done false
13463 4516 Acked Read 8 Done Sent Done false
13464 4517 Acked Write 1 Done Sent Done false
13465 4518 Acked Read 5 Done Sent Done false
13466 4519 Acked Read 3 Done Sent Done false
13467 4520 Acked Read 7 Done Sent Done false
13468 4521 Acked Write 9 Done Sent Done false
13469 4522 Acked Read 9 Done Sent Done false
13470 4523 Acked Write 5 Done Sent Done false
13471 4524 Acked Write 3 Done Sent Done false
13472 4525 Acked Read 10 Done Sent Done false
13473 4526 Acked Flush 0 Done Sent Done false
13474 4527 Acked Write 7 Done Sent Done false
13475 4528 Acked Read 2 Done Sent Done false
13476 4529 Acked Read 3 Done Sent Done false
13477 4530 Acked Read 3 Done Sent Done false
13478 4531 Acked Read 7 Done Sent Done false
13479 4532 Acked Read 2 Done Sent Done false
13480 4533 Acked Write 2 Done Sent Done false
13481 4534 Acked Read 4 Done Sent Done false
13482 4535 Acked Read 4 Done Sent Done false
13483 4536 Acked Read 4 Done Sent Done false
13484 4537 Acked Write 6 Done Sent Done false
13485 4538 Acked Read 8 Done Sent Done false
13486 4539 Acked Read 8 Done Sent Done false
13487 4540 Acked Flush 0 Done Sent Done false
13488 4541 Acked Flush 0 Done Sent Done false
13489 4542 Acked Read 8 Done Sent Done false
13490 4543 Acked Flush 0 Done Sent Done false
13491 4544 Acked Write 8 Done Sent Done false
13492 4545 Acked Flush 0 Done Sent Done false
13493 4546 Acked Flush 0 Done Sent Done false
13494 4547 Acked Read 8 Done Sent Done false
13495 4548 Acked Write 2 Done Sent Done false
13496 4549 Acked Write 9 Done Sent Done false
13497 4550 Acked Write 9 Done Sent Done false
13498 4551 Acked Write 9 Done Sent Done false
13499 4552 Acked Read 5 Done Sent Done false
13500 4553 Acked Read 3 Done Sent Done false
13501 4554 Acked Read 7 Done Sent Done false
13502 4555 Acked Write 3 Done Sent Done false
13503 4556 Acked Flush 0 Done Sent Done false
13504 4557 Acked Read 5 Done Sent Done false
13505 4558 Acked Write 8 Done Sent Done false
13506 4559 Acked Write 6 Done Sent Done false
13507 4560 Acked Flush 0 Done Sent Done false
13508 4561 Acked Read 2 Done Sent Done false
13509 4562 Acked Flush 0 Done Sent Done false
13510 4563 Acked Write 8 Done Sent Done false
13511 4564 Acked Write 10 Done Sent Done false
13512 4565 Acked Read 4 Done Sent Done false
13513 4566 Acked Read 2 Done Sent Done false
13514 4567 Acked Write 7 Done Sent Done false
13515 4568 Acked Flush 0 Done Sent Done false
13516 4569 Acked Read 2 Done Sent Done false
13517 4570 Acked Read 5 Done Sent Done false
13518 4571 Acked Read 10 Done Sent Done false
13519 4572 Acked Read 4 Done Sent Done false
13520 4573 Acked Read 1 Done Sent Done false
13521 4574 Acked Flush 0 Done Sent Done false
13522 4575 Acked Write 3 Done Sent Done false
13523 4576 Acked Write 5 Done Sent Done false
13524 4577 Acked Read 6 Done Sent Done false
13525 4578 Acked Write 7 Done Sent Done false
13526 4579 Acked Write 5 Done Sent Done false
13527 4580 Acked Read 1 Done Sent Done false
13528 4581 Acked Write 6 Done Sent Done false
13529 4582 Acked Flush 0 Done Sent Done false
13530 4583 Acked Read 7 Done Sent Done false
13531 4584 Acked Write 9 Done Sent Done false
13532 4585 Acked Write 6 Done Sent Done false
13533 4586 Acked Flush 0 Done Sent Done false
13534 4587 Acked Write 7 Done Sent Done false
13535 4588 Acked Flush 0 Done Sent Done false
13536 4589 Acked Read 8 Done Sent Done false
13537 4590 Acked Write 3 Done Sent Done false
13538 4591 Acked Read 9 Done Sent Done false
13539 4592 Acked Read 8 Done Sent Done false
13540 4593 Acked Write 9 Done Sent Done false
13541 4594 Acked Write 2 Done Sent Done false
13542 4595 Acked Read 9 Done Sent Done false
13543 4596 Acked Read 9 Done Sent Done false
13544 4597 Acked Write 8 Done Sent Done false
13545 4598 Acked Write 5 Done Sent Done false
13546 4599 Acked Read 6 Done Sent Done false
13547 4600 Acked Write 3 Done Sent Done false
13548 4601 Acked Write 10 Done Sent Done false
13549 4602 Acked Read 2 Done Sent Done false
13550 4603 Acked Read 2 Done Sent Done false
13551 4604 Acked Write 9 Done Sent Done false
13552 4605 Acked Write 7 Done Sent Done false
13553 4606 Acked Write 8 Done Sent Done false
13554 4607 Acked Write 8 Done Sent Done false
13555 4608 Acked Write 5 Done Sent Done false
13556 4609 Acked Read 8 Done Sent Done false
13557 4610 Acked Write 3 Done Sent Done false
13558 4611 Acked Flush 0 Done Sent Done false
13559 4612 Acked Write 10 Done Sent Done false
13560 4613 Acked Flush 0 Done Sent Done false
13561 4614 Acked Flush 0 Done Sent Done false
13562 4615 Acked Write 9 Done Sent Done false
13563 4616 Acked Read 2 Done Sent Done false
13564 4617 Acked Write 3 Done Sent Done false
13565 4618 Acked Read 3 Done Sent Done false
13566 4619 Acked Write 7 Done Sent Done false
13567 4620 Acked Write 6 Done Sent Done false
13568 4621 Acked Read 4 Done Sent Done false
13569 STATES DS:0 DS:1 DS:2 TOTAL
13570 Sent 0 301 0 301
13571 Done 301 0 301 602
13572 Skipped 0 0 0 0
13573 Error 0 0 0 0
13574 Last Flush: Some(JobId(4614)) Some(JobId(4320)) Some(JobId(4614))
13575 Downstairs last five completed: 4320 4319 4318 4317 4316
13576 Upstairs last five completed: 4621 4620 4619 4618 4617
13577 CLIENT: Up:0 ds:301 act:2
135782025-03-17T22:19:15.209ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
135792025-03-17T22:19:15.975ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
135802025-03-17T22:19:16.875ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
135812025-03-17T22:19:17.877ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
13582 [8820] got start action
13583 [8820] I am going to restart
13584 Starting downstairs at port 8820
13585 Make output file at "/tmp/dsc/downstairs-8820.txt"
13586 [8820][1] reports Starting
13587 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1218)
13588 [8820][1] reports Running
13589 ----------------------------------------------------------------
13590 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13591 States: Active Offline Active
13592 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
13593 4321 Acked Read 9 Done Sent Done false
13594 4322 Acked Write 5 Done Sent Done false
13595 4323 Acked Read 5 Done Sent Done false
13596 4324 Acked Read 1 Done Sent Done false
13597 4325 Acked Read 10 Done Sent Done false
13598 4326 Acked Read 2 Done Sent Done false
13599 4327 Acked Read 10 Done Sent Done false
13600 4328 Acked Read 10 Done Sent Done false
13601 4329 Acked Read 2 Done Sent Done false
13602 4330 Acked Read 2 Done Sent Done false
13603 4331 Acked Read 4 Done Sent Done false
13604 4332 Acked Read 1 Done Sent Done false
13605 4333 Acked Read 9 Done Sent Done false
13606 4334 Acked Write 6 Done Sent Done false
13607 4335 Acked Read 8 Done Sent Done false
13608 4336 Acked Read 4 Done Sent Done false
13609 4337 Acked Read 4 Done Sent Done false
13610 4338 Acked Read 6 Done Sent Done false
13611 4339 Acked Write 5 Done Sent Done false
13612 4340 Acked Write 4 Done Sent Done false
13613 4341 Acked Read 4 Done Sent Done false
13614 4342 Acked Write 7 Done Sent Done false
13615 4343 Acked Read 4 Done Sent Done false
13616 4344 Acked Flush 0 Done Sent Done false
13617 4345 Acked Write 2 Done Sent Done false
13618 4346 Acked Write 1 Done Sent Done false
13619 4347 Acked Read 7 Done Sent Done false
13620 4348 Acked Read 5 Done Sent Done false
13621 4349 Acked Read 4 Done Sent Done false
13622 4350 Acked Write 1 Done Sent Done false
13623 4351 Acked Read 9 Done Sent Done false
13624 4352 Acked Write 10 Done Sent Done false
13625 4353 Acked Read 7 Done Sent Done false
13626 4354 Acked Write 7 Done Sent Done false
13627 4355 Acked Read 7 Done Sent Done false
13628 4356 Acked Write 10 Done Sent Done false
13629 4357 Acked Flush 0 Done Sent Done false
13630 4358 Acked Write 6 Done Sent Done false
13631 4359 Acked Flush 0 Done Sent Done false
13632 4360 Acked Read 8 Done Sent Done false
13633 4361 Acked Write 1 Done Sent Done false
13634 4362 Acked Read 9 Done Sent Done false
13635 4363 Acked Read 4 Done Sent Done false
13636 4364 Acked Write 8 Done Sent Done false
13637 4365 Acked Write 9 Done Sent Done false
13638 4366 Acked Read 3 Done Sent Done false
13639 4367 Acked Read 1 Done Sent Done false
13640 4368 Acked Flush 0 Done Sent Done false
13641 4369 Acked Read 2 Done Sent Done false
13642 4370 Acked Write 6 Done Sent Done false
13643 4371 Acked Read 9 Done Sent Done false
13644 4372 Acked Read 10 Done Sent Done false
13645 4373 Acked Read 4 Done Sent Done false
13646 4374 Acked Write 2 Done Sent Done false
13647 4375 Acked Flush 0 Done Sent Done false
13648 4376 Acked Write 3 Done Sent Done false
13649 4377 Acked Write 9 Done Sent Done false
13650 4378 Acked Write 10 Done Sent Done false
13651 4379 Acked Write 10 Done Sent Done false
13652 4380 Acked Read 9 Done Sent Done false
13653 4381 Acked Read 4 Done Sent Done false
13654 4382 Acked Read 9 Done Sent Done false
13655 4383 Acked Write 7 Done Sent Done false
13656 4384 Acked Write 10 Done Sent Done false
13657 4385 Acked Write 4 Done Sent Done false
13658 4386 Acked Flush 0 Done Sent Done false
13659 4387 Acked Write 5 Done Sent Done false
13660 4388 Acked Read 7 Done Sent Done false
13661 4389 Acked Write 7 Done Sent Done false
13662 4390 Acked Read 3 Done Sent Done false
13663 4391 Acked Write 5 Done Sent Done false
13664 4392 Acked Read 1 Done Sent Done false
13665 4393 Acked Read 10 Done Sent Done false
13666 4394 Acked Flush 0 Done Sent Done false
13667 4395 Acked Read 2 Done Sent Done false
13668 4396 Acked Read 8 Done Sent Done false
13669 4397 Acked Write 5 Done Sent Done false
13670 4398 Acked Read 6 Done Sent Done false
13671 4399 Acked Read 5 Done Sent Done false
13672 4400 Acked Write 1 Done Sent Done false
13673 4401 Acked Read 7 Done Sent Done false
13674 4402 Acked Read 6 Done Sent Done false
13675 4403 Acked Write 7 Done Sent Done false
13676 4404 Acked Read 8 Done Sent Done false
13677 4405 Acked Read 3 Done Sent Done false
13678 4406 Acked Read 4 Done Sent Done false
13679 4407 Acked Write 3 Done Sent Done false
13680 4408 Acked Read 2 Done Sent Done false
13681 4409 Acked Write 4 Done Sent Done false
13682 4410 Acked Write 10 Done Sent Done false
13683 4411 Acked Read 6 Done Sent Done false
13684 4412 Acked Read 6 Done Sent Done false
13685 4413 Acked Read 10 Done Sent Done false
13686 4414 Acked Write 5 Done Sent Done false
13687 4415 Acked Read 4 Done Sent Done false
13688 4416 Acked Flush 0 Done Sent Done false
13689 4417 Acked Write 8 Done Sent Done false
13690 4418 Acked Read 9 Done Sent Done false
13691 4419 Acked Read 2 Done Sent Done false
13692 4420 Acked Write 3 Done Sent Done false
13693 4421 Acked Read 4 Done Sent Done false
13694 4422 Acked Write 7 Done Sent Done false
13695 4423 Acked Read 10 Done Sent Done false
13696 4424 Acked Write 7 Done Sent Done false
13697 4425 Acked Write 5 Done Sent Done false
13698 4426 Acked Write 9 Done Sent Done false
13699 4427 Acked Read 10 Done Sent Done false
13700 4428 Acked Write 3 Done Sent Done false
13701 4429 Acked Write 10 Done Sent Done false
13702 4430 Acked Read 3 Done Sent Done false
13703 4431 Acked Write 3 Done Sent Done false
13704 4432 Acked Read 5 Done Sent Done false
13705 4433 Acked Write 4 Done Sent Done false
13706 4434 Acked Write 6 Done Sent Done false
13707 4435 Acked Read 2 Done Sent Done false
13708 4436 Acked Write 3 Done Sent Done false
13709 4437 Acked Write 5 Done Sent Done false
13710 4438 Acked Read 1 Done Sent Done false
13711 4439 Acked Read 9 Done Sent Done false
13712 4440 Acked Write 1 Done Sent Done false
13713 4441 Acked Flush 0 Done Sent Done false
13714 4442 Acked Flush 0 Done Sent Done false
13715 4443 Acked Write 9 Done Sent Done false
13716 4444 Acked Read 4 Done Sent Done false
13717 4445 Acked Write 4 Done Sent Done false
13718 4446 Acked Read 3 Done Sent Done false
13719 4447 Acked Write 3 Done Sent Done false
13720 4448 Acked Write 9 Done Sent Done false
13721 4449 Acked Write 9 Done Sent Done false
13722 4450 Acked Write 10 Done Sent Done false
13723 4451 Acked Read 2 Done Sent Done false
13724 4452 Acked Read 9 Done Sent Done false
13725 4453 Acked Write 3 Done Sent Done false
13726 4454 Acked Read 4 Done Sent Done false
13727 4455 Acked Read 9 Done Sent Done false
13728 4456 Acked Read 8 Done Sent Done false
13729 4457 Acked Write 10 Done Sent Done false
13730 4458 Acked Write 8 Done Sent Done false
13731 4459 Acked Write 9 Done Sent Done false
13732 4460 Acked Read 1 Done Sent Done false
13733 4461 Acked Write 8 Done Sent Done false
13734 4462 Acked Read 4 Done Sent Done false
13735 4463 Acked Write 2 Done Sent Done false
13736 4464 Acked Flush 0 Done Sent Done false
13737 4465 Acked Read 7 Done Sent Done false
13738 4466 Acked Flush 0 Done Sent Done false
13739 4467 Acked Flush 0 Done Sent Done false
13740 4468 Acked Flush 0 Done Sent Done false
13741 4469 Acked Write 3 Done Sent Done false
13742 4470 Acked Read 8 Done Sent Done false
13743 4471 Acked Write 7 Done Sent Done false
13744 4472 Acked Write 1 Done Sent Done false
13745 4473 Acked Read 5 Done Sent Done false
13746 4474 Acked Read 4 Done Sent Done false
13747 4475 Acked Flush 0 Done Sent Done false
13748 4476 Acked Read 2 Done Sent Done false
13749 4477 Acked Read 4 Done Sent Done false
13750 4478 Acked Write 3 Done Sent Done false
13751 4479 Acked Write 6 Done Sent Done false
13752 4480 Acked Write 8 Done Sent Done false
13753 4481 Acked Write 9 Done Sent Done false
13754 4482 Acked Write 6 Done Sent Done false
13755 4483 Acked Read 8 Done Sent Done false
13756 4484 Acked Read 3 Done Sent Done false
13757 4485 Acked Write 4 Done Sent Done false
13758 4486 Acked Flush 0 Done Sent Done false
13759 4487 Acked Write 4 Done Sent Done false
13760 4488 Acked Read 1 Done Sent Done false
13761 4489 Acked Read 10 Done Sent Done false
13762 4490 Acked Read 10 Done Sent Done false
13763 4491 Acked Write 6 Done Sent Done false
13764 4492 Acked Read 10 Done Sent Done false
13765 4493 Acked Write 10 Done Sent Done false
13766 4494 Acked Read 8 Done Sent Done false
13767 4495 Acked Read 1 Done Sent Done false
13768 4496 Acked Read 6 Done Sent Done false
13769 4497 Acked Write 4 Done Sent Done false
13770 4498 Acked Write 2 Done Sent Done false
13771 4499 Acked Read 5 Done Sent Done false
13772 4500 Acked Write 6 Done Sent Done false
13773 4501 Acked Write 4 Done Sent Done false
13774 4502 Acked Read 5 Done Sent Done false
13775 4503 Acked Flush 0 Done Sent Done false
13776 4504 Acked Read 6 Done Sent Done false
13777 4505 Acked Write 6 Done Sent Done false
13778 4506 Acked Read 6 Done Sent Done false
13779 4507 Acked Read 6 Done Sent Done false
13780 4508 Acked Write 2 Done Sent Done false
13781 4509 Acked Write 10 Done Sent Done false
13782 4510 Acked Flush 0 Done Sent Done false
13783 4511 Acked Write 9 Done Sent Done false
13784 4512 Acked Write 1 Done Sent Done false
13785 4513 Acked Write 5 Done Sent Done false
13786 4514 Acked Read 5 Done Sent Done false
13787 4515 Acked Flush 0 Done Sent Done false
13788 4516 Acked Read 8 Done Sent Done false
13789 4517 Acked Write 1 Done Sent Done false
13790 4518 Acked Read 5 Done Sent Done false
13791 4519 Acked Read 3 Done Sent Done false
13792 4520 Acked Read 7 Done Sent Done false
13793 4521 Acked Write 9 Done Sent Done false
13794 4522 Acked Read 9 Done Sent Done false
13795 4523 Acked Write 5 Done Sent Done false
13796 4524 Acked Write 3 Done Sent Done false
13797 4525 Acked Read 10 Done Sent Done false
13798 4526 Acked Flush 0 Done Sent Done false
13799 4527 Acked Write 7 Done Sent Done false
13800 4528 Acked Read 2 Done Sent Done false
13801 4529 Acked Read 3 Done Sent Done false
13802 4530 Acked Read 3 Done Sent Done false
13803 4531 Acked Read 7 Done Sent Done false
13804 4532 Acked Read 2 Done Sent Done false
13805 4533 Acked Write 2 Done Sent Done false
13806 4534 Acked Read 4 Done Sent Done false
13807 4535 Acked Read 4 Done Sent Done false
13808 4536 Acked Read 4 Done Sent Done false
13809 4537 Acked Write 6 Done Sent Done false
13810 4538 Acked Read 8 Done Sent Done false
13811 4539 Acked Read 8 Done Sent Done false
13812 4540 Acked Flush 0 Done Sent Done false
13813 4541 Acked Flush 0 Done Sent Done false
13814 4542 Acked Read 8 Done Sent Done false
13815 4543 Acked Flush 0 Done Sent Done false
13816 4544 Acked Write 8 Done Sent Done false
13817 4545 Acked Flush 0 Done Sent Done false
13818 4546 Acked Flush 0 Done Sent Done false
13819 4547 Acked Read 8 Done Sent Done false
13820 4548 Acked Write 2 Done Sent Done false
13821 4549 Acked Write 9 Done Sent Done false
13822 4550 Acked Write 9 Done Sent Done false
13823 4551 Acked Write 9 Done Sent Done false
13824 4552 Acked Read 5 Done Sent Done false
13825 4553 Acked Read 3 Done Sent Done false
13826 4554 Acked Read 7 Done Sent Done false
13827 4555 Acked Write 3 Done Sent Done false
13828 4556 Acked Flush 0 Done Sent Done false
13829 4557 Acked Read 5 Done Sent Done false
13830 4558 Acked Write 8 Done Sent Done false
13831 4559 Acked Write 6 Done Sent Done false
13832 4560 Acked Flush 0 Done Sent Done false
13833 4561 Acked Read 2 Done Sent Done false
13834 4562 Acked Flush 0 Done Sent Done false
13835 4563 Acked Write 8 Done Sent Done false
13836 4564 Acked Write 10 Done Sent Done false
13837 4565 Acked Read 4 Done Sent Done false
13838 4566 Acked Read 2 Done Sent Done false
13839 4567 Acked Write 7 Done Sent Done false
13840 4568 Acked Flush 0 Done Sent Done false
13841 4569 Acked Read 2 Done Sent Done false
13842 4570 Acked Read 5 Done Sent Done false
13843 4571 Acked Read 10 Done Sent Done false
13844 4572 Acked Read 4 Done Sent Done false
13845 4573 Acked Read 1 Done Sent Done false
13846 4574 Acked Flush 0 Done Sent Done false
13847 4575 Acked Write 3 Done Sent Done false
13848 4576 Acked Write 5 Done Sent Done false
13849 4577 Acked Read 6 Done Sent Done false
13850 4578 Acked Write 7 Done Sent Done false
13851 4579 Acked Write 5 Done Sent Done false
13852 4580 Acked Read 1 Done Sent Done false
13853 4581 Acked Write 6 Done Sent Done false
13854 4582 Acked Flush 0 Done Sent Done false
13855 4583 Acked Read 7 Done Sent Done false
13856 4584 Acked Write 9 Done Sent Done false
13857 4585 Acked Write 6 Done Sent Done false
13858 4586 Acked Flush 0 Done Sent Done false
13859 4587 Acked Write 7 Done Sent Done false
13860 4588 Acked Flush 0 Done Sent Done false
13861 4589 Acked Read 8 Done Sent Done false
13862 4590 Acked Write 3 Done Sent Done false
13863 4591 Acked Read 9 Done Sent Done false
13864 4592 Acked Read 8 Done Sent Done false
13865 4593 Acked Write 9 Done Sent Done false
13866 4594 Acked Write 2 Done Sent Done false
13867 4595 Acked Read 9 Done Sent Done false
13868 4596 Acked Read 9 Done Sent Done false
13869 4597 Acked Write 8 Done Sent Done false
13870 4598 Acked Write 5 Done Sent Done false
13871 4599 Acked Read 6 Done Sent Done false
13872 4600 Acked Write 3 Done Sent Done false
13873 4601 Acked Write 10 Done Sent Done false
13874 4602 Acked Read 2 Done Sent Done false
13875 4603 Acked Read 2 Done Sent Done false
13876 4604 Acked Write 9 Done Sent Done false
13877 4605 Acked Write 7 Done Sent Done false
13878 4606 Acked Write 8 Done Sent Done false
13879 4607 Acked Write 8 Done Sent Done false
13880 4608 Acked Write 5 Done Sent Done false
13881 4609 Acked Read 8 Done Sent Done false
13882 4610 Acked Write 3 Done Sent Done false
13883 4611 Acked Flush 0 Done Sent Done false
13884 4612 Acked Write 10 Done Sent Done false
13885 4613 Acked Flush 0 Done Sent Done false
13886 4614 Acked Flush 0 Done Sent Done false
13887 4615 Acked Write 9 Done Sent Done false
13888 4616 Acked Read 2 Done Sent Done false
13889 4617 Acked Write 3 Done Sent Done false
13890 4618 Acked Read 3 Done Sent Done false
13891 4619 Acked Write 7 Done Sent Done false
13892 4620 Acked Write 6 Done Sent Done false
13893 4621 Acked Read 4 Done Sent Done false
13894 STATES DS:0 DS:1 DS:2 TOTAL
13895 Sent 0 301 0 301
13896 Done 301 0 301 602
13897 Skipped 0 0 0 0
13898 Error 0 0 0 0
13899 Last Flush: Some(JobId(4614)) Some(JobId(4320)) Some(JobId(4614))
13900 Downstairs last five completed: 4320 4319 4318 4317 4316
13901 Upstairs last five completed: 4621 4620 4619 4618 4617
13902 CLIENT: Up:0 ds:301 act:2
139032025-03-17T22:19:18.878ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
139042025-03-17T22:19:19.987ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
139052025-03-17T22:19:19.987ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:62305) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
139062025-03-17T22:19:19.988ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
139072025-03-17T22:19:19.989ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
139082025-03-17T22:19:19.989ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
139092025-03-17T22:19:19.989ZINFOcrucible: send last flush ID to this DS: Some(JobId(4320)) = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
139102025-03-17T22:19:19.989ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
139112025-03-17T22:19:19.989ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(4320)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
139122025-03-17T22:19:19.989ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(4320)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
13913 ----------------------------------------------------------------
13914 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
13915 Downstairs last five completed: 4622 4621 4620 4619 4618
13916 Upstairs last five completed: 4622 4621 4620 4619 4618
13917 CLIENT: Up:0 ds:0 act:3
13918 Replay: All jobs finished
13919 Main task has work to do, go find it
13920 got dsc Stop(0)
13921 stop 0
13922 Mar 17 22:19:22.237 INFO request completed, latency_us: 493, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 810bb374-2aae-4e86-81f5-bcb407517c09, remote_addr: 127.0.0.1:60423, local_addr[8810] Got stop action so:false kr:false
13923 : 127.0.0.1:9998
13924 Mar 17 22:19:22.238 INFO request completed, latency_us: 147, response_code: 200, uri: /state/cid/0, method: GET, req_id: 2ffd7395-a8f5-4cc4-891a-7855525567b1, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
139252025-03-17T22:19:22.238ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
139262025-03-17T22:19:22.238ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
139272025-03-17T22:19:22.238ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
139282025-03-17T22:19:22.238ZINFOcrucible: client task is exiting = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
13929 [8810] Exited with: ExitStatus(unix_wait_status(9))
13930 [8810][0] reports Exit
13931 Mar 17 22:19:26.239 INFO request completed, latency_us: 213, response_code: 200, uri: /state/cid/0, method: GET, req_id: b5000cb3-42dd-4ba6-8f87-5d0968cbc65e, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
13932 001/300 Read block 906 len 36864
13933 002/300 Read block 503 len 36864
13934 003/300 Write block 261 len 4096 data: 6
13935 004/300 Read block 1084 len 16384
13936 005/300 Write block 2085 len 12288 data: 2 2 2
13937 006/300 Read block 866 len 20480
13938 007/300 Write block 2312 len 24576 data: 3 3 2 2 2 2
13939 008/300 Read block 4863 len 32768
13940 009/300 Read block 907 len 12288
13941 010/300 Read block 1274 len 12288
13942 011/300 Write block 2241 len 16384 data: 2 3 3 4
13943 012/300 Write block 613 len 40960 data: 3 4 4 4 4 2 2 2 2 2
13944 013/300 Read block 3111 len 28672
13945 014/300 Read block 609 len 36864
13946 015/300 Read block 817 len 20480
13947 016/300 Read block 317 len 24576
13948 017/300 Write block 857 len 40960 data: 3 4 3 2 2 2 2 2 3 2
13949 018/300 Read block 624 len 40960
13950 019/300 Write block 470 len 12288 data: 3 3 3
13951 020/300 Write block 4531 len 32768 data: 4 4 4 3 3 3 3 3
13952 021/300 Read block 2440 len 28672
13953 022/300 Read block 3005 len 12288
13954 023/300 Write block 3523 len 28672 data: 2 3 3 3 3 2 2
13955 024/300 Write block 2419 len 36864 data: 2 2 3 3 3 3 3 3 4
13956 025/300 Read block 646 len 32768
13957 026/300 Read block 138 len 36864
13958 027/300 Write block 3825 len 16384 data: 4 5 5 6
13959 028/300 Read block 2674 len 12288
13960 029/300 Write block 3227 len 32768 data: 4 3 3 4 5 5 5 5
13961 030/300 Write block 4409 len 4096 data: 5
13962 031/300 Write block 1978 len 4096 data: 3
13963 032/300 Write block 4241 len 8192 data: 3 3
13964 033/300 Read block 3732 len 16384
13965 034/300 Write block 4122 len 24576 data: 2 2 2 2 2 2
13966 035/300 Read block 992 len 24576
13967 036/300 Write block 4156 len 32768 data: 2 2 3 3 4 4 4 4
13968 037/300 Write block 4644 len 8192 data: 3 3
13969 038/300 Read block 2586 len 4096
13970 039/300 Write block 4366 len 12288 data: 4 5 4
13971 040/300 Write block 3 len 32768 data: 3 3 3 3 4 4 3 4
13972 041/300 Read block 2007 len 32768
13973 042/300 Write block 1055 len 28672 data: 2 2 2 2 2 2 2
13974 043/300 Write block 563 len 28672 data: 6 6 6 5 2 2 2
13975 044/300 Read block 586 len 24576
13976 045/300 Write block 1094 len 36864 data: 4 4 4 4 5 5 5 5 7
13977 046/300 Read block 3504 len 32768
13978 047/300 Read block 3949 len 12288
13979 048/300 Read block 2255 len 32768
13980 049/300 Read block 3602 len 32768
13981 050/300 Write block 3415 len 24576 data: 2 4 5 5 4 4
13982 051/300 Read block 1921 len 16384
13983 052/300 Read block 4124 len 36864
13984 053/300 Write block 2708 len 4096 data: 2
13985 054/300 Write block 4856 len 8192 data: 3 4
13986 055/300 Read block 197 len 16384
13987 056/300 Read block 2274 len 8192
13988 057/300 Read block 293 len 36864
13989 058/300 Write block 4166 len 4096 data: 3
13990 059/300 Flush
13991 060/300 Write block 798 len 16384 data: 5 4 5 5
13992 061/300 Flush
13993 062/300 Write block 19 len 12288 data: 4 5 5
13994 063/300 Write block 3098 len 40960 data: 3 2 2 2 3 3 4 4 4 4
13995 064/300 Flush
13996 065/300 Read block 1339 len 20480
13997 066/300 Write block 3826 len 20480 data: 6 6 7 6 5
13998 067/300 Read block 653 len 36864
13999 068/300 Write block 4784 len 32768 data: 6 4 4 4 3 2 2 2
14000 069/300 Write block 3214 len 12288 data: 3 4 4
14001 070/300 Read block 4729 len 24576
14002 071/300 Flush
14003 072/300 Write block 372 len 24576 data: 4 4 4 4 4 3
14004 073/300 Write block 373 len 8192 data: 5 5
14005 074/300 Read block 3291 len 4096
14006 075/300 Read block 3023 len 40960
14007 076/300 Write block 684 len 12288 data: 5 4 4
14008 077/300 Read block 2149 len 40960
14009 078/300 Write block 4653 len 16384 data: 3 3 3 3
14010 079/300 Read block 3825 len 24576
14011 080/300 Write block 1026 len 40960 data: 3 4 4 5 4 4 5 6 7 5
14012 081/300 Read block 669 len 4096
14013 082/300 Flush
14014 083/300 Write block 1125 len 40960 data: 2 3 4 5 3 3 4 3 3 4
14015 084/300 Write block 831 len 24576 data: 4 4 4 4 4 3
14016 085/300 Read block 4972 len 28672
14017 086/300 Read block 1229 len 8192
14018 087/300 Write block 1031 len 20480 data: 5 6 7 8 6
14019 088/300 Write block 1611 len 28672 data: 3 3 4 3 3 3 3
14020 089/300 Read block 3581 len 16384
14021 090/300 Read block 3293 len 36864
14022 091/300 Read block 4566 len 24576
14023 092/300 Read block 2316 len 32768
14024 093/300 Read block 4817 len 12288
14025 094/300 Read block 558 len 8192
14026 095/300 Read block 405 len 40960
14027 096/300 Write block 4031 len 16384 data: 3 3 2 2
14028 097/300 Flush
14029 098/300 Read block 2947 len 4096
14030 099/300 Read block 846 len 28672
14031 100/300 Read block 1256 len 16384
14032 101/300 Read block 2455 len 16384
14033 102/300 Read block 570 len 8192
14034 103/300 Read block 2752 len 12288
14035 104/300 Read block 3421 len 32768
14036 105/300 Write block 4982 len 8192 data: 3 3
14037 106/300 Read block 420 len 24576
14038 107/300 Read block 2648 len 8192
14039 108/300 Read block 108 len 28672
14040 109/300 Read block 3194 len 32768
14041 110/300 Read block 4185 len 16384
14042 111/300 Write block 1138 len 20480 data: 3 3 3 2 2
14043 112/300 Read block 4837 len 4096
14044 113/300 Write block 104 len 28672 data: 3 3 2 2 2 2 2
14045 114/300 Write block 776 len 20480 data: 2 2 2 2 2
14046 115/300 Read block 3075 len 20480
14047 116/300 Read block 3660 len 32768
14048 117/300 Read block 2632 len 36864
14049 118/300 Read block 1521 len 40960
14050 119/300 Read block 4746 len 24576
14051 120/300 Read block 274 len 8192
14052 121/300 Read block 3201 len 28672
14053 122/300 Write block 2034 len 40960 data: 3 3 4 4 3 4 4 3 3 3
14054 123/300 Write block 1065 len 20480 data: 2 3 2 3 3
14055 124/300 Read block 3694 len 16384
14056 125/300 Write block 2274 len 32768 data: 3 3 3 3 5 6 6 6
14057 126/300 Write block 665 len 20480 data: 3 3 3 3 3
14058 127/300 Read block 731 len 16384
14059 128/300 Read block 4902 len 4096
14060 129/300 Write block 420 len 20480 data: 3 2 2 2 3
14061 130/300 Write block 3060 len 32768 data: 3 3 3 3 3 4 4 4
14062 131/300 Read block 3043 len 4096
14063 132/300 Write block 316 len 16384 data: 5 5 5 5
14064 133/300 Write block 2286 len 32768 data: 3 2 2 2 2 2 2 3
14065 134/300 Write block 2267 len 16384 data: 3 3 3 3
14066 135/300 Write block 3443 len 32768 data: 4 5 5 5 4 4 4 4
14067 136/300 Write block 399 len 40960 data: 4 4 4 3 3 3 3 3 3 3
14068 137/300 Read block 3412 len 20480
14069 138/300 Read block 1153 len 4096
14070 139/300 Read block 3303 len 40960
14071 140/300 Read block 1739 len 16384
14072 141/300 Write block 4891 len 40960 data: 3 3 3 2 3 3 3 3 4 5
14073 142/300 Read block 762 len 12288
14074 143/300 Write block 3231 len 36864 data: 6 6 6 6 5 6 6 7 7
14075 144/300 Flush
14076 145/300 Read block 997 len 4096
14077 146/300 Read block 717 len 8192
14078 147/300 Write block 3896 len 40960 data: 6 7 7 7 6 6 5 2 2 2
14079 148/300 Write block 491 len 40960 data: 4 4 5 5 4 4 5 4 3 3
14080 149/300 Read block 2827 len 40960
14081 150/300 Read block 458 len 16384
14082 151/300 Read block 2000 len 28672
14083 152/300 Read block 4682 len 28672
14084 153/300 Read block 2070 len 32768
14085 154/300 Flush
14086 155/300 Read block 4989 len 8192
14087 156/300 Write block 4867 len 16384 data: 4 3 2 4
14088 157/300 Flush
14089 158/300 Read block 765 len 24576
14090 159/300 Write block 4907 len 40960 data: 5 2 2 2 2 2 2 2 3 3
14091 160/300 Read block 3861 len 16384
14092 161/300 Write block 4285 len 8192 data: 3 3
14093 162/300 Write block 1755 len 24576 data: 2 3 3 2 3 3
14094 163/300 Write block 3984 len 4096 data: 3
14095 164/300 Write block 3213 len 24576 data: 3 4 5 5 4 4
14096 165/300 Write block 1753 len 28672 data: 4 4 3 4 4 3 4
14097 166/300 Read block 4449 len 12288
14098 167/300 Write block 1135 len 36864 data: 4 4 3 4 4 4 3 3 2
14099 168/300 Read block 4805 len 4096
14100 169/300 Read block 322 len 16384
14101 170/300 Read block 2882 len 20480
14102 171/300 Write block 2060 len 12288 data: 6 6 5
14103 172/300 Read block 2518 len 40960
14104 173/300 Flush
14105 174/300 Read block 130 len 24576
14106 175/300 Read block 359 len 4096
14107 176/300 Read block 67 len 24576
14108 177/300 Flush
14109 178/300 Write block 1656 len 32768 data: 2 2 2 2 3 3 3 3
14110 179/300 Write block 4404 len 12288 data: 5 4 5
14111 180/300 Write block 14 len 36864 data: 4 4 4 4 4 5 6 6 5
14112 181/300 Write block 701 len 12288 data: 3 2 6
14113 182/300 Write block 2192 len 20480 data: 3 3 3 3 2
14114 183/300 Read block 4619 len 12288
14115 184/300 Read block 2591 len 20480
14116 185/300 Read block 3800 len 12288
14117 186/300 Read block 4975 len 20480
14118 187/300 Write block 713 len 12288 data: 2 4 5
14119 188/300 Write block 4675 len 32768 data: 3 3 2 2 2 2 2 2
14120 189/300 Write block 407 len 8192 data: 4 4
14121 190/300 Write block 2376 len 20480 data: 2 4 4 4 5
14122 191/300 Write block 4972 len 16384 data: 2 2 2 2
14123 192/300 Flush
14124 193/300 Read block 1399 len 36864
14125 194/300 Read block 4618 len 16384
14126 195/300 Read block 4328 len 20480
14127 196/300 Write block 4606 len 24576 data: 7 7 8 7 7 6
14128 197/300 Read block 972 len 24576
14129 198/300 Write block 4794 len 4096 data: 3
14130 199/300 Read block 2470 len 24576
14131 200/300 Read block 4346 len 16384
14132 201/300 Write block 386 len 32768 data: 4 4 4 3 2 2 2 2
14133 202/300 Read block 453 len 32768
14134 203/300 Read block 2841 len 36864
14135 204/300 Write block 2406 len 4096 data: 4
14136 205/300 Write block 3997 len 24576 data: 2 3 3 3 3 4
14137 206/300 Write block 2963 len 24576 data: 5 5 5 5 6 6
14138 207/300 Write block 2716 len 16384 data: 2 2 2 2
14139 208/300 Read block 2724 len 20480
14140 209/300 Read block 86 len 12288
14141 210/300 Read block 2353 len 28672
14142 211/300 Read block 1312 len 28672
14143 212/300 Write block 1637 len 36864 data: 5 5 5 6 6 5 3 3 3
14144 213/300 Read block 2275 len 12288
14145 214/300 Read block 2947 len 32768
14146 215/300 Write block 4621 len 36864 data: 2 2 2 2 3 3 2 3 3
14147 216/300 Write block 4765 len 24576 data: 3 2 2 2 2 2
14148 217/300 Write block 2039 len 16384 data: 5 5 4 4
14149 218/300 Read block 4609 len 16384
14150 219/300 Write block 3529 len 40960 data: 3 2 2 2 3 3 3 3 3 2
14151 220/300 Flush
14152 221/300 Write block 4647 len 24576 data: 3 2 2 2 4 3
14153 222/300 Write block 1451 len 4096 data: 2
14154 223/300 Read block 1717 len 4096
14155 224/300 Read block 4985 len 36864
14156 225/300 Read block 2288 len 16384
14157 226/300 Read block 4643 len 20480
14158 227/300 Write block 4864 len 24576 data: 6 6 5 5 4 3
14159 228/300 Read block 2135 len 40960
14160 229/300 Write block 490 len 12288 data: 4 5 5
14161 230/300 Read block 1701 len 4096
14162 231/300 Read block 3222 len 32768
14163 232/300 Write block 1323 len 4096 data: 2
14164 233/300 Flush
14165 234/300 Read block 3274 len 32768
14166 235/300 Write block 929 len 12288 data: 5 7 6
14167 236/300 Write block 1804 len 20480 data: 4 4 3 5 4
14168 237/300 Read block 1235 len 16384
14169 238/300 Write block 3491 len 36864 data: 5 6 4 5 6 5 5 4 3
14170 239/300 Read block 956 len 36864
14171 240/300 Read block 2931 len 12288
14172 241/300 Write block 4986 len 12288 data: 4 2 2
14173 242/300 Read block 4673 len 28672
14174 243/300 Read block 2131 len 20480
14175 244/300 Write block 1645 len 20480 data: 4 3 3 4 3
14176 245/300 Write block 1341 len 8192 data: 5 5
14177 246/300 Write block 1636 len 12288 data: 4 6 6
14178 247/300 Write block 4186 len 8192 data: 2 2
14179 248/300 Read block 466 len 36864
14180 249/300 Write block 356 len 4096 data: 4
14181 250/300 Write block 3433 len 12288 data: 6 6 5
14182 251/300 Write block 3299 len 32768 data: 3 2 2 2 2 3 3 3
14183 252/300 Read block 2305 len 24576
14184 253/300 Write block 952 len 16384 data: 3 3 4 4
14185 254/300 Read block 2309 len 24576
14186 255/300 Write block 4899 len 16384 data: 5 6 5 5
14187 256/300 Write block 4054 len 16384 data: 4 3 6 7
14188 257/300 Read block 29 len 24576
14189 258/300 Write block 355 len 36864 data: 4 5 4 3 4 4 4 5 4
14190 259/300 Flush
14191 260/300 Read block 1425 len 32768
14192 261/300 Write block 1399 len 40960 data: 5 5 5 5 5 4 4 4 3 2
14193 262/300 Read block 941 len 40960
14194 263/300 Write block 2775 len 20480 data: 3 2 2 3 3
14195 264/300 Read block 4502 len 36864
14196 265/300 Read block 457 len 12288
14197 266/300 Read block 1083 len 8192
14198 267/300 Read block 2737 len 16384
14199 268/300 Read block 1071 len 20480
14200 269/300 Flush
14201 270/300 Flush
14202 271/300 Flush
14203 272/300 Read block 3011 len 20480
14204 273/300 Read block 3468 len 16384
14205 274/300 Read block 4500 len 36864
14206 275/300 Write block 1965 len 24576 data: 6 4 4 4 4 4
14207 276/300 Write block 4787 len 16384 data: 5 4 3 3
14208 277/300 Read block 4060 len 36864
14209 278/300 Read block 4163 len 16384
14210 279/300 Read block 2887 len 36864
14211 280/300 Write block 2941 len 28672 data: 5 4 5 4 3 5 5
14212 281/300 Write block 1583 len 16384 data: 5 5 5 5
14213 282/300 Write block 708 len 20480 data: 5 4 4 3 3
14214 283/300 Write block 6 len 40960 data: 4 5 5 4 5 4 5 4 5 5
14215 284/300 Read block 2833 len 28672
14216 285/300 Write block 3336 len 24576 data: 5 3 4 3 3 2
14217 286/300 Write block 541 len 28672 data: 4 4 4 5 3 3 4
14218 287/300 Flush
14219 288/300 Read block 3830 len 32768
14220 289/300 Write block 3952 len 24576 data: 3 3 2 2 2 3
14221 290/300 Write block 954 len 32768 data: 5 5 4 4 3 5 4 4
14222 291/300 Read block 1510 len 36864
14223 292/300 Read block 4208 len 32768
14224 293/300 Write block 4082 len 24576 data: 3 4 4 5 5 5
14225 294/300 Write block 4921 len 40960 data: 4 5 5 5 4 4 4 5 4 4
14226 295/300 Read block 4141 len 24576
14227 296/300 Write block 14 len 8192 data: 6 6
14228 297/300 Write block 4848 len 40960 data: 3 3 3 4 4 3 3 3 4 5
14229 298/300 Read block 684 len 24576
14230 299/300 Write block 1893 len 36864 data: 2 2 3 3 3 3 3 3 3
14231 300/300 Flush
14232 301/300 Write block 1985 len 16384 data: 4 6 5 5
14233 Main task has work to do, go find it
14234 got dsc Start(0)
14235 start 0
14236 Mar 17 22:19:26.492 INFO request completed, latency_us: 252, response_code: 204, uri: /start/cid/0, method: POST, req_id: 513b8dfe-91e6-420b-9549-ccadcce92ab9, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
14237 Replay: started 0, returned:Ok(())
14238 ----------------------------------------------------------------
14239 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14240 States: Offline Active Active
14241 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14242 4623 Acked Read 9 Sent Done Done false
14243 4624 Acked Read 9 Sent Done Done false
14244 4625 Acked Write 1 Sent Done Done false
14245 4626 Acked Read 4 Sent Done Done false
14246 4627 Acked Write 3 Sent Done Done false
14247 4628 Acked Read 5 Sent Done Done false
14248 4629 Acked Write 6 Sent Done Done false
14249 4630 Acked Read 8 Sent Done Done false
14250 4631 Acked Read 3 Sent Done Done false
14251 4632 Acked Read 3 Sent Done Done false
14252 4633 Acked Write 4 Sent Done Done false
14253 4634 Acked Write 10 Sent Done Done false
14254 4635 Acked Read 7 Sent Done Done false
14255 4636 Acked Read 9 Sent Done Done false
14256 4637 Acked Read 5 Sent Done Done false
14257 4638 Acked Read 6 Sent Done Done false
14258 4639 Acked Write 10 Sent Done Done false
14259 4640 Acked Read 10 Sent Done Done false
14260 4641 Acked Write 3 Sent Done Done false
14261 4642 Acked Write 8 Sent Done Done false
14262 4643 Acked Read 7 Sent Done Done false
14263 4644 Acked Read 3 Sent Done Done false
14264 4645 Acked Write 7 Sent Done Done false
14265 4646 Acked Write 9 Sent Done Done false
14266 4647 Acked Read 8 Sent Done Done false
14267 4648 Acked Read 9 Sent Done Done false
14268 4649 Acked Write 4 Sent Done Done false
14269 4650 Acked Read 3 Sent Done Done false
14270 4651 Acked Write 8 Sent Done Done false
14271 4652 Acked Write 1 Sent Done Done false
14272 4653 Acked Write 1 Sent Done Done false
14273 4654 Acked Write 2 Sent Done Done false
14274 4655 Acked Read 4 Sent Done Done false
14275 4656 Acked Write 6 Sent Done Done false
14276 4657 Acked Read 6 Sent Done Done false
14277 4658 Acked Write 8 Sent Done Done false
14278 4659 Acked Write 2 Sent Done Done false
14279 4660 Acked Read 1 Sent Done Done false
14280 4661 Acked Write 3 Sent Done Done false
14281 4662 Acked Write 8 Sent Done Done false
14282 4663 Acked Read 8 Sent Done Done false
14283 4664 Acked Write 7 Sent Done Done false
14284 4665 Acked Write 7 Sent Done Done false
14285 4666 Acked Read 6 Sent Done Done false
14286 4667 Acked Write 9 Sent Done Done false
14287 4668 Acked Read 8 Sent Done Done false
14288 4669 Acked Read 3 Sent Done Done false
14289 4670 Acked Read 8 Sent Done Done false
14290 4671 Acked Read 8 Sent Done Done false
14291 4672 Acked Write 6 Sent Done Done false
14292 4673 Acked Read 4 Sent Done Done false
14293 4674 Acked Read 9 Sent Done Done false
14294 4675 Acked Write 1 Sent Done Done false
14295 4676 Acked Write 2 Sent Done Done false
14296 4677 Acked Read 4 Sent Done Done false
14297 4678 Acked Read 2 Sent Done Done false
14298 4679 Acked Read 9 Sent Done Done false
14299 4680 Acked Write 1 Sent Done Done false
14300 4681 Acked Flush 0 Sent Done Done false
14301 4682 Acked Write 4 Sent Done Done false
14302 4683 Acked Flush 0 Sent Done Done false
14303 4684 Acked Write 3 Sent Done Done false
14304 4685 Acked Write 10 Sent Done Done false
14305 4686 Acked Flush 0 Sent Done Done false
14306 4687 Acked Read 5 Sent Done Done false
14307 4688 Acked Write 5 Sent Done Done false
14308 4689 Acked Read 9 Sent Done Done false
14309 4690 Acked Write 8 Sent Done Done false
14310 4691 Acked Write 3 Sent Done Done false
14311 4692 Acked Read 6 Sent Done Done false
14312 4693 Acked Flush 0 Sent Done Done false
14313 4694 Acked Write 6 Sent Done Done false
14314 4695 Acked Write 2 Sent Done Done false
14315 4696 Acked Read 1 Sent Done Done false
14316 4697 Acked Read 10 Sent Done Done false
14317 4698 Acked Write 3 Sent Done Done false
14318 4699 Acked Read 10 Sent Done Done false
14319 4700 Acked Write 4 Sent Done Done false
14320 4701 Acked Read 6 Sent Done Done false
14321 4702 Acked Write 10 Sent Done Done false
14322 4703 Acked Read 1 Sent Done Done false
14323 4704 Acked Flush 0 Sent Done Done false
14324 4705 Acked Write 10 Sent Done Done false
14325 4706 Acked Write 6 Sent Done Done false
14326 4707 Acked Read 7 Sent Done Done false
14327 4708 Acked Read 2 Sent Done Done false
14328 4709 Acked Write 5 Sent Done Done false
14329 4710 Acked Write 7 Sent Done Done false
14330 4711 Acked Read 4 Sent Done Done false
14331 4712 Acked Read 9 Sent Done Done false
14332 4713 Acked Read 6 Sent Done Done false
14333 4714 Acked Read 8 Sent Done Done false
14334 4715 Acked Read 3 Sent Done Done false
14335 4716 Acked Read 2 Sent Done Done false
14336 4717 Acked Read 10 Sent Done Done false
14337 4718 Acked Write 4 Sent Done Done false
14338 4719 Acked Flush 0 Sent Done Done false
14339 4720 Acked Read 1 Sent Done Done false
14340 4721 Acked Read 7 Sent Done Done false
14341 4722 Acked Read 4 Sent Done Done false
14342 4723 Acked Read 4 Sent Done Done false
14343 4724 Acked Read 2 Sent Done Done false
14344 4725 Acked Read 3 Sent Done Done false
14345 4726 Acked Read 8 Sent Done Done false
14346 4727 Acked Write 2 Sent Done Done false
14347 4728 Acked Read 6 Sent Done Done false
14348 4729 Acked Read 2 Sent Done Done false
14349 4730 Acked Read 7 Sent Done Done false
14350 4731 Acked Read 8 Sent Done Done false
14351 4732 Acked Read 4 Sent Done Done false
14352 4733 Acked Write 5 Sent Done Done false
14353 4734 Acked Read 1 Sent Done Done false
14354 4735 Acked Write 7 Sent Done Done false
14355 4736 Acked Write 5 Sent Done Done false
14356 4737 Acked Read 5 Sent Done Done false
14357 4738 Acked Read 8 Sent Done Done false
14358 4739 Acked Read 9 Sent Done Done false
14359 4740 Acked Read 10 Sent Done Done false
14360 4741 Acked Read 6 Sent Done Done false
14361 4742 Acked Read 2 Sent Done Done false
14362 4743 Acked Read 7 Sent Done Done false
14363 4744 Acked Write 10 Sent Done Done false
14364 4745 Acked Write 5 Sent Done Done false
14365 4746 Acked Read 4 Sent Done Done false
14366 4747 Acked Write 8 Sent Done Done false
14367 4748 Acked Write 5 Sent Done Done false
14368 4749 Acked Read 4 Sent Done Done false
14369 4750 Acked Read 1 Sent Done Done false
14370 4751 Acked Write 5 Sent Done Done false
14371 4752 Acked Write 8 Sent Done Done false
14372 4753 Acked Read 1 Sent Done Done false
14373 4754 Acked Write 4 Sent Done Done false
14374 4755 Acked Write 8 Sent Done Done false
14375 4756 Acked Write 4 Sent Done Done false
14376 4757 Acked Write 8 Sent Done Done false
14377 4758 Acked Write 10 Sent Done Done false
14378 4759 Acked Read 5 Sent Done Done false
14379 4760 Acked Read 1 Sent Done Done false
14380 4761 Acked Read 10 Sent Done Done false
14381 4762 Acked Read 4 Sent Done Done false
14382 4763 Acked Write 10 Sent Done Done false
14383 4764 Acked Read 3 Sent Done Done false
14384 4765 Acked Write 9 Sent Done Done false
14385 4766 Acked Flush 0 Sent Done Done false
14386 4767 Acked Read 1 Sent Done Done false
14387 4768 Acked Read 2 Sent Done Done false
14388 4769 Acked Write 10 Sent Done Done false
14389 4770 Acked Write 10 Sent Done Done false
14390 4771 Acked Read 10 Sent Done Done false
14391 4772 Acked Read 4 Sent Done Done false
14392 4773 Acked Read 7 Sent Done Done false
14393 4774 Acked Read 7 Sent Done Done false
14394 4775 Acked Read 8 Sent Done Done false
14395 4776 Acked Flush 0 Sent Done Done false
14396 4777 Acked Read 2 Sent Done Done false
14397 4778 Acked Write 4 Sent Done Done false
14398 4779 Acked Flush 0 Sent Done Done false
14399 4780 Acked Read 6 Sent Done Done false
14400 4781 Acked Write 10 Sent Done Done false
14401 4782 Acked Read 4 Sent Done Done false
14402 4783 Acked Write 2 Sent Done Done false
14403 4784 Acked Write 6 Sent Done Done false
14404 4785 Acked Write 1 Sent Done Done false
14405 4786 Acked Write 6 Sent Done Done false
14406 4787 Acked Write 7 Sent Done Done false
14407 4788 Acked Read 3 Sent Done Done false
14408 4789 Acked Write 9 Sent Done Done false
14409 4790 Acked Read 1 Sent Done Done false
14410 4791 Acked Read 4 Sent Done Done false
14411 4792 Acked Read 5 Sent Done Done false
14412 4793 Acked Write 3 Sent Done Done false
14413 4794 Acked Read 10 Sent Done Done false
14414 4795 Acked Flush 0 Sent Done Done false
14415 4796 Acked Read 6 Sent Done Done false
14416 4797 Acked Read 1 Sent Done Done false
14417 4798 Acked Read 6 Sent Done Done false
14418 4799 Acked Flush 0 Sent Done Done false
14419 4800 Acked Write 8 Sent Done Done false
14420 4801 Acked Write 3 Sent Done Done false
14421 4802 Acked Write 9 Sent Done Done false
14422 4803 Acked Write 3 Sent Done Done false
14423 4804 Acked Write 5 Sent Done Done false
14424 4805 Acked Read 3 Sent Done Done false
14425 4806 Acked Read 5 Sent Done Done false
14426 4807 Acked Read 3 Sent Done Done false
14427 4808 Acked Read 5 Sent Done Done false
14428 4809 Acked Write 3 Sent Done Done false
14429 4810 Acked Write 8 Sent Done Done false
14430 4811 Acked Write 2 Sent Done Done false
14431 4812 Acked Write 5 Sent Done Done false
14432 4813 Acked Write 4 Sent Done Done false
14433 4814 Acked Flush 0 Sent Done Done false
14434 4815 Acked Read 9 Sent Done Done false
14435 4816 Acked Read 4 Sent Done Done false
14436 4817 Acked Read 5 Sent Done Done false
14437 4818 Acked Write 6 Sent Done Done false
14438 4819 Acked Read 6 Sent Done Done false
14439 4820 Acked Write 1 Sent Done Done false
14440 4821 Acked Read 6 Sent Done Done false
14441 4822 Acked Read 4 Sent Done Done false
14442 4823 Acked Write 8 Sent Done Done false
14443 4824 Acked Read 8 Sent Done Done false
14444 4825 Acked Read 9 Sent Done Done false
14445 4826 Acked Write 1 Sent Done Done false
14446 4827 Acked Write 6 Sent Done Done false
14447 4828 Acked Write 6 Sent Done Done false
14448 4829 Acked Write 4 Sent Done Done false
14449 4830 Acked Read 5 Sent Done Done false
14450 4831 Acked Read 3 Sent Done Done false
14451 4832 Acked Read 7 Sent Done Done false
14452 4833 Acked Read 7 Sent Done Done false
14453 4834 Acked Write 9 Sent Done Done false
14454 4835 Acked Read 3 Sent Done Done false
14455 4836 Acked Read 8 Sent Done Done false
14456 4837 Acked Write 9 Sent Done Done false
14457 4838 Acked Write 6 Sent Done Done false
14458 4839 Acked Write 4 Sent Done Done false
14459 4840 Acked Read 4 Sent Done Done false
14460 4841 Acked Write 10 Sent Done Done false
14461 4842 Acked Flush 0 Sent Done Done false
14462 4843 Acked Write 6 Sent Done Done false
14463 4844 Acked Write 1 Sent Done Done false
14464 4845 Acked Read 1 Sent Done Done false
14465 4846 Acked Read 9 Sent Done Done false
14466 4847 Acked Read 4 Sent Done Done false
14467 4848 Acked Read 5 Sent Done Done false
14468 4849 Acked Write 6 Sent Done Done false
14469 4850 Acked Read 10 Sent Done Done false
14470 4851 Acked Write 3 Sent Done Done false
14471 4852 Acked Read 1 Sent Done Done false
14472 4853 Acked Read 8 Sent Done Done false
14473 4854 Acked Write 1 Sent Done Done false
14474 4855 Acked Flush 0 Sent Done Done false
14475 4856 Acked Read 8 Sent Done Done false
14476 4857 Acked Write 3 Sent Done Done false
14477 4858 Acked Write 5 Sent Done Done false
14478 4859 Acked Read 4 Sent Done Done false
14479 4860 Acked Write 9 Sent Done Done false
14480 4861 Acked Read 9 Sent Done Done false
14481 4862 Acked Read 3 Sent Done Done false
14482 4863 Acked Write 3 Sent Done Done false
14483 4864 Acked Read 7 Sent Done Done false
14484 4865 Acked Read 5 Sent Done Done false
14485 4866 Acked Write 5 Sent Done Done false
14486 4867 Acked Write 2 Sent Done Done false
14487 4868 Acked Write 3 Sent Done Done false
14488 4869 Acked Write 2 Sent Done Done false
14489 4870 Acked Read 9 Sent Done Done false
14490 4871 Acked Write 1 Sent Done Done false
14491 4872 Acked Write 3 Sent Done Done false
14492 4873 Acked Write 8 Sent Done Done false
14493 4874 Acked Read 6 Sent Done Done false
14494 4875 Acked Write 4 Sent Done Done false
14495 4876 Acked Read 6 Sent Done Done false
14496 4877 Acked Write 4 Sent Done Done false
14497 4878 Acked Write 4 Sent Done Done false
14498 4879 Acked Read 6 Sent Done Done false
14499 4880 Acked Write 9 Sent Done Done false
14500 4881 Acked Flush 0 Sent Done Done false
14501 4882 Acked Read 8 Sent Done Done false
14502 4883 Acked Write 10 Sent Done Done false
14503 4884 Acked Read 10 Sent Done Done false
14504 4885 Acked Write 5 Sent Done Done false
14505 4886 Acked Read 9 Sent Done Done false
14506 4887 Acked Read 3 Sent Done Done false
14507 4888 Acked Read 2 Sent Done Done false
14508 4889 Acked Read 4 Sent Done Done false
14509 4890 Acked Read 5 Sent Done Done false
14510 4891 Acked Flush 0 Sent Done Done false
14511 4892 Acked Flush 0 Sent Done Done false
14512 4893 Acked Flush 0 Sent Done Done false
14513 4894 Acked Read 5 Sent Done Done false
14514 4895 Acked Read 4 Sent Done Done false
14515 4896 Acked Read 9 Sent Done Done false
14516 4897 Acked Write 6 Sent Done Done false
14517 4898 Acked Write 4 Sent Done Done false
14518 4899 Acked Read 9 Sent Done Done false
14519 4900 Acked Read 4 Sent Done Done false
14520 4901 Acked Read 9 Sent Done Done false
14521 4902 Acked Write 7 Sent Done Done false
14522 4903 Acked Write 4 Sent Done Done false
14523 4904 Acked Write 5 Sent Done Done false
14524 4905 Acked Write 10 Sent Done Done false
14525 4906 Acked Read 7 Sent Done Done false
14526 4907 Acked Write 6 Sent Done Done false
14527 4908 Acked Write 7 Sent Done Done false
14528 4909 Acked Flush 0 Sent Done Done false
14529 4910 Acked Read 8 Sent Done Done false
14530 4911 Acked Write 6 Sent Done Done false
14531 4912 Acked Write 8 Sent Done Done false
14532 4913 Acked Read 9 Sent Done Done false
14533 4914 Acked Read 8 Sent Done Done false
14534 4915 Acked Write 6 Sent Done Done false
14535 4916 Acked Write 10 Sent Done Done false
14536 4917 Acked Read 6 Sent Done Done false
14537 4918 Acked Write 2 Sent Done Done false
14538 4919 Acked Write 10 Sent Done Done false
14539 4920 Acked Read 6 Sent Done Done false
14540 4921 Acked Write 9 Sent Done Done false
14541 4922 Acked Flush 0 Sent Done Done false
14542 4923 Acked Write 4 Sent Done Done false
14543 STATES DS:0 DS:1 DS:2 TOTAL
14544 Sent 301 0 0 301
14545 Done 0 301 301 602
14546 Skipped 0 0 0 0
14547 Error 0 0 0 0
14548 Last Flush: Some(JobId(4622)) Some(JobId(4922)) Some(JobId(4922))
14549 Downstairs last five completed: 4622 4621 4620 4619 4618
14550 Upstairs last five completed: 4923 4922 4921 4920 4919
14551 CLIENT: Up:0 ds:301 act:2
145522025-03-17T22:19:26.995ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
145532025-03-17T22:19:27.893ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
14554 [8810] got start action
14555 [8810] I am going to restart
14556 Starting downstairs at port 8810
14557 Make output file at "/tmp/dsc/downstairs-8810.txt"
14558 [8810][0] reports Starting
14559 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1219)
14560 [8810][0] reports Running
145612025-03-17T22:19:28.895ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
145622025-03-17T22:19:29.897ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
14563 ----------------------------------------------------------------
14564 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14565 States: Offline Active Active
14566 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
14567 4623 Acked Read 9 Sent Done Done false
14568 4624 Acked Read 9 Sent Done Done false
14569 4625 Acked Write 1 Sent Done Done false
14570 4626 Acked Read 4 Sent Done Done false
14571 4627 Acked Write 3 Sent Done Done false
14572 4628 Acked Read 5 Sent Done Done false
14573 4629 Acked Write 6 Sent Done Done false
14574 4630 Acked Read 8 Sent Done Done false
14575 4631 Acked Read 3 Sent Done Done false
14576 4632 Acked Read 3 Sent Done Done false
14577 4633 Acked Write 4 Sent Done Done false
14578 4634 Acked Write 10 Sent Done Done false
14579 4635 Acked Read 7 Sent Done Done false
14580 4636 Acked Read 9 Sent Done Done false
14581 4637 Acked Read 5 Sent Done Done false
14582 4638 Acked Read 6 Sent Done Done false
14583 4639 Acked Write 10 Sent Done Done false
14584 4640 Acked Read 10 Sent Done Done false
14585 4641 Acked Write 3 Sent Done Done false
14586 4642 Acked Write 8 Sent Done Done false
14587 4643 Acked Read 7 Sent Done Done false
14588 4644 Acked Read 3 Sent Done Done false
14589 4645 Acked Write 7 Sent Done Done false
14590 4646 Acked Write 9 Sent Done Done false
14591 4647 Acked Read 8 Sent Done Done false
14592 4648 Acked Read 9 Sent Done Done false
14593 4649 Acked Write 4 Sent Done Done false
14594 4650 Acked Read 3 Sent Done Done false
14595 4651 Acked Write 8 Sent Done Done false
14596 4652 Acked Write 1 Sent Done Done false
14597 4653 Acked Write 1 Sent Done Done false
14598 4654 Acked Write 2 Sent Done Done false
14599 4655 Acked Read 4 Sent Done Done false
14600 4656 Acked Write 6 Sent Done Done false
14601 4657 Acked Read 6 Sent Done Done false
14602 4658 Acked Write 8 Sent Done Done false
14603 4659 Acked Write 2 Sent Done Done false
14604 4660 Acked Read 1 Sent Done Done false
14605 4661 Acked Write 3 Sent Done Done false
14606 4662 Acked Write 8 Sent Done Done false
14607 4663 Acked Read 8 Sent Done Done false
14608 4664 Acked Write 7 Sent Done Done false
14609 4665 Acked Write 7 Sent Done Done false
14610 4666 Acked Read 6 Sent Done Done false
14611 4667 Acked Write 9 Sent Done Done false
14612 4668 Acked Read 8 Sent Done Done false
14613 4669 Acked Read 3 Sent Done Done false
14614 4670 Acked Read 8 Sent Done Done false
14615 4671 Acked Read 8 Sent Done Done false
14616 4672 Acked Write 6 Sent Done Done false
14617 4673 Acked Read 4 Sent Done Done false
14618 4674 Acked Read 9 Sent Done Done false
14619 4675 Acked Write 1 Sent Done Done false
14620 4676 Acked Write 2 Sent Done Done false
14621 4677 Acked Read 4 Sent Done Done false
14622 4678 Acked Read 2 Sent Done Done false
14623 4679 Acked Read 9 Sent Done Done false
14624 4680 Acked Write 1 Sent Done Done false
14625 4681 Acked Flush 0 Sent Done Done false
14626 4682 Acked Write 4 Sent Done Done false
14627 4683 Acked Flush 0 Sent Done Done false
14628 4684 Acked Write 3 Sent Done Done false
14629 4685 Acked Write 10 Sent Done Done false
14630 4686 Acked Flush 0 Sent Done Done false
14631 4687 Acked Read 5 Sent Done Done false
14632 4688 Acked Write 5 Sent Done Done false
14633 4689 Acked Read 9 Sent Done Done false
14634 4690 Acked Write 8 Sent Done Done false
14635 4691 Acked Write 3 Sent Done Done false
14636 4692 Acked Read 6 Sent Done Done false
14637 4693 Acked Flush 0 Sent Done Done false
14638 4694 Acked Write 6 Sent Done Done false
14639 4695 Acked Write 2 Sent Done Done false
14640 4696 Acked Read 1 Sent Done Done false
14641 4697 Acked Read 10 Sent Done Done false
14642 4698 Acked Write 3 Sent Done Done false
14643 4699 Acked Read 10 Sent Done Done false
14644 4700 Acked Write 4 Sent Done Done false
14645 4701 Acked Read 6 Sent Done Done false
14646 4702 Acked Write 10 Sent Done Done false
14647 4703 Acked Read 1 Sent Done Done false
14648 4704 Acked Flush 0 Sent Done Done false
14649 4705 Acked Write 10 Sent Done Done false
14650 4706 Acked Write 6 Sent Done Done false
14651 4707 Acked Read 7 Sent Done Done false
14652 4708 Acked Read 2 Sent Done Done false
14653 4709 Acked Write 5 Sent Done Done false
14654 4710 Acked Write 7 Sent Done Done false
14655 4711 Acked Read 4 Sent Done Done false
14656 4712 Acked Read 9 Sent Done Done false
14657 4713 Acked Read 6 Sent Done Done false
14658 4714 Acked Read 8 Sent Done Done false
14659 4715 Acked Read 3 Sent Done Done false
14660 4716 Acked Read 2 Sent Done Done false
14661 4717 Acked Read 10 Sent Done Done false
14662 4718 Acked Write 4 Sent Done Done false
14663 4719 Acked Flush 0 Sent Done Done false
14664 4720 Acked Read 1 Sent Done Done false
14665 4721 Acked Read 7 Sent Done Done false
14666 4722 Acked Read 4 Sent Done Done false
14667 4723 Acked Read 4 Sent Done Done false
14668 4724 Acked Read 2 Sent Done Done false
14669 4725 Acked Read 3 Sent Done Done false
14670 4726 Acked Read 8 Sent Done Done false
14671 4727 Acked Write 2 Sent Done Done false
14672 4728 Acked Read 6 Sent Done Done false
14673 4729 Acked Read 2 Sent Done Done false
14674 4730 Acked Read 7 Sent Done Done false
14675 4731 Acked Read 8 Sent Done Done false
14676 4732 Acked Read 4 Sent Done Done false
14677 4733 Acked Write 5 Sent Done Done false
14678 4734 Acked Read 1 Sent Done Done false
14679 4735 Acked Write 7 Sent Done Done false
14680 4736 Acked Write 5 Sent Done Done false
14681 4737 Acked Read 5 Sent Done Done false
14682 4738 Acked Read 8 Sent Done Done false
14683 4739 Acked Read 9 Sent Done Done false
14684 4740 Acked Read 10 Sent Done Done false
14685 4741 Acked Read 6 Sent Done Done false
14686 4742 Acked Read 2 Sent Done Done false
14687 4743 Acked Read 7 Sent Done Done false
14688 4744 Acked Write 10 Sent Done Done false
14689 4745 Acked Write 5 Sent Done Done false
14690 4746 Acked Read 4 Sent Done Done false
14691 4747 Acked Write 8 Sent Done Done false
14692 4748 Acked Write 5 Sent Done Done false
14693 4749 Acked Read 4 Sent Done Done false
14694 4750 Acked Read 1 Sent Done Done false
14695 4751 Acked Write 5 Sent Done Done false
14696 4752 Acked Write 8 Sent Done Done false
14697 4753 Acked Read 1 Sent Done Done false
14698 4754 Acked Write 4 Sent Done Done false
14699 4755 Acked Write 8 Sent Done Done false
14700 4756 Acked Write 4 Sent Done Done false
14701 4757 Acked Write 8 Sent Done Done false
14702 4758 Acked Write 10 Sent Done Done false
14703 4759 Acked Read 5 Sent Done Done false
14704 4760 Acked Read 1 Sent Done Done false
14705 4761 Acked Read 10 Sent Done Done false
14706 4762 Acked Read 4 Sent Done Done false
14707 4763 Acked Write 10 Sent Done Done false
14708 4764 Acked Read 3 Sent Done Done false
14709 4765 Acked Write 9 Sent Done Done false
14710 4766 Acked Flush 0 Sent Done Done false
14711 4767 Acked Read 1 Sent Done Done false
14712 4768 Acked Read 2 Sent Done Done false
14713 4769 Acked Write 10 Sent Done Done false
14714 4770 Acked Write 10 Sent Done Done false
14715 4771 Acked Read 10 Sent Done Done false
14716 4772 Acked Read 4 Sent Done Done false
14717 4773 Acked Read 7 Sent Done Done false
14718 4774 Acked Read 7 Sent Done Done false
14719 4775 Acked Read 8 Sent Done Done false
14720 4776 Acked Flush 0 Sent Done Done false
14721 4777 Acked Read 2 Sent Done Done false
14722 4778 Acked Write 4 Sent Done Done false
14723 4779 Acked Flush 0 Sent Done Done false
14724 4780 Acked Read 6 Sent Done Done false
14725 4781 Acked Write 10 Sent Done Done false
14726 4782 Acked Read 4 Sent Done Done false
14727 4783 Acked Write 2 Sent Done Done false
14728 4784 Acked Write 6 Sent Done Done false
14729 4785 Acked Write 1 Sent Done Done false
14730 4786 Acked Write 6 Sent Done Done false
14731 4787 Acked Write 7 Sent Done Done false
14732 4788 Acked Read 3 Sent Done Done false
14733 4789 Acked Write 9 Sent Done Done false
14734 4790 Acked Read 1 Sent Done Done false
14735 4791 Acked Read 4 Sent Done Done false
14736 4792 Acked Read 5 Sent Done Done false
14737 4793 Acked Write 3 Sent Done Done false
14738 4794 Acked Read 10 Sent Done Done false
14739 4795 Acked Flush 0 Sent Done Done false
14740 4796 Acked Read 6 Sent Done Done false
14741 4797 Acked Read 1 Sent Done Done false
14742 4798 Acked Read 6 Sent Done Done false
14743 4799 Acked Flush 0 Sent Done Done false
14744 4800 Acked Write 8 Sent Done Done false
14745 4801 Acked Write 3 Sent Done Done false
14746 4802 Acked Write 9 Sent Done Done false
14747 4803 Acked Write 3 Sent Done Done false
14748 4804 Acked Write 5 Sent Done Done false
14749 4805 Acked Read 3 Sent Done Done false
14750 4806 Acked Read 5 Sent Done Done false
14751 4807 Acked Read 3 Sent Done Done false
14752 4808 Acked Read 5 Sent Done Done false
14753 4809 Acked Write 3 Sent Done Done false
14754 4810 Acked Write 8 Sent Done Done false
14755 4811 Acked Write 2 Sent Done Done false
14756 4812 Acked Write 5 Sent Done Done false
14757 4813 Acked Write 4 Sent Done Done false
14758 4814 Acked Flush 0 Sent Done Done false
14759 4815 Acked Read 9 Sent Done Done false
14760 4816 Acked Read 4 Sent Done Done false
14761 4817 Acked Read 5 Sent Done Done false
14762 4818 Acked Write 6 Sent Done Done false
14763 4819 Acked Read 6 Sent Done Done false
14764 4820 Acked Write 1 Sent Done Done false
14765 4821 Acked Read 6 Sent Done Done false
14766 4822 Acked Read 4 Sent Done Done false
14767 4823 Acked Write 8 Sent Done Done false
14768 4824 Acked Read 8 Sent Done Done false
14769 4825 Acked Read 9 Sent Done Done false
14770 4826 Acked Write 1 Sent Done Done false
14771 4827 Acked Write 6 Sent Done Done false
14772 4828 Acked Write 6 Sent Done Done false
14773 4829 Acked Write 4 Sent Done Done false
14774 4830 Acked Read 5 Sent Done Done false
14775 4831 Acked Read 3 Sent Done Done false
14776 4832 Acked Read 7 Sent Done Done false
14777 4833 Acked Read 7 Sent Done Done false
14778 4834 Acked Write 9 Sent Done Done false
14779 4835 Acked Read 3 Sent Done Done false
14780 4836 Acked Read 8 Sent Done Done false
14781 4837 Acked Write 9 Sent Done Done false
14782 4838 Acked Write 6 Sent Done Done false
14783 4839 Acked Write 4 Sent Done Done false
14784 4840 Acked Read 4 Sent Done Done false
14785 4841 Acked Write 10 Sent Done Done false
14786 4842 Acked Flush 0 Sent Done Done false
14787 4843 Acked Write 6 Sent Done Done false
14788 4844 Acked Write 1 Sent Done Done false
14789 4845 Acked Read 1 Sent Done Done false
14790 4846 Acked Read 9 Sent Done Done false
14791 4847 Acked Read 4 Sent Done Done false
14792 4848 Acked Read 5 Sent Done Done false
14793 4849 Acked Write 6 Sent Done Done false
14794 4850 Acked Read 10 Sent Done Done false
14795 4851 Acked Write 3 Sent Done Done false
14796 4852 Acked Read 1 Sent Done Done false
14797 4853 Acked Read 8 Sent Done Done false
14798 4854 Acked Write 1 Sent Done Done false
14799 4855 Acked Flush 0 Sent Done Done false
14800 4856 Acked Read 8 Sent Done Done false
14801 4857 Acked Write 3 Sent Done Done false
14802 4858 Acked Write 5 Sent Done Done false
14803 4859 Acked Read 4 Sent Done Done false
14804 4860 Acked Write 9 Sent Done Done false
14805 4861 Acked Read 9 Sent Done Done false
14806 4862 Acked Read 3 Sent Done Done false
14807 4863 Acked Write 3 Sent Done Done false
14808 4864 Acked Read 7 Sent Done Done false
14809 4865 Acked Read 5 Sent Done Done false
14810 4866 Acked Write 5 Sent Done Done false
14811 4867 Acked Write 2 Sent Done Done false
14812 4868 Acked Write 3 Sent Done Done false
14813 4869 Acked Write 2 Sent Done Done false
14814 4870 Acked Read 9 Sent Done Done false
14815 4871 Acked Write 1 Sent Done Done false
14816 4872 Acked Write 3 Sent Done Done false
14817 4873 Acked Write 8 Sent Done Done false
14818 4874 Acked Read 6 Sent Done Done false
14819 4875 Acked Write 4 Sent Done Done false
14820 4876 Acked Read 6 Sent Done Done false
14821 4877 Acked Write 4 Sent Done Done false
14822 4878 Acked Write 4 Sent Done Done false
14823 4879 Acked Read 6 Sent Done Done false
14824 4880 Acked Write 9 Sent Done Done false
14825 4881 Acked Flush 0 Sent Done Done false
14826 4882 Acked Read 8 Sent Done Done false
14827 4883 Acked Write 10 Sent Done Done false
14828 4884 Acked Read 10 Sent Done Done false
14829 4885 Acked Write 5 Sent Done Done false
14830 4886 Acked Read 9 Sent Done Done false
14831 4887 Acked Read 3 Sent Done Done false
14832 4888 Acked Read 2 Sent Done Done false
14833 4889 Acked Read 4 Sent Done Done false
14834 4890 Acked Read 5 Sent Done Done false
14835 4891 Acked Flush 0 Sent Done Done false
14836 4892 Acked Flush 0 Sent Done Done false
14837 4893 Acked Flush 0 Sent Done Done false
14838 4894 Acked Read 5 Sent Done Done false
14839 4895 Acked Read 4 Sent Done Done false
14840 4896 Acked Read 9 Sent Done Done false
14841 4897 Acked Write 6 Sent Done Done false
14842 4898 Acked Write 4 Sent Done Done false
14843 4899 Acked Read 9 Sent Done Done false
14844 4900 Acked Read 4 Sent Done Done false
14845 4901 Acked Read 9 Sent Done Done false
14846 4902 Acked Write 7 Sent Done Done false
14847 4903 Acked Write 4 Sent Done Done false
14848 4904 Acked Write 5 Sent Done Done false
14849 4905 Acked Write 10 Sent Done Done false
14850 4906 Acked Read 7 Sent Done Done false
14851 4907 Acked Write 6 Sent Done Done false
14852 4908 Acked Write 7 Sent Done Done false
14853 4909 Acked Flush 0 Sent Done Done false
14854 4910 Acked Read 8 Sent Done Done false
14855 4911 Acked Write 6 Sent Done Done false
14856 4912 Acked Write 8 Sent Done Done false
14857 4913 Acked Read 9 Sent Done Done false
14858 4914 Acked Read 8 Sent Done Done false
14859 4915 Acked Write 6 Sent Done Done false
14860 4916 Acked Write 10 Sent Done Done false
14861 4917 Acked Read 6 Sent Done Done false
14862 4918 Acked Write 2 Sent Done Done false
14863 4919 Acked Write 10 Sent Done Done false
14864 4920 Acked Read 6 Sent Done Done false
14865 4921 Acked Write 9 Sent Done Done false
14866 4922 Acked Flush 0 Sent Done Done false
14867 4923 Acked Write 4 Sent Done Done false
14868 STATES DS:0 DS:1 DS:2 TOTAL
14869 Sent 301 0 0 301
14870 Done 0 301 301 602
14871 Skipped 0 0 0 0
14872 Error 0 0 0 0
14873 Last Flush: Some(JobId(4622)) Some(JobId(4922)) Some(JobId(4922))
14874 Downstairs last five completed: 4622 4621 4620 4619 4618
14875 Upstairs last five completed: 4923 4922 4921 4920 4919
14876 CLIENT: Up:0 ds:301 act:2
148772025-03-17T22:19:30.997ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
148782025-03-17T22:19:31.899ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
148792025-03-17T22:19:32.239ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
148802025-03-17T22:19:32.240ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:33684) = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
148812025-03-17T22:19:32.241ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
148822025-03-17T22:19:32.241ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
148832025-03-17T22:19:32.241ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
148842025-03-17T22:19:32.241ZINFOcrucible: send last flush ID to this DS: Some(JobId(4622)) = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
148852025-03-17T22:19:32.241ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
148862025-03-17T22:19:32.241ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(4622)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
148872025-03-17T22:19:32.242ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(4622)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
14888 ----------------------------------------------------------------
14889 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
14890 Downstairs last five completed: 4924 4923 4922 4921 4920
14891 Upstairs last five completed: 4924 4923 4922 4921 4920
14892 CLIENT: Up:0 ds:0 act:3
14893 Replay: All jobs finished
14894 Main task has work to do, go find it
14895 got dsc Stop(0)
14896 stop 0
14897 Mar 17 22:19:34.498 INFO[8810] Got stop action so:false kr:false
14898 request completed, latency_us: 364, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 703bf0aa-7ae0-4ff1-a79e-db7e75560ea5, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
14899 Mar 17 22:19:34.499 INFO request completed, latency_us: 138, response_code: {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:19:34.499349007Z","hostname":"w-01JPK2S810SFAA9787XEG8GFX5","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"0f8310d7-099e-4ead-8484-91ddc162d307"}
14900 200, uri: /state/cid/0, method: GET, req_id: fd5a5aec-d4a2-49ef-9c12-23eeed9e86ef, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
149012025-03-17T22:19:34.499ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
149022025-03-17T22:19:34.499ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
149032025-03-17T22:19:34.499ZINFOcrucible: client task is exiting = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
14904 [8810] Exited with: ExitStatus(unix_wait_status(9))
14905 [8810][0] reports Exit
14906 Mar 17 22:19:38.502 INFO request completed, latency_us: 226, response_code: 200, uri: /state/cid/0, method: GET, req_id: 5842d1b5-71d0-4fee-b0a3-f76cbfa1d6a0, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
14907 001/300 Flush
14908 002/300 Write block 3364 len 36864 data: 2 2 2 2 2 2 2 3 3
14909 003/300 Read block 4594 len 36864
14910 004/300 Flush
14911 005/300 Read block 1982 len 36864
14912 006/300 Read block 2578 len 8192
14913 007/300 Read block 4825 len 20480
14914 008/300 Write block 3915 len 40960 data: 3 4 5 5 4 3 3 3 3 2
14915 009/300 Read block 4534 len 40960
14916 010/300 Write block 117 len 20480 data: 4 4 4 4 4
14917 011/300 Write block 3803 len 8192 data: 6 5
14918 012/300 Flush
14919 013/300 Write block 4883 len 20480 data: 6 7 6 7 5
14920 014/300 Read block 367 len 40960
14921 015/300 Flush
14922 016/300 Read block 291 len 28672
14923 017/300 Write block 3774 len 28672 data: 3 3 3 2 2 2 3
14924 018/300 Read block 2902 len 8192
14925 019/300 Read block 4587 len 40960
14926 020/300 Flush
14927 021/300 Read block 2438 len 40960
14928 022/300 Write block 2997 len 16384 data: 4 5 5 4
14929 023/300 Read block 3773 len 4096
14930 024/300 Write block 4830 len 36864 data: 3 3 2 3 3 3 3 4 4
14931 025/300 Read block 1277 len 16384
14932 026/300 Write block 2587 len 4096 data: 4
14933 027/300 Write block 1511 len 4096 data: 3
14934 028/300 Read block 3080 len 4096
14935 029/300 Read block 2578 len 24576
14936 030/300 Write block 2562 len 28672 data: 4 4 4 5 5 4 4
14937 031/300 Read block 2516 len 12288
14938 032/300 Read block 3822 len 16384
14939 033/300 Read block 3501 len 36864
14940 034/300 Write block 1909 len 4096 data: 2
14941 035/300 Write block 4298 len 24576 data: 4 5 5 7 5 4
14942 036/300 Write block 3747 len 24576 data: 5 4 5 3 2 2
14943 037/300 Write block 3945 len 40960 data: 3 3 3 3 3 3 3 4 4 3
14944 038/300 Read block 4452 len 16384
14945 039/300 Read block 81 len 28672
14946 040/300 Write block 2469 len 16384 data: 2 2 3 4
14947 041/300 Read block 4321 len 8192
14948 042/300 Write block 2011 len 12288 data: 2 3 3
14949 043/300 Write block 3097 len 32768 data: 4 4 3 3 3 4 4 5
14950 044/300 Read block 1994 len 8192
14951 045/300 Read block 3631 len 40960
14952 046/300 Flush
14953 047/300 Write block 1283 len 28672 data: 5 4 3 3 3 3 3
14954 048/300 Read block 678 len 16384
14955 049/300 Flush
14956 050/300 Read block 4987 len 8192
14957 051/300 Read block 2954 len 12288
14958 052/300 Read block 2165 len 20480
14959 053/300 Read block 452 len 36864
14960 054/300 Write block 2821 len 28672 data: 5 5 5 7 7 7 7
14961 055/300 Write block 2222 len 36864 data: 6 5 5 5 4 3 3 4 4
14962 056/300 Write block 183 len 16384 data: 4 5 6 5
14963 057/300 Write block 306 len 40960 data: 4 4 3 4 3 3 2 3 3 3
14964 058/300 Write block 2784 len 12288 data: 3 4 3
14965 059/300 Read block 132 len 36864
14966 060/300 Read block 3719 len 32768
14967 061/300 Read block 3239 len 36864
14968 062/300 Read block 2236 len 40960
14969 063/300 Read block 822 len 28672
14970 064/300 Read block 3054 len 28672
14971 065/300 Read block 1403 len 32768
14972 066/300 Write block 4074 len 16384 data: 2 2 2 3
14973 067/300 Flush
14974 068/300 Read block 4638 len 4096
14975 069/300 Read block 1186 len 28672
14976 070/300 Write block 1578 len 12288 data: 3 3 5
14977 071/300 Write block 1951 len 36864 data: 3 4 4 3 3 4 3 4 4
14978 072/300 Read block 4956 len 24576
14979 073/300 Read block 916 len 36864
14980 074/300 Write block 2213 len 4096 data: 5
14981 075/300 Read block 2096 len 4096
14982 076/300 Write block 2457 len 8192 data: 2 3
14983 077/300 Read block 2412 len 12288
14984 078/300 Write block 2276 len 28672 data: 4 4 6 7 7 7 6
14985 079/300 Read block 2523 len 24576
14986 080/300 Read block 689 len 20480
14987 081/300 Read block 1545 len 20480
14988 082/300 Read block 4406 len 28672
14989 083/300 Flush
14990 084/300 Write block 2998 len 40960 data: 6 6 5 5 4 4 4 4 3 3
14991 085/300 Read block 4124 len 36864
14992 086/300 Read block 1669 len 8192
14993 087/300 Read block 1323 len 8192
14994 088/300 Write block 780 len 40960 data: 3 2 2 2 3 4 3 3 3 2
14995 089/300 Read block 1812 len 12288
14996 090/300 Read block 3725 len 12288
14997 091/300 Write block 4656 len 32768 data: 4 2 2 2 2 2 2 2
14998 092/300 Write block 3056 len 28672 data: 3 3 3 3 4 4 4
14999 093/300 Read block 174 len 12288
15000 094/300 Read block 403 len 36864
15001 095/300 Read block 2111 len 36864
15002 096/300 Flush
15003 097/300 Read block 3199 len 32768
15004 098/300 Read block 3734 len 40960
15005 099/300 Read block 4205 len 24576
15006 100/300 Read block 1414 len 20480
15007 101/300 Read block 2142 len 36864
15008 102/300 Read block 2313 len 32768
15009 103/300 Read block 1820 len 4096
15010 104/300 Write block 482 len 40960 data: 3 3 4 4 2 2 2 3 5 6
15011 105/300 Write block 4616 len 8192 data: 3 3
15012 106/300 Write block 1039 len 12288 data: 4 5 6
15013 107/300 Read block 573 len 8192
15014 108/300 Write block 4128 len 4096 data: 3
15015 109/300 Write block 2527 len 20480 data: 2 3 3 4 4
15016 110/300 Write block 4228 len 32768 data: 3 3 3 4 4 4 4 5
15017 111/300 Read block 4958 len 32768
15018 112/300 Write block 3127 len 16384 data: 4 4 6 7
15019 113/300 Flush
15020 114/300 Write block 1350 len 8192 data: 2 3
15021 115/300 Flush
15022 116/300 Write block 2194 len 40960 data: 4 4 3 2 2 2 5 6 6 6
15023 117/300 Read block 662 len 36864
15024 118/300 Read block 4417 len 36864
15025 119/300 Read block 2182 len 4096
15026 120/300 Write block 1162 len 8192 data: 4 3
15027 121/300 Write block 1856 len 20480 data: 3 4 5 4 6
15028 122/300 Read block 2921 len 8192
15029 123/300 Write block 4127 len 8192 data: 3 4
15030 124/300 Write block 4420 len 16384 data: 4 7 6 6
15031 125/300 Read block 805 len 4096
15032 126/300 Flush
15033 127/300 Read block 2981 len 16384
15034 128/300 Read block 4771 len 8192
15035 129/300 Read block 2407 len 40960
15036 130/300 Flush
15037 131/300 Write block 3635 len 16384 data: 2 3 3 3
15038 132/300 Write block 2955 len 32768 data: 4 6 7 7 8 8 7 6
15039 133/300 Read block 1151 len 32768
15040 134/300 Read block 479 len 40960
15041 135/300 Write block 2323 len 4096 data: 4
15042 136/300 Read block 4933 len 4096
15043 137/300 Read block 3843 len 4096
15044 138/300 Write block 2142 len 24576 data: 4 4 3 3 4 4
15045 139/300 Read block 3527 len 12288
15046 140/300 Write block 3535 len 40960 data: 4 4 4 3 2 2 2 2 3 3
15047 141/300 Read block 671 len 8192
15048 142/300 Flush
15049 143/300 Read block 627 len 20480
15050 144/300 Read block 312 len 24576
15051 145/300 Read block 858 len 8192
15052 146/300 Write block 726 len 4096 data: 3
15053 147/300 Write block 1061 len 28672 data: 3 2 3 3 3 4 3
15054 148/300 Read block 1412 len 36864
15055 149/300 Write block 2696 len 40960 data: 4 5 5 4 5 4 4 6 6 5
15056 150/300 Read block 1815 len 36864
15057 151/300 Write block 11 len 12288 data: 5 6 5
15058 152/300 Write block 797 len 12288 data: 4 6 5
15059 153/300 Read block 4 len 12288
15060 154/300 Read block 3148 len 20480
15061 155/300 Write block 2915 len 28672 data: 4 4 4 4 4 4 4
15062 156/300 Flush
15063 157/300 Flush
15064 158/300 Write block 2086 len 4096 data: 3
15065 159/300 Read block 4247 len 8192
15066 160/300 Flush
15067 161/300 Read block 4916 len 40960
15068 162/300 Write block 3289 len 36864 data: 5 5 5 5 6 4 4 3 3
15069 163/300 Write block 1328 len 36864 data: 4 3 3 3 2 2 2 2 2
15070 164/300 Read block 4928 len 12288
15071 165/300 Write block 2388 len 16384 data: 2 2 2 2
15072 166/300 Read block 3970 len 20480
15073 167/300 Read block 1352 len 12288
15074 168/300 Write block 1993 len 24576 data: 5 5 4 6 5 4
15075 169/300 Read block 1328 len 4096
15076 170/300 Read block 2076 len 4096
15077 171/300 Write block 3084 len 32768 data: 2 2 2 2 2 3 3 3
15078 172/300 Flush
15079 173/300 Write block 2892 len 28672 data: 3 3 3 3 2 3 3
15080 174/300 Read block 2444 len 32768
15081 175/300 Read block 1187 len 4096
15082 176/300 Read block 1880 len 4096
15083 177/300 Write block 2346 len 12288 data: 2 2 2
15084 178/300 Write block 2956 len 4096 data: 7
15085 179/300 Write block 1178 len 40960 data: 5 4 4 4 5 5 5 5 3 3
15086 180/300 Read block 837 len 32768
15087 181/300 Read block 3416 len 24576
15088 182/300 Read block 3846 len 12288
15089 183/300 Read block 3059 len 20480
15090 184/300 Read block 2051 len 40960
15091 185/300 Read block 4954 len 20480
15092 186/300 Write block 3004 len 20480 data: 5 5 4 4 3
15093 187/300 Write block 2777 len 20480 data: 3 4 4 3 3
15094 188/300 Write block 4633 len 28672 data: 4 4 4 4 4 4 4
15095 189/300 Write block 2757 len 24576 data: 5 5 5 5 4 4
15096 190/300 Read block 2026 len 36864
15097 191/300 Read block 2622 len 40960
15098 192/300 Read block 2136 len 8192
15099 193/300 Write block 905 len 40960 data: 3 4 4 3 3 3 3 2 2 2
15100 194/300 Write block 1211 len 28672 data: 3 4 3 3 4 4 4
15101 195/300 Write block 4606 len 4096 data: 8
15102 196/300 Write block 1597 len 12288 data: 5 6 7
15103 197/300 Read block 2979 len 12288
15104 198/300 Read block 2232 len 40960
15105 199/300 Read block 1480 len 4096
15106 200/300 Read block 1643 len 20480
15107 201/300 Write block 4916 len 24576 data: 4 4 4 4 4 5
15108 202/300 Read block 452 len 40960
15109 203/300 Write block 4087 len 40960 data: 6 4 4 4 3 2 3 3 4 3
15110 204/300 Read block 1425 len 12288
15111 205/300 Write block 4067 len 8192 data: 3 4
15112 206/300 Write block 2780 len 20480 data: 4 4 3 3 4
15113 207/300 Read block 713 len 12288
15114 208/300 Read block 4879 len 12288
15115 209/300 Read block 3871 len 24576
15116 210/300 Read block 4351 len 4096
15117 211/300 Read block 476 len 16384
15118 212/300 Read block 3702 len 36864
15119 213/300 Read block 4667 len 4096
15120 214/300 Flush
15121 215/300 Read block 2565 len 40960
15122 216/300 Write block 1422 len 36864 data: 3 3 3 3 3 3 3 2 3
15123 217/300 Read block 1046 len 12288
15124 218/300 Write block 1522 len 28672 data: 5 5 6 7 7 8 7
15125 219/300 Read block 1022 len 20480
15126 220/300 Read block 1278 len 4096
15127 221/300 Read block 3606 len 16384
15128 222/300 Read block 2960 len 12288
15129 223/300 Write block 707 len 16384 data: 5 6 5 5
15130 224/300 Flush
15131 225/300 Read block 2951 len 32768
15132 226/300 Read block 149 len 12288
15133 227/300 Write block 1653 len 32768 data: 2 2 2 3 3 3 3 4
15134 228/300 Read block 487 len 20480
15135 229/300 Write block 1336 len 16384 data: 3 2 2 2
15136 230/300 Write block 984 len 24576 data: 2 2 3 3 5 5
15137 231/300 Read block 2396 len 20480
15138 232/300 Read block 4446 len 40960
15139 233/300 Flush
15140 234/300 Write block 2936 len 4096 data: 4
15141 235/300 Read block 788 len 8192
15142 236/300 Read block 2199 len 24576
15143 237/300 Read block 2467 len 24576
15144 238/300 Write block 2235 len 28672 data: 2 2 2 2 2 2 3
15145 239/300 Read block 3731 len 40960
15146 240/300 Read block 4031 len 24576
15147 241/300 Read block 4246 len 12288
15148 242/300 Read block 2602 len 8192
15149 243/300 Read block 1604 len 8192
15150 244/300 Read block 2648 len 28672
15151 245/300 Write block 4810 len 8192 data: 6 5
15152 246/300 Read block 3311 len 16384
15153 247/300 Read block 2803 len 24576
15154 248/300 Flush
15155 249/300 Write block 2417 len 20480 data: 4 2 3 3 4
15156 250/300 Read block 3318 len 36864
15157 251/300 Read block 2621 len 4096
15158 252/300 Read block 928 len 12288
15159 253/300 Read block 2873 len 40960
15160 254/300 Write block 2293 len 36864 data: 4 3 4 4 4 4 4 4 2
15161 255/300 Flush
15162 256/300 Read block 433 len 12288
15163 257/300 Read block 2908 len 12288
15164 258/300 Read block 678 len 28672
15165 259/300 Read block 1123 len 28672
15166 260/300 Read block 1044 len 32768
15167 261/300 Read block 4687 len 12288
15168 262/300 Read block 2984 len 8192
15169 263/300 Write block 4166 len 36864 data: 4 2 3 3 3 3 3 2 2
15170 264/300 Write block 2275 len 4096 data: 4
15171 265/300 Read block 125 len 32768
15172 266/300 Write block 1914 len 24576 data: 2 3 3 3 3 3
15173 267/300 Read block 1847 len 16384
15174 268/300 Flush
15175 269/300 Write block 4859 len 36864 data: 6 7 7 7 7 7 7 6 6
15176 270/300 Write block 10 len 28672 data: 6 6 7 6 7 7 5
15177 271/300 Read block 3165 len 28672
15178 272/300 Write block 3458 len 40960 data: 4 4 5 5 4 4 5 4 4 3
15179 273/300 Read block 3530 len 4096
15180 274/300 Read block 3854 len 36864
15181 275/300 Read block 1540 len 20480
15182 276/300 Write block 1373 len 12288 data: 3 3 3
15183 277/300 Read block 3210 len 24576
15184 278/300 Write block 658 len 32768 data: 4 4 3 2 2 2 2 4
15185 279/300 Write block 4825 len 4096 data: 3
15186 280/300 Flush
15187 281/300 Write block 333 len 20480 data: 7 7 8 8 8
15188 282/300 Read block 3497 len 36864
15189 283/300 Write block 3033 len 32768 data: 2 2 2 4 4 4 5 3
15190 284/300 Read block 72 len 16384
15191 285/300 Write block 796 len 28672 data: 3 5 7 6 6 6 4
15192 286/300 Read block 1839 len 8192
15193 287/300 Write block 4274 len 40960 data: 4 4 3 3 3 2 3 3 3 3
15194 288/300 Write block 3701 len 40960 data: 4 4 4 4 3 2 2 2 2 3
15195 289/300 Read block 4974 len 20480
15196 290/300 Read block 1369 len 4096
15197 291/300 Write block 4137 len 28672 data: 2 2 2 2 2 2 2
15198 292/300 Read block 3931 len 24576
15199 293/300 Read block 4945 len 8192
15200 294/300 Write block 1323 len 20480 data: 3 2 2 4 4
15201 295/300 Flush
15202 296/300 Read block 4698 len 36864
15203 297/300 Write block 484 len 40960 data: 5 5 3 3 3 4 6 7 6 6
15204 298/300 Read block 365 len 20480
15205 299/300 Write block 2696 len 28672 data: 5 6 6 5 6 5 5
15206 300/300 Read block 3519 len 40960
15207 301/300 Read block 659 len 8192
15208 Main task has work to do, go find it
15209 got dsc Start(0)
15210 start 0
15211 Mar 17 22:19:38.757 INFO request completed, latency_us: 218, response_code: 204, uri: /start/cid/0, method: POST, req_id: 4c3adaa3-7af8-42f4-a48a-6c2c328b82bc, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
15212 Replay: started 0, returned:Ok(())
15213 ----------------------------------------------------------------
15214 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15215 States: Offline Active Active
15216 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15217 4925 Acked Flush 0 Sent Done Done false
15218 4926 Acked Write 9 Sent Done Done false
15219 4927 Acked Read 9 Sent Done Done false
15220 4928 Acked Flush 0 Sent Done Done false
15221 4929 Acked Read 9 Sent Done Done false
15222 4930 Acked Read 2 Sent Done Done false
15223 4931 Acked Read 5 Sent Done Done false
15224 4932 Acked Write 10 Sent Done Done false
15225 4933 Acked Read 10 Sent Done Done false
15226 4934 Acked Write 5 Sent Done Done false
15227 4935 Acked Write 2 Sent Done Done false
15228 4936 Acked Flush 0 Sent Done Done false
15229 4937 Acked Write 5 Sent Done Done false
15230 4938 Acked Read 10 Sent Done Done false
15231 4939 Acked Flush 0 Sent Done Done false
15232 4940 Acked Read 7 Sent Done Done false
15233 4941 Acked Write 7 Sent Done Done false
15234 4942 Acked Read 2 Sent Done Done false
15235 4943 Acked Read 10 Sent Done Done false
15236 4944 Acked Flush 0 Sent Done Done false
15237 4945 Acked Read 10 Sent Done Done false
15238 4946 Acked Write 4 Sent Done Done false
15239 4947 Acked Read 1 Sent Done Done false
15240 4948 Acked Write 9 Sent Done Done false
15241 4949 Acked Read 4 Sent Done Done false
15242 4950 Acked Write 1 Sent Done Done false
15243 4951 Acked Write 1 Sent Done Done false
15244 4952 Acked Read 1 Sent Done Done false
15245 4953 Acked Read 6 Sent Done Done false
15246 4954 Acked Write 7 Sent Done Done false
15247 4955 Acked Read 3 Sent Done Done false
15248 4956 Acked Read 4 Sent Done Done false
15249 4957 Acked Read 9 Sent Done Done false
15250 4958 Acked Write 1 Sent Done Done false
15251 4959 Acked Write 6 Sent Done Done false
15252 4960 Acked Write 6 Sent Done Done false
15253 4961 Acked Write 10 Sent Done Done false
15254 4962 Acked Read 4 Sent Done Done false
15255 4963 Acked Read 7 Sent Done Done false
15256 4964 Acked Write 4 Sent Done Done false
15257 4965 Acked Read 2 Sent Done Done false
15258 4966 Acked Write 3 Sent Done Done false
15259 4967 Acked Write 8 Sent Done Done false
15260 4968 Acked Read 2 Sent Done Done false
15261 4969 Acked Read 10 Sent Done Done false
15262 4970 Acked Flush 0 Sent Done Done false
15263 4971 Acked Write 7 Sent Done Done false
15264 4972 Acked Read 4 Sent Done Done false
15265 4973 Acked Flush 0 Sent Done Done false
15266 4974 Acked Read 2 Sent Done Done false
15267 4975 Acked Read 3 Sent Done Done false
15268 4976 Acked Read 5 Sent Done Done false
15269 4977 Acked Read 9 Sent Done Done false
15270 4978 Acked Write 7 Sent Done Done false
15271 4979 Acked Write 9 Sent Done Done false
15272 4980 Acked Write 4 Sent Done Done false
15273 4981 Acked Write 10 Sent Done Done false
15274 4982 Acked Write 3 Sent Done Done false
15275 4983 Acked Read 9 Sent Done Done false
15276 4984 Acked Read 8 Sent Done Done false
15277 4985 Acked Read 9 Sent Done Done false
15278 4986 Acked Read 10 Sent Done Done false
15279 4987 Acked Read 7 Sent Done Done false
15280 4988 Acked Read 7 Sent Done Done false
15281 4989 Acked Read 8 Sent Done Done false
15282 4990 Acked Write 4 Sent Done Done false
15283 4991 Acked Flush 0 Sent Done Done false
15284 4992 Acked Read 1 Sent Done Done false
15285 4993 Acked Read 7 Sent Done Done false
15286 4994 Acked Write 3 Sent Done Done false
15287 4995 Acked Write 9 Sent Done Done false
15288 4996 Acked Read 6 Sent Done Done false
15289 4997 Acked Read 9 Sent Done Done false
15290 4998 Acked Write 1 Sent Done Done false
15291 4999 Acked Read 1 Sent Done Done false
15292 5000 Acked Write 2 Sent Done Done false
15293 5001 Acked Read 3 Sent Done Done false
15294 5002 Acked Write 7 Sent Done Done false
15295 5003 Acked Read 6 Sent Done Done false
15296 5004 Acked Read 5 Sent Done Done false
15297 5005 Acked Read 5 Sent Done Done false
15298 5006 Acked Read 7 Sent Done Done false
15299 5007 Acked Flush 0 Sent Done Done false
15300 5008 Acked Write 10 Sent Done Done false
15301 5009 Acked Read 9 Sent Done Done false
15302 5010 Acked Read 2 Sent Done Done false
15303 5011 Acked Read 2 Sent Done Done false
15304 5012 Acked Write 10 Sent Done Done false
15305 5013 Acked Read 3 Sent Done Done false
15306 5014 Acked Read 3 Sent Done Done false
15307 5015 Acked Write 8 Sent Done Done false
15308 5016 Acked Write 7 Sent Done Done false
15309 5017 Acked Read 3 Sent Done Done false
15310 5018 Acked Read 9 Sent Done Done false
15311 5019 Acked Read 9 Sent Done Done false
15312 5020 Acked Flush 0 Sent Done Done false
15313 5021 Acked Read 8 Sent Done Done false
15314 5022 Acked Read 10 Sent Done Done false
15315 5023 Acked Read 6 Sent Done Done false
15316 5024 Acked Read 5 Sent Done Done false
15317 5025 Acked Read 9 Sent Done Done false
15318 5026 Acked Read 8 Sent Done Done false
15319 5027 Acked Read 1 Sent Done Done false
15320 5028 Acked Write 10 Sent Done Done false
15321 5029 Acked Write 2 Sent Done Done false
15322 5030 Acked Write 3 Sent Done Done false
15323 5031 Acked Read 2 Sent Done Done false
15324 5032 Acked Write 1 Sent Done Done false
15325 5033 Acked Write 5 Sent Done Done false
15326 5034 Acked Write 8 Sent Done Done false
15327 5035 Acked Read 8 Sent Done Done false
15328 5036 Acked Write 4 Sent Done Done false
15329 5037 Acked Flush 0 Sent Done Done false
15330 5038 Acked Write 2 Sent Done Done false
15331 5039 Acked Flush 0 Sent Done Done false
15332 5040 Acked Write 10 Sent Done Done false
15333 5041 Acked Read 9 Sent Done Done false
15334 5042 Acked Read 9 Sent Done Done false
15335 5043 Acked Read 1 Sent Done Done false
15336 5044 Acked Write 2 Sent Done Done false
15337 5045 Acked Write 5 Sent Done Done false
15338 5046 Acked Read 2 Sent Done Done false
15339 5047 Acked Write 2 Sent Done Done false
15340 5048 Acked Write 4 Sent Done Done false
15341 5049 Acked Read 1 Sent Done Done false
15342 5050 Acked Flush 0 Sent Done Done false
15343 5051 Acked Read 4 Sent Done Done false
15344 5052 Acked Read 2 Sent Done Done false
15345 5053 Acked Read 10 Sent Done Done false
15346 5054 Acked Flush 0 Sent Done Done false
15347 5055 Acked Write 4 Sent Done Done false
15348 5056 Acked Write 8 Sent Done Done false
15349 5057 Acked Read 8 Sent Done Done false
15350 5058 Acked Read 10 Sent Done Done false
15351 5059 Acked Write 1 Sent Done Done false
15352 5060 Acked Read 1 Sent Done Done false
15353 5061 Acked Read 1 Sent Done Done false
15354 5062 Acked Write 6 Sent Done Done false
15355 5063 Acked Read 3 Sent Done Done false
15356 5064 Acked Write 10 Sent Done Done false
15357 5065 Acked Read 2 Sent Done Done false
15358 5066 Acked Flush 0 Sent Done Done false
15359 5067 Acked Read 5 Sent Done Done false
15360 5068 Acked Read 6 Sent Done Done false
15361 5069 Acked Read 2 Sent Done Done false
15362 5070 Acked Write 1 Sent Done Done false
15363 5071 Acked Write 7 Sent Done Done false
15364 5072 Acked Read 9 Sent Done Done false
15365 5073 Acked Write 10 Sent Done Done false
15366 5074 Acked Read 9 Sent Done Done false
15367 5075 Acked Write 3 Sent Done Done false
15368 5076 Acked Write 3 Sent Done Done false
15369 5077 Acked Read 3 Sent Done Done false
15370 5078 Acked Read 5 Sent Done Done false
15371 5079 Acked Write 7 Sent Done Done false
15372 5080 Acked Flush 0 Sent Done Done false
15373 5081 Acked Flush 0 Sent Done Done false
15374 5082 Acked Write 1 Sent Done Done false
15375 5083 Acked Read 2 Sent Done Done false
15376 5084 Acked Flush 0 Sent Done Done false
15377 5085 Acked Read 10 Sent Done Done false
15378 5086 Acked Write 9 Sent Done Done false
15379 5087 Acked Write 9 Sent Done Done false
15380 5088 Acked Read 3 Sent Done Done false
15381 5089 Acked Write 4 Sent Done Done false
15382 5090 Acked Read 5 Sent Done Done false
15383 5091 Acked Read 3 Sent Done Done false
15384 5092 Acked Write 6 Sent Done Done false
15385 5093 Acked Read 1 Sent Done Done false
15386 5094 Acked Read 1 Sent Done Done false
15387 5095 Acked Write 8 Sent Done Done false
15388 5096 Acked Flush 0 Sent Done Done false
15389 5097 Acked Write 7 Sent Done Done false
15390 5098 Acked Read 8 Sent Done Done false
15391 5099 Acked Read 1 Sent Done Done false
15392 5100 Acked Read 1 Sent Done Done false
15393 5101 Acked Write 3 Sent Done Done false
15394 5102 Acked Write 1 Sent Done Done false
15395 5103 Acked Write 10 Sent Done Done false
15396 5104 Acked Read 8 Sent Done Done false
15397 5105 Acked Read 6 Sent Done Done false
15398 5106 Acked Read 3 Sent Done Done false
15399 5107 Acked Read 5 Sent Done Done false
15400 5108 Acked Read 10 Sent Done Done false
15401 5109 Acked Read 5 Sent Done Done false
15402 5110 Acked Write 5 Sent Done Done false
15403 5111 Acked Write 5 Sent Done Done false
15404 5112 Acked Write 7 Sent Done Done false
15405 5113 Acked Write 6 Sent Done Done false
15406 5114 Acked Read 9 Sent Done Done false
15407 5115 Acked Read 10 Sent Done Done false
15408 5116 Acked Read 2 Sent Done Done false
15409 5117 Acked Write 10 Sent Done Done false
15410 5118 Acked Write 7 Sent Done Done false
15411 5119 Acked Write 1 Sent Done Done false
15412 5120 Acked Write 3 Sent Done Done false
15413 5121 Acked Read 3 Sent Done Done false
15414 5122 Acked Read 10 Sent Done Done false
15415 5123 Acked Read 1 Sent Done Done false
15416 5124 Acked Read 5 Sent Done Done false
15417 5125 Acked Write 6 Sent Done Done false
15418 5126 Acked Read 10 Sent Done Done false
15419 5127 Acked Write 10 Sent Done Done false
15420 5128 Acked Read 3 Sent Done Done false
15421 5129 Acked Write 2 Sent Done Done false
15422 5130 Acked Write 5 Sent Done Done false
15423 5131 Acked Read 3 Sent Done Done false
15424 5132 Acked Read 3 Sent Done Done false
15425 5133 Acked Read 6 Sent Done Done false
15426 5134 Acked Read 1 Sent Done Done false
15427 5135 Acked Read 4 Sent Done Done false
15428 5136 Acked Read 9 Sent Done Done false
15429 5137 Acked Read 1 Sent Done Done false
15430 5138 Acked Flush 0 Sent Done Done false
15431 5139 Acked Read 10 Sent Done Done false
15432 5140 Acked Write 9 Sent Done Done false
15433 5141 Acked Read 3 Sent Done Done false
15434 5142 Acked Write 7 Sent Done Done false
15435 5143 Acked Read 5 Sent Done Done false
15436 5144 Acked Read 1 Sent Done Done false
15437 5145 Acked Read 4 Sent Done Done false
15438 5146 Acked Read 3 Sent Done Done false
15439 5147 Acked Write 4 Sent Done Done false
15440 5148 Acked Flush 0 Sent Done Done false
15441 5149 Acked Read 8 Sent Done Done false
15442 5150 Acked Read 3 Sent Done Done false
15443 5151 Acked Write 8 Sent Done Done false
15444 5152 Acked Read 5 Sent Done Done false
15445 5153 Acked Write 4 Sent Done Done false
15446 5154 Acked Write 6 Sent Done Done false
15447 5155 Acked Read 5 Sent Done Done false
15448 5156 Acked Read 10 Sent Done Done false
15449 5157 Acked Flush 0 Sent Done Done false
15450 5158 Acked Write 1 Sent Done Done false
15451 5159 Acked Read 2 Sent Done Done false
15452 5160 Acked Read 6 Sent Done Done false
15453 5161 Acked Read 6 Sent Done Done false
15454 5162 Acked Write 7 Sent Done Done false
15455 5163 Acked Read 10 Sent Done Done false
15456 5164 Acked Read 6 Sent Done Done false
15457 5165 Acked Read 3 Sent Done Done false
15458 5166 Acked Read 2 Sent Done Done false
15459 5167 Acked Read 2 Sent Done Done false
15460 5168 Acked Read 7 Sent Done Done false
15461 5169 Acked Write 2 Sent Done Done false
15462 5170 Acked Read 4 Sent Done Done false
15463 5171 Acked Read 6 Sent Done Done false
15464 5172 Acked Flush 0 Sent Done Done false
15465 5173 Acked Write 5 Sent Done Done false
15466 5174 Acked Read 9 Sent Done Done false
15467 5175 Acked Read 1 Sent Done Done false
15468 5176 Acked Read 3 Sent Done Done false
15469 5177 Acked Read 10 Sent Done Done false
15470 5178 Acked Write 9 Sent Done Done false
15471 5179 Acked Flush 0 Sent Done Done false
15472 5180 Acked Read 3 Sent Done Done false
15473 5181 Acked Read 3 Sent Done Done false
15474 5182 Acked Read 7 Sent Done Done false
15475 5183 Acked Read 7 Sent Done Done false
15476 5184 Acked Read 8 Sent Done Done false
15477 5185 Acked Read 3 Sent Done Done false
15478 5186 Acked Read 2 Sent Done Done false
15479 5187 Acked Write 9 Sent Done Done false
15480 5188 Acked Write 1 Sent Done Done false
15481 5189 Acked Read 8 Sent Done Done false
15482 5190 Acked Write 6 Sent Done Done false
15483 5191 Acked Read 4 Sent Done Done false
15484 5192 Acked Flush 0 Sent Done Done false
15485 5193 Acked Write 9 Sent Done Done false
15486 5194 Acked Write 7 Sent Done Done false
15487 5195 Acked Read 7 Sent Done Done false
15488 5196 Acked Write 10 Sent Done Done false
15489 5197 Acked Read 1 Sent Done Done false
15490 5198 Acked Read 9 Sent Done Done false
15491 5199 Acked Read 5 Sent Done Done false
15492 5200 Acked Write 3 Sent Done Done false
15493 5201 Acked Read 6 Sent Done Done false
15494 5202 Acked Write 8 Sent Done Done false
15495 5203 Acked Write 1 Sent Done Done false
15496 5204 Acked Flush 0 Sent Done Done false
15497 5205 Acked Write 5 Sent Done Done false
15498 5206 Acked Read 9 Sent Done Done false
15499 5207 Acked Write 8 Sent Done Done false
15500 5208 Acked Read 4 Sent Done Done false
15501 5209 Acked Write 7 Sent Done Done false
15502 5210 Acked Read 2 Sent Done Done false
15503 5211 Acked Write 10 Sent Done Done false
15504 5212 Acked Write 10 Sent Done Done false
15505 5213 Acked Read 5 Sent Done Done false
15506 5214 Acked Read 1 Sent Done Done false
15507 5215 Acked Write 7 Sent Done Done false
15508 5216 Acked Read 6 Sent Done Done false
15509 5217 Acked Read 2 Sent Done Done false
15510 5218 Acked Write 5 Sent Done Done false
15511 5219 Acked Flush 0 Sent Done Done false
15512 5220 Acked Read 9 Sent Done Done false
15513 5221 Acked Write 10 Sent Done Done false
15514 5222 Acked Read 5 Sent Done Done false
15515 5223 Acked Write 7 Sent Done Done false
15516 5224 Acked Read 10 Sent Done Done false
15517 5225 Acked Read 2 Sent Done Done false
15518 STATES DS:0 DS:1 DS:2 TOTAL
15519 Sent 301 0 0 301
15520 Done 0 301 301 602
15521 Skipped 0 0 0 0
15522 Error 0 0 0 0
15523 Last Flush: Some(JobId(4924)) Some(JobId(5219)) Some(JobId(5219))
15524 Downstairs last five completed: 4924 4923 4922 4921 4920
15525 Upstairs last five completed: 5225 5224 5223 5222 5221
15526 CLIENT: Up:0 ds:301 act:2
155272025-03-17T22:19:39.259ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
155282025-03-17T22:19:39.911ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
15529 [8810] got start action
15530 [8810] I am going to restart
15531 Starting downstairs at port 8810
15532 Make output file at "/tmp/dsc/downstairs-8810.txt"
15533 [8810][0] reports Starting
15534 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1220)
15535 [8810][0] reports Running
155362025-03-17T22:19:40.978ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
155372025-03-17T22:19:41.915ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
15538 ----------------------------------------------------------------
15539 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15540 States: Offline Active Active
15541 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
15542 4925 Acked Flush 0 Sent Done Done false
15543 4926 Acked Write 9 Sent Done Done false
15544 4927 Acked Read 9 Sent Done Done false
15545 4928 Acked Flush 0 Sent Done Done false
15546 4929 Acked Read 9 Sent Done Done false
15547 4930 Acked Read 2 Sent Done Done false
15548 4931 Acked Read 5 Sent Done Done false
15549 4932 Acked Write 10 Sent Done Done false
15550 4933 Acked Read 10 Sent Done Done false
15551 4934 Acked Write 5 Sent Done Done false
15552 4935 Acked Write 2 Sent Done Done false
15553 4936 Acked Flush 0 Sent Done Done false
15554 4937 Acked Write 5 Sent Done Done false
15555 4938 Acked Read 10 Sent Done Done false
15556 4939 Acked Flush 0 Sent Done Done false
15557 4940 Acked Read 7 Sent Done Done false
15558 4941 Acked Write 7 Sent Done Done false
15559 4942 Acked Read 2 Sent Done Done false
15560 4943 Acked Read 10 Sent Done Done false
15561 4944 Acked Flush 0 Sent Done Done false
15562 4945 Acked Read 10 Sent Done Done false
15563 4946 Acked Write 4 Sent Done Done false
15564 4947 Acked Read 1 Sent Done Done false
15565 4948 Acked Write 9 Sent Done Done false
15566 4949 Acked Read 4 Sent Done Done false
15567 4950 Acked Write 1 Sent Done Done false
15568 4951 Acked Write 1 Sent Done Done false
15569 4952 Acked Read 1 Sent Done Done false
15570 4953 Acked Read 6 Sent Done Done false
15571 4954 Acked Write 7 Sent Done Done false
15572 4955 Acked Read 3 Sent Done Done false
15573 4956 Acked Read 4 Sent Done Done false
15574 4957 Acked Read 9 Sent Done Done false
15575 4958 Acked Write 1 Sent Done Done false
15576 4959 Acked Write 6 Sent Done Done false
15577 4960 Acked Write 6 Sent Done Done false
15578 4961 Acked Write 10 Sent Done Done false
15579 4962 Acked Read 4 Sent Done Done false
15580 4963 Acked Read 7 Sent Done Done false
15581 4964 Acked Write 4 Sent Done Done false
15582 4965 Acked Read 2 Sent Done Done false
15583 4966 Acked Write 3 Sent Done Done false
15584 4967 Acked Write 8 Sent Done Done false
15585 4968 Acked Read 2 Sent Done Done false
15586 4969 Acked Read 10 Sent Done Done false
15587 4970 Acked Flush 0 Sent Done Done false
15588 4971 Acked Write 7 Sent Done Done false
15589 4972 Acked Read 4 Sent Done Done false
15590 4973 Acked Flush 0 Sent Done Done false
15591 4974 Acked Read 2 Sent Done Done false
15592 4975 Acked Read 3 Sent Done Done false
15593 4976 Acked Read 5 Sent Done Done false
15594 4977 Acked Read 9 Sent Done Done false
15595 4978 Acked Write 7 Sent Done Done false
15596 4979 Acked Write 9 Sent Done Done false
15597 4980 Acked Write 4 Sent Done Done false
15598 4981 Acked Write 10 Sent Done Done false
15599 4982 Acked Write 3 Sent Done Done false
15600 4983 Acked Read 9 Sent Done Done false
15601 4984 Acked Read 8 Sent Done Done false
15602 4985 Acked Read 9 Sent Done Done false
15603 4986 Acked Read 10 Sent Done Done false
15604 4987 Acked Read 7 Sent Done Done false
15605 4988 Acked Read 7 Sent Done Done false
15606 4989 Acked Read 8 Sent Done Done false
15607 4990 Acked Write 4 Sent Done Done false
15608 4991 Acked Flush 0 Sent Done Done false
15609 4992 Acked Read 1 Sent Done Done false
15610 4993 Acked Read 7 Sent Done Done false
15611 4994 Acked Write 3 Sent Done Done false
15612 4995 Acked Write 9 Sent Done Done false
15613 4996 Acked Read 6 Sent Done Done false
15614 4997 Acked Read 9 Sent Done Done false
15615 4998 Acked Write 1 Sent Done Done false
15616 4999 Acked Read 1 Sent Done Done false
15617 5000 Acked Write 2 Sent Done Done false
15618 5001 Acked Read 3 Sent Done Done false
15619 5002 Acked Write 7 Sent Done Done false
15620 5003 Acked Read 6 Sent Done Done false
15621 5004 Acked Read 5 Sent Done Done false
15622 5005 Acked Read 5 Sent Done Done false
15623 5006 Acked Read 7 Sent Done Done false
15624 5007 Acked Flush 0 Sent Done Done false
15625 5008 Acked Write 10 Sent Done Done false
15626 5009 Acked Read 9 Sent Done Done false
15627 5010 Acked Read 2 Sent Done Done false
15628 5011 Acked Read 2 Sent Done Done false
15629 5012 Acked Write 10 Sent Done Done false
15630 5013 Acked Read 3 Sent Done Done false
15631 5014 Acked Read 3 Sent Done Done false
15632 5015 Acked Write 8 Sent Done Done false
15633 5016 Acked Write 7 Sent Done Done false
15634 5017 Acked Read 3 Sent Done Done false
15635 5018 Acked Read 9 Sent Done Done false
15636 5019 Acked Read 9 Sent Done Done false
15637 5020 Acked Flush 0 Sent Done Done false
15638 5021 Acked Read 8 Sent Done Done false
15639 5022 Acked Read 10 Sent Done Done false
15640 5023 Acked Read 6 Sent Done Done false
15641 5024 Acked Read 5 Sent Done Done false
15642 5025 Acked Read 9 Sent Done Done false
15643 5026 Acked Read 8 Sent Done Done false
15644 5027 Acked Read 1 Sent Done Done false
15645 5028 Acked Write 10 Sent Done Done false
15646 5029 Acked Write 2 Sent Done Done false
15647 5030 Acked Write 3 Sent Done Done false
15648 5031 Acked Read 2 Sent Done Done false
15649 5032 Acked Write 1 Sent Done Done false
15650 5033 Acked Write 5 Sent Done Done false
15651 5034 Acked Write 8 Sent Done Done false
15652 5035 Acked Read 8 Sent Done Done false
15653 5036 Acked Write 4 Sent Done Done false
15654 5037 Acked Flush 0 Sent Done Done false
15655 5038 Acked Write 2 Sent Done Done false
15656 5039 Acked Flush 0 Sent Done Done false
15657 5040 Acked Write 10 Sent Done Done false
15658 5041 Acked Read 9 Sent Done Done false
15659 5042 Acked Read 9 Sent Done Done false
15660 5043 Acked Read 1 Sent Done Done false
15661 5044 Acked Write 2 Sent Done Done false
15662 5045 Acked Write 5 Sent Done Done false
15663 5046 Acked Read 2 Sent Done Done false
15664 5047 Acked Write 2 Sent Done Done false
15665 5048 Acked Write 4 Sent Done Done false
15666 5049 Acked Read 1 Sent Done Done false
15667 5050 Acked Flush 0 Sent Done Done false
15668 5051 Acked Read 4 Sent Done Done false
15669 5052 Acked Read 2 Sent Done Done false
15670 5053 Acked Read 10 Sent Done Done false
15671 5054 Acked Flush 0 Sent Done Done false
15672 5055 Acked Write 4 Sent Done Done false
15673 5056 Acked Write 8 Sent Done Done false
15674 5057 Acked Read 8 Sent Done Done false
15675 5058 Acked Read 10 Sent Done Done false
15676 5059 Acked Write 1 Sent Done Done false
15677 5060 Acked Read 1 Sent Done Done false
15678 5061 Acked Read 1 Sent Done Done false
15679 5062 Acked Write 6 Sent Done Done false
15680 5063 Acked Read 3 Sent Done Done false
15681 5064 Acked Write 10 Sent Done Done false
15682 5065 Acked Read 2 Sent Done Done false
15683 5066 Acked Flush 0 Sent Done Done false
15684 5067 Acked Read 5 Sent Done Done false
15685 5068 Acked Read 6 Sent Done Done false
15686 5069 Acked Read 2 Sent Done Done false
15687 5070 Acked Write 1 Sent Done Done false
15688 5071 Acked Write 7 Sent Done Done false
15689 5072 Acked Read 9 Sent Done Done false
15690 5073 Acked Write 10 Sent Done Done false
15691 5074 Acked Read 9 Sent Done Done false
15692 5075 Acked Write 3 Sent Done Done false
15693 5076 Acked Write 3 Sent Done Done false
15694 5077 Acked Read 3 Sent Done Done false
15695 5078 Acked Read 5 Sent Done Done false
15696 5079 Acked Write 7 Sent Done Done false
15697 5080 Acked Flush 0 Sent Done Done false
15698 5081 Acked Flush 0 Sent Done Done false
15699 5082 Acked Write 1 Sent Done Done false
15700 5083 Acked Read 2 Sent Done Done false
15701 5084 Acked Flush 0 Sent Done Done false
15702 5085 Acked Read 10 Sent Done Done false
15703 5086 Acked Write 9 Sent Done Done false
15704 5087 Acked Write 9 Sent Done Done false
15705 5088 Acked Read 3 Sent Done Done false
15706 5089 Acked Write 4 Sent Done Done false
15707 5090 Acked Read 5 Sent Done Done false
15708 5091 Acked Read 3 Sent Done Done false
15709 5092 Acked Write 6 Sent Done Done false
15710 5093 Acked Read 1 Sent Done Done false
15711 5094 Acked Read 1 Sent Done Done false
15712 5095 Acked Write 8 Sent Done Done false
15713 5096 Acked Flush 0 Sent Done Done false
15714 5097 Acked Write 7 Sent Done Done false
15715 5098 Acked Read 8 Sent Done Done false
15716 5099 Acked Read 1 Sent Done Done false
15717 5100 Acked Read 1 Sent Done Done false
15718 5101 Acked Write 3 Sent Done Done false
15719 5102 Acked Write 1 Sent Done Done false
15720 5103 Acked Write 10 Sent Done Done false
15721 5104 Acked Read 8 Sent Done Done false
15722 5105 Acked Read 6 Sent Done Done false
15723 5106 Acked Read 3 Sent Done Done false
15724 5107 Acked Read 5 Sent Done Done false
15725 5108 Acked Read 10 Sent Done Done false
15726 5109 Acked Read 5 Sent Done Done false
15727 5110 Acked Write 5 Sent Done Done false
15728 5111 Acked Write 5 Sent Done Done false
15729 5112 Acked Write 7 Sent Done Done false
15730 5113 Acked Write 6 Sent Done Done false
15731 5114 Acked Read 9 Sent Done Done false
15732 5115 Acked Read 10 Sent Done Done false
15733 5116 Acked Read 2 Sent Done Done false
15734 5117 Acked Write 10 Sent Done Done false
15735 5118 Acked Write 7 Sent Done Done false
15736 5119 Acked Write 1 Sent Done Done false
15737 5120 Acked Write 3 Sent Done Done false
15738 5121 Acked Read 3 Sent Done Done false
15739 5122 Acked Read 10 Sent Done Done false
15740 5123 Acked Read 1 Sent Done Done false
15741 5124 Acked Read 5 Sent Done Done false
15742 5125 Acked Write 6 Sent Done Done false
15743 5126 Acked Read 10 Sent Done Done false
15744 5127 Acked Write 10 Sent Done Done false
15745 5128 Acked Read 3 Sent Done Done false
15746 5129 Acked Write 2 Sent Done Done false
15747 5130 Acked Write 5 Sent Done Done false
15748 5131 Acked Read 3 Sent Done Done false
15749 5132 Acked Read 3 Sent Done Done false
15750 5133 Acked Read 6 Sent Done Done false
15751 5134 Acked Read 1 Sent Done Done false
15752 5135 Acked Read 4 Sent Done Done false
15753 5136 Acked Read 9 Sent Done Done false
15754 5137 Acked Read 1 Sent Done Done false
15755 5138 Acked Flush 0 Sent Done Done false
15756 5139 Acked Read 10 Sent Done Done false
15757 5140 Acked Write 9 Sent Done Done false
15758 5141 Acked Read 3 Sent Done Done false
15759 5142 Acked Write 7 Sent Done Done false
15760 5143 Acked Read 5 Sent Done Done false
15761 5144 Acked Read 1 Sent Done Done false
15762 5145 Acked Read 4 Sent Done Done false
15763 5146 Acked Read 3 Sent Done Done false
15764 5147 Acked Write 4 Sent Done Done false
15765 5148 Acked Flush 0 Sent Done Done false
15766 5149 Acked Read 8 Sent Done Done false
15767 5150 Acked Read 3 Sent Done Done false
15768 5151 Acked Write 8 Sent Done Done false
15769 5152 Acked Read 5 Sent Done Done false
15770 5153 Acked Write 4 Sent Done Done false
15771 5154 Acked Write 6 Sent Done Done false
15772 5155 Acked Read 5 Sent Done Done false
15773 5156 Acked Read 10 Sent Done Done false
15774 5157 Acked Flush 0 Sent Done Done false
15775 5158 Acked Write 1 Sent Done Done false
15776 5159 Acked Read 2 Sent Done Done false
15777 5160 Acked Read 6 Sent Done Done false
15778 5161 Acked Read 6 Sent Done Done false
15779 5162 Acked Write 7 Sent Done Done false
15780 5163 Acked Read 10 Sent Done Done false
15781 5164 Acked Read 6 Sent Done Done false
15782 5165 Acked Read 3 Sent Done Done false
15783 5166 Acked Read 2 Sent Done Done false
15784 5167 Acked Read 2 Sent Done Done false
15785 5168 Acked Read 7 Sent Done Done false
15786 5169 Acked Write 2 Sent Done Done false
15787 5170 Acked Read 4 Sent Done Done false
15788 5171 Acked Read 6 Sent Done Done false
15789 5172 Acked Flush 0 Sent Done Done false
15790 5173 Acked Write 5 Sent Done Done false
15791 5174 Acked Read 9 Sent Done Done false
15792 5175 Acked Read 1 Sent Done Done false
15793 5176 Acked Read 3 Sent Done Done false
15794 5177 Acked Read 10 Sent Done Done false
15795 5178 Acked Write 9 Sent Done Done false
15796 5179 Acked Flush 0 Sent Done Done false
15797 5180 Acked Read 3 Sent Done Done false
15798 5181 Acked Read 3 Sent Done Done false
15799 5182 Acked Read 7 Sent Done Done false
15800 5183 Acked Read 7 Sent Done Done false
15801 5184 Acked Read 8 Sent Done Done false
15802 5185 Acked Read 3 Sent Done Done false
15803 5186 Acked Read 2 Sent Done Done false
15804 5187 Acked Write 9 Sent Done Done false
15805 5188 Acked Write 1 Sent Done Done false
15806 5189 Acked Read 8 Sent Done Done false
15807 5190 Acked Write 6 Sent Done Done false
15808 5191 Acked Read 4 Sent Done Done false
15809 5192 Acked Flush 0 Sent Done Done false
15810 5193 Acked Write 9 Sent Done Done false
15811 5194 Acked Write 7 Sent Done Done false
15812 5195 Acked Read 7 Sent Done Done false
15813 5196 Acked Write 10 Sent Done Done false
15814 5197 Acked Read 1 Sent Done Done false
15815 5198 Acked Read 9 Sent Done Done false
15816 5199 Acked Read 5 Sent Done Done false
15817 5200 Acked Write 3 Sent Done Done false
15818 5201 Acked Read 6 Sent Done Done false
15819 5202 Acked Write 8 Sent Done Done false
15820 5203 Acked Write 1 Sent Done Done false
15821 5204 Acked Flush 0 Sent Done Done false
15822 5205 Acked Write 5 Sent Done Done false
15823 5206 Acked Read 9 Sent Done Done false
15824 5207 Acked Write 8 Sent Done Done false
15825 5208 Acked Read 4 Sent Done Done false
15826 5209 Acked Write 7 Sent Done Done false
15827 5210 Acked Read 2 Sent Done Done false
15828 5211 Acked Write 10 Sent Done Done false
15829 5212 Acked Write 10 Sent Done Done false
15830 5213 Acked Read 5 Sent Done Done false
15831 5214 Acked Read 1 Sent Done Done false
15832 5215 Acked Write 7 Sent Done Done false
15833 5216 Acked Read 6 Sent Done Done false
15834 5217 Acked Read 2 Sent Done Done false
15835 5218 Acked Write 5 Sent Done Done false
15836 5219 Acked Flush 0 Sent Done Done false
15837 5220 Acked Read 9 Sent Done Done false
15838 5221 Acked Write 10 Sent Done Done false
15839 5222 Acked Read 5 Sent Done Done false
15840 5223 Acked Write 7 Sent Done Done false
15841 5224 Acked Read 10 Sent Done Done false
15842 5225 Acked Read 2 Sent Done Done false
15843 STATES DS:0 DS:1 DS:2 TOTAL
15844 Sent 301 0 0 301
15845 Done 0 301 301 602
15846 Skipped 0 0 0 0
15847 Error 0 0 0 0
15848 Last Flush: Some(JobId(4924)) Some(JobId(5219)) Some(JobId(5219))
15849 Downstairs last five completed: 4924 4923 4922 4921 4920
15850 Upstairs last five completed: 5225 5224 5223 5222 5221
15851 CLIENT: Up:0 ds:301 act:2
158522025-03-17T22:19:43.261ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
158532025-03-17T22:19:43.917ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
158542025-03-17T22:19:44.500ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
158552025-03-17T22:19:44.500ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:41658) = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
158562025-03-17T22:19:44.501ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
158572025-03-17T22:19:44.502ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
158582025-03-17T22:19:44.502ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
158592025-03-17T22:19:44.502ZINFOcrucible: send last flush ID to this DS: Some(JobId(4924)) = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
158602025-03-17T22:19:44.502ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
158612025-03-17T22:19:44.502ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(4924)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
158622025-03-17T22:19:44.502ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(4924)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
15863 ----------------------------------------------------------------
15864 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
15865 Downstairs last five completed: 5226 5225 5224 5223 5222
15866 Upstairs last five completed: 5226 5225 5224 5223 5222
15867 CLIENT: Up:0 ds:0 act:3
15868 Replay: All jobs finished
15869 Main task has work to do, go find it
15870 got dsc Stop(2)
15871 stop 2
15872 Mar 17 22:19:46.762 INFO[8830] Got stop action so:false kr:false
15873 request completed, latency_us: 370, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 2b9a0952-431f-479d-a255-9c32a5921dbf, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
158742025-03-17T22:19:46.762ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
158752025-03-17T22:19:46.762ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
158762025-03-17T22:19:46.762ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
15877 Mar 17 22:19:46.762 INFO request completed, latency_us: 133, response_code: 200, uri: /state/cid/2, method: GET, req_id: cb81b19b-98a6-412d-9891-b7689cd47ea3, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
158782025-03-17T22:19:46.763ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
15879 [8830] Exited with: ExitStatus(unix_wait_status(9))
15880 [8830][2] reports Exit
15881 Mar 17 22:19:50.764 INFO request completed, latency_us: 217, response_code: 200, uri: /state/cid/2, method: GET, req_id: 9212fa33-4e4d-49a2-9eb3-c441ae5934da, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
15882 001/300 Flush
15883 002/300 Flush
15884 003/300 Read block 2974 len 40960
15885 004/300 Read block 4911 len 4096
15886 005/300 Read block 2126 len 4096
15887 006/300 Write block 4049 len 8192 data: 3 3
15888 007/300 Write block 3269 len 32768 data: 4 4 3 3 3 3 4 4
15889 008/300 Write block 1965 len 4096 data: 7
15890 009/300 Write block 1639 len 36864 data: 6 7 7 6 4 4 5 4 4
15891 010/300 Read block 2792 len 4096
15892 011/300 Read block 3167 len 28672
15893 012/300 Read block 3218 len 12288
15894 013/300 Write block 4545 len 24576 data: 4 3 5 4 5 5
15895 014/300 Read block 2007 len 36864
15896 015/300 Write block 4815 len 24576 data: 3 3 3 3 3 3
15897 016/300 Read block 2915 len 36864
15898 017/300 Write block 1884 len 12288 data: 4 4 4
15899 018/300 Read block 3439 len 36864
15900 019/300 Write block 3500 len 8192 data: 3 3
15901 020/300 Write block 659 len 16384 data: 5 4 3 3
15902 021/300 Write block 3254 len 20480 data: 3 3 3 3 3
15903 022/300 Read block 3725 len 36864
15904 023/300 Read block 4018 len 8192
15905 024/300 Read block 1690 len 24576
15906 025/300 Write block 4051 len 12288 data: 3 3 4
15907 026/300 Read block 4712 len 24576
15908 027/300 Read block 4279 len 8192
15909 028/300 Write block 1932 len 4096 data: 4
15910 029/300 Read block 2955 len 24576
15911 030/300 Flush
15912 031/300 Read block 161 len 16384
15913 032/300 Read block 520 len 24576
15914 033/300 Read block 4626 len 4096
15915 034/300 Write block 396 len 36864 data: 3 3 5 5 5 5 4 4 4
15916 035/300 Read block 291 len 16384
15917 036/300 Write block 3562 len 20480 data: 3 3 4 4 4
15918 037/300 Read block 2679 len 12288
15919 038/300 Flush
15920 039/300 Read block 4770 len 40960
15921 040/300 Write block 2388 len 36864 data: 3 3 3 3 2 2 2 2 2
15922 041/300 Write block 1166 len 32768 data: 4 4 4 4 4 3 3 3
15923 042/300 Read block 2277 len 8192
15924 043/300 Read block 108 len 40960
15925 044/300 Write block 3119 len 28672 data: 4 3 3 3 3 3 3
15926 045/300 Read block 1535 len 36864
15927 046/300 Read block 125 len 24576
15928 047/300 Write block 641 len 12288 data: 3 3 3
15929 048/300 Write block 4083 len 20480 data: 5 5 6 6 7
15930 049/300 Read block 760 len 12288
15931 050/300 Flush
15932 051/300 Read block 4550 len 16384
15933 052/300 Flush
15934 053/300 Read block 941 len 4096
15935 054/300 Write block 2270 len 24576 data: 4 3 3 3 4 5
15936 055/300 Write block 912 len 36864 data: 3 3 3 3 2 2 2 2 2
15937 056/300 Write block 1082 len 40960 data: 3 3 2 3 4 3 3 4 4 4
15938 057/300 Write block 4545 len 4096 data: 5
15939 058/300 Write block 226 len 4096 data: 2
15940 059/300 Read block 2980 len 40960
15941 060/300 Write block 3029 len 8192 data: 2 2
15942 061/300 Write block 1777 len 16384 data: 4 4 3 3
15943 062/300 Write block 1048 len 24576 data: 3 4 4 4 3 3
15944 063/300 Write block 724 len 16384 data: 3 3 4 3
15945 064/300 Read block 259 len 36864
15946 065/300 Flush
15947 066/300 Write block 2069 len 12288 data: 4 4 5
15948 067/300 Read block 1550 len 4096
15949 068/300 Write block 3613 len 24576 data: 5 4 4 4 4 4
15950 069/300 Read block 2600 len 40960
15951 070/300 Flush
15952 071/300 Read block 2765 len 8192
15953 072/300 Read block 1506 len 4096
15954 073/300 Read block 4883 len 32768
15955 074/300 Write block 4419 len 28672 data: 4 5 8 7 7 6 5
15956 075/300 Write block 1495 len 16384 data: 2 2 2 2
15957 076/300 Read block 3483 len 32768
15958 077/300 Write block 2862 len 32768 data: 3 4 4 3 3 2 4 4
15959 078/300 Write block 3281 len 12288 data: 4 3 4
15960 079/300 Read block 4151 len 24576
15961 080/300 Read block 3228 len 4096
15962 081/300 Write block 4745 len 36864 data: 4 5 5 5 5 4 5 5 4
15963 082/300 Read block 91 len 32768
15964 083/300 Read block 1557 len 36864
15965 084/300 Write block 1288 len 24576 data: 4 4 2 3 3 2
15966 085/300 Read block 946 len 20480
15967 086/300 Write block 1714 len 24576 data: 3 3 3 3 3 3
15968 087/300 Write block 4140 len 28672 data: 3 3 3 3 3 3 4
15969 088/300 Read block 3079 len 20480
15970 089/300 Write block 2178 len 36864 data: 7 5 4 5 4 4 4 4 4
15971 090/300 Write block 4945 len 40960 data: 3 4 4 3 4 5 4 3 3 3
15972 091/300 Read block 2417 len 8192
15973 092/300 Read block 4910 len 24576
15974 093/300 Flush
15975 094/300 Write block 3459 len 12288 data: 5 6 6
15976 095/300 Write block 3199 len 36864 data: 4 3 3 4 4 4 4 4 4
15977 096/300 Read block 3146 len 24576
15978 097/300 Write block 2272 len 12288 data: 4 4 5
15979 098/300 Write block 4184 len 32768 data: 3 2 3 3 2 2 2 2
15980 099/300 Read block 2795 len 12288
15981 100/300 Write block 1324 len 20480 data: 3 3 5 5 5
15982 101/300 Write block 2315 len 8192 data: 3 3
15983 102/300 Write block 1527 len 36864 data: 9 8 7 7 7 6 7 6 5
15984 103/300 Read block 2981 len 8192
15985 104/300 Read block 2796 len 12288
15986 105/300 Write block 3247 len 36864 data: 4 4 4 4 4 3 3 4 4
15987 106/300 Write block 2548 len 40960 data: 5 5 5 3 3 3 3 4 4 4
15988 107/300 Read block 4871 len 40960
15989 108/300 Write block 4245 len 16384 data: 3 3 4 4
15990 109/300 Read block 2986 len 16384
15991 110/300 Write block 4437 len 16384 data: 4 5 5 5
15992 111/300 Read block 3022 len 40960
15993 112/300 Read block 2928 len 4096
15994 113/300 Read block 1422 len 36864
15995 114/300 Read block 1261 len 16384
15996 115/300 Write block 479 len 24576 data: 2 3 3 4 4 6
15997 116/300 Write block 3162 len 16384 data: 3 3 3 3
15998 117/300 Read block 2841 len 36864
15999 118/300 Read block 2304 len 32768
16000 119/300 Read block 3121 len 36864
16001 120/300 Write block 4720 len 24576 data: 5 6 6 7 7 5
16002 121/300 Read block 3604 len 36864
16003 122/300 Write block 685 len 4096 data: 5
16004 123/300 Write block 1233 len 28672 data: 2 3 3 3 3 3 2
16005 124/300 Read block 2213 len 36864
16006 125/300 Write block 3017 len 16384 data: 8 8 7 5
16007 126/300 Flush
16008 127/300 Read block 3818 len 8192
16009 128/300 Read block 2739 len 20480
16010 129/300 Write block 1881 len 36864 data: 4 6 4 5 5 5 4 4 3
16011 130/300 Flush
16012 131/300 Write block 2598 len 12288 data: 5 6 5
16013 132/300 Read block 1208 len 20480
16014 133/300 Read block 2757 len 36864
16015 134/300 Write block 1999 len 8192 data: 4 4
16016 135/300 Write block 4258 len 20480 data: 5 5 4 4 3
16017 136/300 Read block 3495 len 28672
16018 137/300 Write block 1373 len 36864 data: 4 4 4 3 3 3 3 3 3
16019 138/300 Read block 722 len 32768
16020 139/300 Write block 3901 len 16384 data: 7 6 3 3
16021 140/300 Read block 736 len 32768
16022 141/300 Read block 3217 len 16384
16023 142/300 Read block 4053 len 32768
16024 143/300 Write block 3026 len 4096 data: 4
16025 144/300 Write block 3621 len 36864 data: 2 2 2 2 2 2 2 2 2
16026 145/300 Write block 3383 len 28672 data: 2 2 3 4 5 5 5
16027 146/300 Read block 2386 len 32768
16028 147/300 Write block 4388 len 8192 data: 2 2
16029 148/300 Read block 2813 len 32768
16030 149/300 Read block 3177 len 36864
16031 150/300 Write block 318 len 8192 data: 6 6
16032 151/300 Read block 1191 len 40960
16033 152/300 Read block 4384 len 8192
16034 153/300 Read block 1553 len 40960
16035 154/300 Read block 164 len 32768
16036 155/300 Flush
16037 156/300 Write block 3225 len 32768 data: 4 4 5 4 4 5 7 7
16038 157/300 Flush
16039 158/300 Read block 2136 len 40960
16040 159/300 Flush
16041 160/300 Read block 4879 len 4096
16042 161/300 Flush
16043 162/300 Read block 2817 len 32768
16044 163/300 Read block 624 len 40960
16045 164/300 Read block 3922 len 36864
16046 165/300 Flush
16047 166/300 Read block 476 len 12288
16048 167/300 Read block 191 len 40960
16049 168/300 Read block 730 len 24576
16050 169/300 Write block 1308 len 28672 data: 4 4 4 4 4 4 4
16051 170/300 Read block 4425 len 4096
16052 171/300 Write block 850 len 40960 data: 4 5 4 4 5 4 3 4 5 4
16053 172/300 Write block 4167 len 32768 data: 3 4 4 4 4 4 3 3
16054 173/300 Write block 849 len 32768 data: 4 5 6 5 5 6 5 4
16055 174/300 Read block 4269 len 36864
16056 175/300 Read block 635 len 12288
16057 176/300 Write block 2966 len 32768 data: 6 7 7 5 3 3 3 5
16058 177/300 Write block 1676 len 24576 data: 5 4 3 3 5 5
16059 178/300 Read block 2051 len 24576
16060 179/300 Read block 3672 len 32768
16061 180/300 Read block 4279 len 4096
16062 181/300 Write block 4066 len 32768 data: 3 4 5 4 3 3 3 2
16063 182/300 Read block 4171 len 4096
16064 183/300 Read block 1418 len 28672
16065 184/300 Write block 555 len 16384 data: 4 5 7 7
16066 185/300 Write block 2845 len 32768 data: 6 7 6 5 4 5 4 4
16067 186/300 Read block 4497 len 20480
16068 187/300 Flush
16069 188/300 Read block 2109 len 32768
16070 189/300 Read block 1006 len 4096
16071 190/300 Write block 1597 len 8192 data: 6 7
16072 191/300 Write block 4062 len 16384 data: 5 5 5 3
16073 192/300 Write block 125 len 28672 data: 3 4 4 4 4 5 5
16074 193/300 Read block 2150 len 40960
16075 194/300 Flush
16076 195/300 Write block 4663 len 16384 data: 3 2 2 3
16077 196/300 Read block 4180 len 32768
16078 197/300 Read block 1824 len 36864
16079 198/300 Read block 1685 len 12288
16080 199/300 Read block 3737 len 36864
16081 200/300 Write block 3836 len 8192 data: 3 3
16082 201/300 Flush
16083 202/300 Read block 3198 len 12288
16084 203/300 Write block 245 len 24576 data: 2 2 3 3 3 4
16085 204/300 Read block 342 len 36864
16086 205/300 Flush
16087 206/300 Read block 4074 len 16384
16088 207/300 Flush
16089 208/300 Read block 1387 len 20480
16090 209/300 Write block 575 len 32768 data: 2 3 3 3 3 3 3 3
16091 210/300 Write block 3352 len 40960 data: 4 5 6 6 6 4 4 4 3 3
16092 211/300 Write block 1512 len 28672 data: 3 3 2 2 2 2 3
16093 212/300 Read block 2453 len 4096
16094 213/300 Read block 2353 len 8192
16095 214/300 Read block 2387 len 4096
16096 215/300 Flush
16097 216/300 Read block 4745 len 40960
16098 217/300 Read block 746 len 32768
16099 218/300 Write block 1166 len 32768 data: 5 5 5 5 5 4 4 4
16100 219/300 Read block 1296 len 12288
16101 220/300 Write block 4811 len 24576 data: 6 5 4 3 4 4
16102 221/300 Write block 2849 len 4096 data: 5
16103 222/300 Read block 1235 len 20480
16104 223/300 Read block 1987 len 16384
16105 224/300 Write block 1521 len 24576 data: 3 6 6 7 8 8
16106 225/300 Write block 2058 len 16384 data: 5 7 7 7
16107 226/300 Read block 4783 len 20480
16108 227/300 Flush
16109 228/300 Write block 67 len 8192 data: 4 4
16110 229/300 Write block 3708 len 12288 data: 3 3 4
16111 230/300 Read block 222 len 40960
16112 231/300 Read block 2580 len 24576
16113 232/300 Write block 1602 len 4096 data: 4
16114 233/300 Write block 3386 len 12288 data: 5 6 6
16115 234/300 Read block 2814 len 12288
16116 235/300 Write block 1178 len 16384 data: 6 5 5 5
16117 236/300 Write block 4917 len 24576 data: 5 5 5 5 6 6
16118 237/300 Write block 3557 len 40960 data: 3 3 4 4 4 4 4 5 5 5
16119 238/300 Write block 3912 len 8192 data: 2 3
16120 239/300 Read block 3856 len 24576
16121 240/300 Read block 232 len 16384
16122 241/300 Read block 1661 len 28672
16123 242/300 Write block 1381 len 12288 data: 4 3 2
16124 243/300 Read block 2401 len 12288
16125 244/300 Read block 3662 len 20480
16126 245/300 Read block 282 len 28672
16127 246/300 Flush
16128 247/300 Read block 4524 len 36864
16129 248/300 Read block 4270 len 12288
16130 249/300 Read block 3597 len 16384
16131 250/300 Write block 2041 len 36864 data: 5 5 4 4 3 4 4 5 5
16132 251/300 Flush
16133 252/300 Read block 2461 len 40960
16134 253/300 Read block 2525 len 28672
16135 254/300 Write block 122 len 12288 data: 4 4 4
16136 255/300 Write block 2896 len 24576 data: 3 4 4 3 4 3
16137 256/300 Write block 2007 len 8192 data: 2 2
16138 257/300 Read block 959 len 40960
16139 258/300 Read block 4224 len 8192
16140 259/300 Write block 4349 len 4096 data: 3
16141 260/300 Write block 1066 len 8192 data: 5 4
16142 261/300 Write block 811 len 4096 data: 3
16143 262/300 Write block 4446 len 40960 data: 3 3 4 5 5 5 5 4 5 4
16144 263/300 Read block 3500 len 12288
16145 264/300 Read block 4967 len 28672
16146 265/300 Read block 3452 len 8192
16147 266/300 Write block 848 len 12288 data: 3 5 6
16148 267/300 Read block 3919 len 16384
16149 268/300 Write block 1473 len 24576 data: 4 4 4 4 6 7
16150 269/300 Read block 3518 len 28672
16151 270/300 Read block 3323 len 36864
16152 271/300 Write block 2130 len 32768 data: 3 5 6 6 5 5 5 5
16153 272/300 Read block 1938 len 32768
16154 273/300 Read block 2027 len 8192
16155 274/300 Write block 2674 len 32768 data: 5 5 5 4 4 2 2 2
16156 275/300 Write block 4886 len 8192 data: 8 6
16157 276/300 Write block 2936 len 4096 data: 5
16158 277/300 Read block 1530 len 20480
16159 278/300 Read block 2734 len 8192
16160 279/300 Read block 2806 len 4096
16161 280/300 Write block 1162 len 28672 data: 5 4 3 4 6 6 6
16162 281/300 Flush
16163 282/300 Read block 883 len 40960
16164 283/300 Write block 2495 len 24576 data: 2 2 2 2 2 2
16165 284/300 Read block 3478 len 12288
16166 285/300 Read block 708 len 20480
16167 286/300 Flush
16168 287/300 Read block 1776 len 8192
16169 288/300 Flush
16170 289/300 Read block 716 len 20480
16171 290/300 Read block 3331 len 40960
16172 291/300 Read block 3939 len 40960
16173 292/300 Read block 372 len 28672
16174 293/300 Read block 2395 len 32768
16175 294/300 Read block 4904 len 20480
16176 295/300 Read block 3407 len 36864
16177 296/300 Read block 3561 len 12288
16178 297/300 Write block 4297 len 20480 data: 4 5 6 6 8
16179 298/300 Flush
16180 299/300 Read block 2362 len 32768
16181 300/300 Write block 2843 len 40960 data: 3 4 7 8 7 6 6 6 5 5
16182 301/300 Write block 4092 len 28672 data: 3 4 4 5 4 3 4
16183 Main task has work to do, go find it
16184 got dsc Start(2)
16185 start 2
16186 Mar 17 22:19:51.018 INFO request completed, latency_us: 217, response_code: 204, uri: /start/cid/2, method: POST, req_id: ced3626d-4a13-49dc-871b-98eaec67c42c, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
16187 Replay: started 2, returned:Ok(())
16188 ----------------------------------------------------------------
16189 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16190 States: Active Active Offline
16191 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16192 5227 Acked Flush 0 Done Done Sent false
16193 5228 Acked Flush 0 Done Done Sent false
16194 5229 Acked Read 10 Done Done Sent false
16195 5230 Acked Read 1 Done Done Sent false
16196 5231 Acked Read 1 Done Done Sent false
16197 5232 Acked Write 2 Done Done Sent false
16198 5233 Acked Write 8 Done Done Sent false
16199 5234 Acked Write 1 Done Done Sent false
16200 5235 Acked Write 9 Done Done Sent false
16201 5236 Acked Read 1 Done Done Sent false
16202 5237 Acked Read 7 Done Done Sent false
16203 5238 Acked Read 3 Done Done Sent false
16204 5239 Acked Write 6 Done Done Sent false
16205 5240 Acked Read 9 Done Done Sent false
16206 5241 Acked Write 6 Done Done Sent false
16207 5242 Acked Read 9 Done Done Sent false
16208 5243 Acked Write 3 Done Done Sent false
16209 5244 Acked Read 9 Done Done Sent false
16210 5245 Acked Write 2 Done Done Sent false
16211 5246 Acked Write 4 Done Done Sent false
16212 5247 Acked Write 5 Done Done Sent false
16213 5248 Acked Read 9 Done Done Sent false
16214 5249 Acked Read 2 Done Done Sent false
16215 5250 Acked Read 6 Done Done Sent false
16216 5251 Acked Write 3 Done Done Sent false
16217 5252 Acked Read 6 Done Done Sent false
16218 5253 Acked Read 2 Done Done Sent false
16219 5254 Acked Write 1 Done Done Sent false
16220 5255 Acked Read 6 Done Done Sent false
16221 5256 Acked Flush 0 Done Done Sent false
16222 5257 Acked Read 4 Done Done Sent false
16223 5258 Acked Read 6 Done Done Sent false
16224 5259 Acked Read 1 Done Done Sent false
16225 5260 Acked Write 9 Done Done Sent false
16226 5261 Acked Read 4 Done Done Sent false
16227 5262 Acked Write 5 Done Done Sent false
16228 5263 Acked Read 3 Done Done Sent false
16229 5264 Acked Flush 0 Done Done Sent false
16230 5265 Acked Read 10 Done Done Sent false
16231 5266 Acked Write 9 Done Done Sent false
16232 5267 Acked Write 8 Done Done Sent false
16233 5268 Acked Read 2 Done Done Sent false
16234 5269 Acked Read 10 Done Done Sent false
16235 5270 Acked Write 7 Done Done Sent false
16236 5271 Acked Read 9 Done Done Sent false
16237 5272 Acked Read 6 Done Done Sent false
16238 5273 Acked Write 3 Done Done Sent false
16239 5274 Acked Write 5 Done Done Sent false
16240 5275 Acked Read 3 Done Done Sent false
16241 5276 Acked Flush 0 Done Done Sent false
16242 5277 Acked Read 4 Done Done Sent false
16243 5278 Acked Flush 0 Done Done Sent false
16244 5279 Acked Read 1 Done Done Sent false
16245 5280 Acked Write 6 Done Done Sent false
16246 5281 Acked Write 9 Done Done Sent false
16247 5282 Acked Write 10 Done Done Sent false
16248 5283 Acked Write 1 Done Done Sent false
16249 5284 Acked Write 1 Done Done Sent false
16250 5285 Acked Read 10 Done Done Sent false
16251 5286 Acked Write 2 Done Done Sent false
16252 5287 Acked Write 4 Done Done Sent false
16253 5288 Acked Write 6 Done Done Sent false
16254 5289 Acked Write 4 Done Done Sent false
16255 5290 Acked Read 9 Done Done Sent false
16256 5291 Acked Flush 0 Done Done Sent false
16257 5292 Acked Write 3 Done Done Sent false
16258 5293 Acked Read 1 Done Done Sent false
16259 5294 Acked Write 6 Done Done Sent false
16260 5295 Acked Read 10 Done Done Sent false
16261 5296 Acked Flush 0 Done Done Sent false
16262 5297 Acked Read 2 Done Done Sent false
16263 5298 Acked Read 1 Done Done Sent false
16264 5299 Acked Read 8 Done Done Sent false
16265 5300 Acked Write 7 Done Done Sent false
16266 5301 Acked Write 4 Done Done Sent false
16267 5302 Acked Read 8 Done Done Sent false
16268 5303 Acked Write 8 Done Done Sent false
16269 5304 Acked Write 3 Done Done Sent false
16270 5305 Acked Read 6 Done Done Sent false
16271 5306 Acked Read 1 Done Done Sent false
16272 5307 Acked Write 9 Done Done Sent false
16273 5308 Acked Read 8 Done Done Sent false
16274 5309 Acked Read 9 Done Done Sent false
16275 5310 Acked Write 6 Done Done Sent false
16276 5311 Acked Read 5 Done Done Sent false
16277 5312 Acked Write 6 Done Done Sent false
16278 5313 Acked Write 7 Done Done Sent false
16279 5314 Acked Read 5 Done Done Sent false
16280 5315 Acked Write 9 Done Done Sent false
16281 5316 Acked Write 10 Done Done Sent false
16282 5317 Acked Read 2 Done Done Sent false
16283 5318 Acked Read 6 Done Done Sent false
16284 5319 Acked Flush 0 Done Done Sent false
16285 5320 Acked Write 3 Done Done Sent false
16286 5321 Acked Write 9 Done Done Sent false
16287 5322 Acked Read 6 Done Done Sent false
16288 5323 Acked Write 3 Done Done Sent false
16289 5324 Acked Write 8 Done Done Sent false
16290 5325 Acked Read 3 Done Done Sent false
16291 5326 Acked Write 5 Done Done Sent false
16292 5327 Acked Write 2 Done Done Sent false
16293 5328 Acked Write 9 Done Done Sent false
16294 5329 Acked Read 2 Done Done Sent false
16295 5330 Acked Read 3 Done Done Sent false
16296 5331 Acked Write 9 Done Done Sent false
16297 5332 Acked Write 10 Done Done Sent false
16298 5333 Acked Read 10 Done Done Sent false
16299 5334 Acked Write 4 Done Done Sent false
16300 5335 Acked Read 4 Done Done Sent false
16301 5336 Acked Write 4 Done Done Sent false
16302 5337 Acked Read 10 Done Done Sent false
16303 5338 Acked Read 1 Done Done Sent false
16304 5339 Acked Read 9 Done Done Sent false
16305 5340 Acked Read 4 Done Done Sent false
16306 5341 Acked Write 6 Done Done Sent false
16307 5342 Acked Write 4 Done Done Sent false
16308 5343 Acked Read 9 Done Done Sent false
16309 5344 Acked Read 8 Done Done Sent false
16310 5345 Acked Read 9 Done Done Sent false
16311 5346 Acked Write 6 Done Done Sent false
16312 5347 Acked Read 9 Done Done Sent false
16313 5348 Acked Write 1 Done Done Sent false
16314 5349 Acked Write 7 Done Done Sent false
16315 5350 Acked Read 9 Done Done Sent false
16316 5351 Acked Write 4 Done Done Sent false
16317 5352 Acked Flush 0 Done Done Sent false
16318 5353 Acked Read 2 Done Done Sent false
16319 5354 Acked Read 5 Done Done Sent false
16320 5355 Acked Write 9 Done Done Sent false
16321 5356 Acked Flush 0 Done Done Sent false
16322 5357 Acked Write 3 Done Done Sent false
16323 5358 Acked Read 5 Done Done Sent false
16324 5359 Acked Read 9 Done Done Sent false
16325 5360 Acked Write 2 Done Done Sent false
16326 5361 Acked Write 5 Done Done Sent false
16327 5362 Acked Read 7 Done Done Sent false
16328 5363 Acked Write 9 Done Done Sent false
16329 5364 Acked Read 8 Done Done Sent false
16330 5365 Acked Write 4 Done Done Sent false
16331 5366 Acked Read 8 Done Done Sent false
16332 5367 Acked Read 4 Done Done Sent false
16333 5368 Acked Read 8 Done Done Sent false
16334 5369 Acked Write 1 Done Done Sent false
16335 5370 Acked Write 9 Done Done Sent false
16336 5371 Acked Write 7 Done Done Sent false
16337 5372 Acked Read 8 Done Done Sent false
16338 5373 Acked Write 2 Done Done Sent false
16339 5374 Acked Read 8 Done Done Sent false
16340 5375 Acked Read 9 Done Done Sent false
16341 5376 Acked Write 2 Done Done Sent false
16342 5377 Acked Read 10 Done Done Sent false
16343 5378 Acked Read 2 Done Done Sent false
16344 5379 Acked Read 10 Done Done Sent false
16345 5380 Acked Read 8 Done Done Sent false
16346 5381 Acked Flush 0 Done Done Sent false
16347 5382 Acked Write 8 Done Done Sent false
16348 5383 Acked Flush 0 Done Done Sent false
16349 5384 Acked Read 10 Done Done Sent false
16350 5385 Acked Flush 0 Done Done Sent false
16351 5386 Acked Read 1 Done Done Sent false
16352 5387 Acked Flush 0 Done Done Sent false
16353 5388 Acked Read 8 Done Done Sent false
16354 5389 Acked Read 10 Done Done Sent false
16355 5390 Acked Read 9 Done Done Sent false
16356 5391 Acked Flush 0 Done Done Sent false
16357 5392 Acked Read 3 Done Done Sent false
16358 5393 Acked Read 10 Done Done Sent false
16359 5394 Acked Read 6 Done Done Sent false
16360 5395 Acked Write 7 Done Done Sent false
16361 5396 Acked Read 1 Done Done Sent false
16362 5397 Acked Write 10 Done Done Sent false
16363 5398 Acked Write 8 Done Done Sent false
16364 5399 Acked Write 8 Done Done Sent false
16365 5400 Acked Read 9 Done Done Sent false
16366 5401 Acked Read 3 Done Done Sent false
16367 5402 Acked Write 8 Done Done Sent false
16368 5403 Acked Write 6 Done Done Sent false
16369 5404 Acked Read 6 Done Done Sent false
16370 5405 Acked Read 8 Done Done Sent false
16371 5406 Acked Read 1 Done Done Sent false
16372 5407 Acked Write 8 Done Done Sent false
16373 5408 Acked Read 1 Done Done Sent false
16374 5409 Acked Read 7 Done Done Sent false
16375 5410 Acked Write 4 Done Done Sent false
16376 5411 Acked Write 8 Done Done Sent false
16377 5412 Acked Read 5 Done Done Sent false
16378 5413 Acked Flush 0 Done Done Sent false
16379 5414 Acked Read 8 Done Done Sent false
16380 5415 Acked Read 1 Done Done Sent false
16381 5416 Acked Write 2 Done Done Sent false
16382 5417 Acked Write 4 Done Done Sent false
16383 5418 Acked Write 7 Done Done Sent false
16384 5419 Acked Read 10 Done Done Sent false
16385 5420 Acked Flush 0 Done Done Sent false
16386 5421 Acked Write 4 Done Done Sent false
16387 5422 Acked Read 8 Done Done Sent false
16388 5423 Acked Read 9 Done Done Sent false
16389 5424 Acked Read 3 Done Done Sent false
16390 5425 Acked Read 9 Done Done Sent false
16391 5426 Acked Write 2 Done Done Sent false
16392 5427 Acked Flush 0 Done Done Sent false
16393 5428 Acked Read 3 Done Done Sent false
16394 5429 Acked Write 6 Done Done Sent false
16395 5430 Acked Read 9 Done Done Sent false
16396 5431 Acked Flush 0 Done Done Sent false
16397 5432 Acked Read 4 Done Done Sent false
16398 5433 Acked Flush 0 Done Done Sent false
16399 5434 Acked Read 5 Done Done Sent false
16400 5435 Acked Write 8 Done Done Sent false
16401 5436 Acked Write 10 Done Done Sent false
16402 5437 Acked Write 7 Done Done Sent false
16403 5438 Acked Read 1 Done Done Sent false
16404 5439 Acked Read 2 Done Done Sent false
16405 5440 Acked Read 1 Done Done Sent false
16406 5441 Acked Flush 0 Done Done Sent false
16407 5442 Acked Read 10 Done Done Sent false
16408 5443 Acked Read 8 Done Done Sent false
16409 5444 Acked Write 8 Done Done Sent false
16410 5445 Acked Read 3 Done Done Sent false
16411 5446 Acked Write 6 Done Done Sent false
16412 5447 Acked Write 1 Done Done Sent false
16413 5448 Acked Read 5 Done Done Sent false
16414 5449 Acked Read 4 Done Done Sent false
16415 5450 Acked Write 6 Done Done Sent false
16416 5451 Acked Write 4 Done Done Sent false
16417 5452 Acked Read 5 Done Done Sent false
16418 5453 Acked Flush 0 Done Done Sent false
16419 5454 Acked Write 2 Done Done Sent false
16420 5455 Acked Write 3 Done Done Sent false
16421 5456 Acked Read 10 Done Done Sent false
16422 5457 Acked Read 6 Done Done Sent false
16423 5458 Acked Write 1 Done Done Sent false
16424 5459 Acked Write 3 Done Done Sent false
16425 5460 Acked Read 3 Done Done Sent false
16426 5461 Acked Write 4 Done Done Sent false
16427 5462 Acked Write 6 Done Done Sent false
16428 5463 Acked Write 10 Done Done Sent false
16429 5464 Acked Write 2 Done Done Sent false
16430 5465 Acked Read 6 Done Done Sent false
16431 5466 Acked Read 4 Done Done Sent false
16432 5467 Acked Read 7 Done Done Sent false
16433 5468 Acked Write 3 Done Done Sent false
16434 5469 Acked Read 3 Done Done Sent false
16435 5470 Acked Read 5 Done Done Sent false
16436 5471 Acked Read 7 Done Done Sent false
16437 5472 Acked Flush 0 Done Done Sent false
16438 5473 Acked Read 9 Done Done Sent false
16439 5474 Acked Read 3 Done Done Sent false
16440 5475 Acked Read 4 Done Done Sent false
16441 5476 Acked Write 9 Done Done Sent false
16442 5477 Acked Flush 0 Done Done Sent false
16443 5478 Acked Read 10 Done Done Sent false
16444 5479 Acked Read 7 Done Done Sent false
16445 5480 Acked Write 3 Done Done Sent false
16446 5481 Acked Write 6 Done Done Sent false
16447 5482 Acked Write 2 Done Done Sent false
16448 5483 Acked Read 10 Done Done Sent false
16449 5484 Acked Read 2 Done Done Sent false
16450 5485 Acked Write 1 Done Done Sent false
16451 5486 Acked Write 2 Done Done Sent false
16452 5487 Acked Write 1 Done Done Sent false
16453 5488 Acked Write 10 Done Done Sent false
16454 5489 Acked Read 3 Done Done Sent false
16455 5490 Acked Read 7 Done Done Sent false
16456 5491 Acked Read 2 Done Done Sent false
16457 5492 Acked Write 3 Done Done Sent false
16458 5493 Acked Read 4 Done Done Sent false
16459 5494 Acked Write 6 Done Done Sent false
16460 5495 Acked Read 7 Done Done Sent false
16461 5496 Acked Read 9 Done Done Sent false
16462 5497 Acked Write 8 Done Done Sent false
16463 5498 Acked Read 8 Done Done Sent false
16464 5499 Acked Read 2 Done Done Sent false
16465 5500 Acked Write 8 Done Done Sent false
16466 5501 Acked Write 2 Done Done Sent false
16467 5502 Acked Write 1 Done Done Sent false
16468 5503 Acked Read 5 Done Done Sent false
16469 5504 Acked Read 2 Done Done Sent false
16470 5505 Acked Read 1 Done Done Sent false
16471 5506 Acked Write 7 Done Done Sent false
16472 5507 Acked Flush 0 Done Done Sent false
16473 5508 Acked Read 10 Done Done Sent false
16474 5509 Acked Write 6 Done Done Sent false
16475 5510 Acked Read 3 Done Done Sent false
16476 5511 Acked Read 5 Done Done Sent false
16477 5512 Acked Flush 0 Done Done Sent false
16478 5513 Acked Read 2 Done Done Sent false
16479 5514 Acked Flush 0 Done Done Sent false
16480 5515 Acked Read 5 Done Done Sent false
16481 5516 Acked Read 10 Done Done Sent false
16482 5517 Acked Read 10 Done Done Sent false
16483 5518 Acked Read 7 Done Done Sent false
16484 5519 Acked Read 8 Done Done Sent false
16485 5520 Acked Read 5 Done Done Sent false
16486 5521 Acked Read 9 Done Done Sent false
16487 5522 Acked Read 3 Done Done Sent false
16488 5523 Acked Write 5 Done Done Sent false
16489 5524 Acked Flush 0 Done Done Sent false
16490 5525 Acked Read 8 Done Done Sent false
16491 5526 Acked Write 10 Done Done Sent false
16492 5527 Acked Write 7 Sent Sent Sent false
16493 STATES DS:0 DS:1 DS:2 TOTAL
16494 Sent 1 1 301 303
16495 Done 300 300 0 600
16496 Skipped 0 0 0 0
16497 Error 0 0 0 0
16498 Last Flush: Some(JobId(5524)) Some(JobId(5524)) Some(JobId(5226))
16499 Downstairs last five completed: 5226 5225 5224 5223 5222
16500 Upstairs last five completed: 5527 5526 5525 5524 5523
16501 CLIENT: Up:0 ds:301 act:2
165022025-03-17T22:19:51.931ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
165032025-03-17T22:19:52.933ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
165042025-03-17T22:19:53.936ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
16505 ----------------------------------------------------------------
16506 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16507 States: Active Active Offline
16508 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16509 5227 Acked Flush 0 Done Done Sent false
16510 5228 Acked Flush 0 Done Done Sent false
16511 5229 Acked Read 10 Done Done Sent false
16512 5230 Acked Read 1 Done Done Sent false
16513 5231 Acked Read 1 Done Done Sent false
16514 5232 Acked Write 2 Done Done Sent false
16515 5233 Acked Write 8 Done Done Sent false
16516 5234 Acked Write 1 Done Done Sent false
16517 5235 Acked Write 9 Done Done Sent false
16518 5236 Acked Read 1 Done Done Sent false
16519 5237 Acked Read 7 Done Done Sent false
16520 5238 Acked Read 3 Done Done Sent false
16521 5239 Acked Write 6 Done Done Sent false
16522 5240 Acked Read 9 Done Done Sent false
16523 5241 Acked Write 6 Done Done Sent false
16524 5242 Acked Read 9 Done Done Sent false
16525 5243 Acked Write 3 Done Done Sent false
16526 5244 Acked Read 9 Done Done Sent false
16527 5245 Acked Write 2 Done Done Sent false
16528 5246 Acked Write 4 Done Done Sent false
16529 5247 Acked Write 5 Done Done Sent false
16530 5248 Acked Read 9 Done Done Sent false
16531 5249 Acked Read 2 Done Done Sent false
16532 5250 Acked Read 6 Done Done Sent false
16533 5251 Acked Write 3 Done Done Sent false
16534 5252 Acked Read 6 Done Done Sent false
16535 5253 Acked Read 2 Done Done Sent false
16536 5254 Acked Write 1 Done Done Sent false
16537 5255 Acked Read 6 Done Done Sent false
16538 5256 Acked Flush 0 Done Done Sent false
16539 5257 Acked Read 4 Done Done Sent false
16540 5258 Acked Read 6 Done Done Sent false
16541 5259 Acked Read 1 Done Done Sent false
16542 5260 Acked Write 9 Done Done Sent false
16543 5261 Acked Read 4 Done Done Sent false
16544 5262 Acked Write 5 Done Done Sent false
16545 5263 Acked Read 3 Done Done Sent false
16546 5264 Acked Flush 0 Done Done Sent false
16547 5265 Acked Read 10 Done Done Sent false
16548 5266 Acked Write 9 Done Done Sent false
16549 5267 Acked Write 8 Done Done Sent false
16550 5268 Acked Read 2 Done Done Sent false
16551 5269 Acked Read 10 Done Done Sent false
16552 5270 Acked Write 7 Done Done Sent false
16553 5271 Acked Read 9 Done Done Sent false
16554 5272 Acked Read 6 Done Done Sent false
16555 5273 Acked Write 3 Done Done Sent false
16556 5274 Acked Write 5 Done Done Sent false
16557 5275 Acked Read 3 Done Done Sent false
16558 5276 Acked Flush 0 Done Done Sent false
16559 5277 Acked Read 4 Done Done Sent false
16560 5278 Acked Flush 0 Done Done Sent false
16561 5279 Acked Read 1 Done Done Sent false
16562 5280 Acked Write 6 Done Done Sent false
16563 5281 Acked Write 9 Done Done Sent false
16564 5282 Acked Write 10 Done Done Sent false
16565 5283 Acked Write 1 Done Done Sent false
16566 5284 Acked Write 1 Done Done Sent false
16567 5285 Acked Read 10 Done Done Sent false
16568 5286 Acked Write 2 Done Done Sent false
16569 5287 Acked Write 4 Done Done Sent false
16570 5288 Acked Write 6 Done Done Sent false
16571 5289 Acked Write 4 Done Done Sent false
16572 5290 Acked Read 9 Done Done Sent false
16573 5291 Acked Flush 0 Done Done Sent false
16574 5292 Acked Write 3 Done Done Sent false
16575 5293 Acked Read 1 Done Done Sent false
16576 5294 Acked Write 6 Done Done Sent false
16577 5295 Acked Read 10 Done Done Sent false
16578 5296 Acked Flush 0 Done Done Sent false
16579 5297 Acked Read 2 Done Done Sent false
16580 5298 Acked Read 1 Done Done Sent false
16581 5299 Acked Read 8 Done Done Sent false
16582 5300 Acked Write 7 Done Done Sent false
16583 5301 Acked Write 4 Done Done Sent false
16584 5302 Acked Read 8 Done Done Sent false
16585 5303 Acked Write 8 Done Done Sent false
16586 5304 Acked Write 3 Done Done Sent false
16587 5305 Acked Read 6 Done Done Sent false
16588 5306 Acked Read 1 Done Done Sent false
16589 5307 Acked Write 9 Done Done Sent false
16590 5308 Acked Read 8 Done Done Sent false
16591 5309 Acked Read 9 Done Done Sent false
16592 5310 Acked Write 6 Done Done Sent false
16593 5311 Acked Read 5 Done Done Sent false
16594 5312 Acked Write 6 Done Done Sent false
16595 5313 Acked Write 7 Done Done Sent false
16596 5314 Acked Read 5 Done Done Sent false
16597 5315 Acked Write 9 Done Done Sent false
16598 5316 Acked Write 10 Done Done Sent false
16599 5317 Acked Read 2 Done Done Sent false
16600 5318 Acked Read 6 Done Done Sent false
16601 5319 Acked Flush 0 Done Done Sent false
16602 5320 Acked Write 3 Done Done Sent false
16603 5321 Acked Write 9 Done Done Sent false
16604 5322 Acked Read 6 Done Done Sent false
16605 5323 Acked Write 3 Done Done Sent false
16606 5324 Acked Write 8 Done Done Sent false
16607 5325 Acked Read 3 Done Done Sent false
16608 5326 Acked Write 5 Done Done Sent false
16609 5327 Acked Write 2 Done Done Sent false
16610 5328 Acked Write 9 Done Done Sent false
16611 5329 Acked Read 2 Done Done Sent false
16612 5330 Acked Read 3 Done Done Sent false
16613 5331 Acked Write 9 Done Done Sent false
16614 5332 Acked Write 10 Done Done Sent false
16615 5333 Acked Read 10 Done Done Sent false
16616 5334 Acked Write 4 Done Done Sent false
16617 5335 Acked Read 4 Done Done Sent false
16618 5336 Acked Write 4 Done Done Sent false
16619 5337 Acked Read 10 Done Done Sent false
16620 5338 Acked Read 1 Done Done Sent false
16621 5339 Acked Read 9 Done Done Sent false
16622 5340 Acked Read 4 Done Done Sent false
16623 5341 Acked Write 6 Done Done Sent false
16624 5342 Acked Write 4 Done Done Sent false
16625 5343 Acked Read 9 Done Done Sent false
16626 5344 Acked Read 8 Done Done Sent false
16627 5345 Acked Read 9 Done Done Sent false
16628 5346 Acked Write 6 Done Done Sent false
16629 5347 Acked Read 9 Done Done Sent false
16630 5348 Acked Write 1 Done Done Sent false
16631 5349 Acked Write 7 Done Done Sent false
16632 5350 Acked Read 9 Done Done Sent false
16633 5351 Acked Write 4 Done Done Sent false
16634 5352 Acked Flush 0 Done Done Sent false
16635 5353 Acked Read 2 Done Done Sent false
16636 5354 Acked Read 5 Done Done Sent false
16637 5355 Acked Write 9 Done Done Sent false
16638 5356 Acked Flush 0 Done Done Sent false
16639 5357 Acked Write 3 Done Done Sent false
16640 5358 Acked Read 5 Done Done Sent false
16641 5359 Acked Read 9 Done Done Sent false
16642 5360 Acked Write 2 Done Done Sent false
16643 5361 Acked Write 5 Done Done Sent false
16644 5362 Acked Read 7 Done Done Sent false
16645 5363 Acked Write 9 Done Done Sent false
16646 5364 Acked Read 8 Done Done Sent false
16647 5365 Acked Write 4 Done Done Sent false
16648 5366 Acked Read 8 Done Done Sent false
16649 5367 Acked Read 4 Done Done Sent false
16650 5368 Acked Read 8 Done Done Sent false
16651 5369 Acked Write 1 Done Done Sent false
16652 5370 Acked Write 9 Done Done Sent false
16653 5371 Acked Write 7 Done Done Sent false
16654 5372 Acked Read 8 Done Done Sent false
16655 5373 Acked Write 2 Done Done Sent false
16656 5374 Acked Read 8 Done Done Sent false
16657 5375 Acked Read 9 Done Done Sent false
16658 5376 Acked Write 2 Done Done Sent false
16659 5377 Acked Read 10 Done Done Sent false
16660 5378 Acked Read 2 Done Done Sent false
16661 5379 Acked Read 10 Done Done Sent false
16662 5380 Acked Read 8 Done Done Sent false
16663 5381 Acked Flush 0 Done Done Sent false
16664 5382 Acked Write 8 Done Done Sent false
16665 5383 Acked Flush 0 Done Done Sent false
16666 5384 Acked Read 10 Done Done Sent false
16667 5385 Acked Flush 0 Done Done Sent false
16668 5386 Acked Read 1 Done Done Sent false
16669 5387 Acked Flush 0 Done Done Sent false
16670 5388 Acked Read 8 Done Done Sent false
16671 5389 Acked Read 10 Done Done Sent false
16672 5390 Acked Read 9 Done Done Sent false
16673 5391 Acked Flush 0 Done Done Sent false
16674 5392 Acked Read 3 Done Done Sent false
16675 5393 Acked Read 10 Done Done Sent false
16676 5394 Acked Read 6 Done Done Sent false
16677 5395 Acked Write 7 Done Done Sent false
16678 5396 Acked Read 1 Done Done Sent false
16679 5397 Acked Write 10 Done Done Sent false
16680 5398 Acked Write 8 Done Done Sent false
16681 5399 Acked Write 8 Done Done Sent false
16682 5400 Acked Read 9 Done Done Sent false
16683 5401 Acked Read 3 Done Done Sent false
16684 5402 Acked Write 8 Done Done Sent false
16685 5403 Acked Write 6 Done Done Sent false
16686 5404 Acked Read 6 Done Done Sent false
16687 5405 Acked Read 8 Done Done Sent false
16688 5406 Acked Read 1 Done Done Sent false
16689 5407 Acked Write 8 Done Done Sent false
16690 5408 Acked Read 1 Done Done Sent false
16691 5409 Acked Read 7 Done Done Sent false
16692 5410 Acked Write 4 Done Done Sent false
16693 5411 Acked Write 8 Done Done Sent false
16694 5412 Acked Read 5 Done Done Sent false
16695 5413 Acked Flush 0 Done Done Sent false
16696 5414 Acked Read 8 Done Done Sent false
16697 5415 Acked Read 1 Done Done Sent false
16698 5416 Acked Write 2 Done Done Sent false
16699 5417 Acked Write 4 Done Done Sent false
16700 5418 Acked Write 7 Done Done Sent false
16701 5419 Acked Read 10 Done Done Sent false
16702 5420 Acked Flush 0 Done Done Sent false
16703 5421 Acked Write 4 Done Done Sent false
16704 5422 Acked Read 8 Done Done Sent false
16705 5423 Acked Read 9 Done Done Sent false
16706 5424 Acked Read 3 Done Done Sent false
16707 5425 Acked Read 9 Done Done Sent false
16708 5426 Acked Write 2 Done Done Sent false
16709 5427 Acked Flush 0 Done Done Sent false
16710 5428 Acked Read 3 Done Done Sent false
16711 5429 Acked Write 6 Done Done Sent false
16712 5430 Acked Read 9 Done Done Sent false
16713 5431 Acked Flush 0 Done Done Sent false
16714 5432 Acked Read 4 Done Done Sent false
16715 5433 Acked Flush 0 Done Done Sent false
16716 5434 Acked Read 5 Done Done Sent false
16717 5435 Acked Write 8 Done Done Sent false
16718 5436 Acked Write 10 Done Done Sent false
16719 5437 Acked Write 7 Done Done Sent false
16720 5438 Acked Read 1 Done Done Sent false
16721 5439 Acked Read 2 Done Done Sent false
16722 5440 Acked Read 1 Done Done Sent false
16723 5441 Acked Flush 0 Done Done Sent false
16724 5442 Acked Read 10 Done Done Sent false
16725 5443 Acked Read 8 Done Done Sent false
16726 5444 Acked Write 8 Done Done Sent false
16727 5445 Acked Read 3 Done Done Sent false
16728 5446 Acked Write 6 Done Done Sent false
16729 5447 Acked Write 1 Done Done Sent false
16730 5448 Acked Read 5 Done Done Sent false
16731 5449 Acked Read 4 Done Done Sent false
16732 5450 Acked Write 6 Done Done Sent false
16733 5451 Acked Write 4 Done Done Sent false
16734 5452 Acked Read 5 Done Done Sent false
16735 5453 Acked Flush 0 Done Done Sent false
16736 5454 Acked Write 2 Done Done Sent false
16737 5455 Acked Write 3 Done Done Sent false
16738 5456 Acked Read 10 Done Done Sent false
16739 5457 Acked Read 6 Done Done Sent false
16740 5458 Acked Write 1 Done Done Sent false
16741 5459 Acked Write 3 Done Done Sent false
16742 5460 Acked Read 3 Done Done Sent false
16743 5461 Acked Write 4 Done Done Sent false
16744 5462 Acked Write 6 Done Done Sent false
16745 5463 Acked Write 10 Done Done Sent false
16746 5464 Acked Write 2 Done Done Sent false
16747 5465 Acked Read 6 Done Done Sent false
16748 5466 Acked Read 4 Done Done Sent false
16749 5467 Acked Read 7 Done Done Sent false
16750 5468 Acked Write 3 Done Done Sent false
16751 5469 Acked Read 3 Done Done Sent false
16752 5470 Acked Read 5 Done Done Sent false
16753 5471 Acked Read 7 Done Done Sent false
16754 5472 Acked Flush 0 Done Done Sent false
16755 5473 Acked Read 9 Done Done Sent false
16756 5474 Acked Read 3 Done Done Sent false
16757 5475 Acked Read 4 Done Done Sent false
16758 5476 Acked Write 9 Done Done Sent false
16759 5477 Acked Flush 0 Done Done Sent false
16760 5478 Acked Read 10 Done Done Sent false
16761 5479 Acked Read 7 Done Done Sent false
16762 5480 Acked Write 3 Done Done Sent false
16763 5481 Acked Write 6 Done Done Sent false
16764 5482 Acked Write 2 Done Done Sent false
16765 5483 Acked Read 10 Done Done Sent false
16766 5484 Acked Read 2 Done Done Sent false
16767 5485 Acked Write 1 Done Done Sent false
16768 5486 Acked Write 2 Done Done Sent false
16769 5487 Acked Write 1 Done Done Sent false
16770 5488 Acked Write 10 Done Done Sent false
16771 5489 Acked Read 3 Done Done Sent false
16772 5490 Acked Read 7 Done Done Sent false
16773 5491 Acked Read 2 Done Done Sent false
16774 5492 Acked Write 3 Done Done Sent false
16775 5493 Acked Read 4 Done Done Sent false
16776 5494 Acked Write 6 Done Done Sent false
16777 5495 Acked Read 7 Done Done Sent false
16778 5496 Acked Read 9 Done Done Sent false
16779 5497 Acked Write 8 Done Done Sent false
16780 5498 Acked Read 8 Done Done Sent false
16781 5499 Acked Read 2 Done Done Sent false
16782 5500 Acked Write 8 Done Done Sent false
16783 5501 Acked Write 2 Done Done Sent false
16784 5502 Acked Write 1 Done Done Sent false
16785 5503 Acked Read 5 Done Done Sent false
16786 5504 Acked Read 2 Done Done Sent false
16787 5505 Acked Read 1 Done Done Sent false
16788 5506 Acked Write 7 Done Done Sent false
16789 5507 Acked Flush 0 Done Done Sent false
16790 5508 Acked Read 10 Done Done Sent false
16791 5509 Acked Write 6 Done Done Sent false
16792 5510 Acked Read 3 Done Done Sent false
16793 5511 Acked Read 5 Done Done Sent false
16794 5512 Acked Flush 0 Done Done Sent false
16795 5513 Acked Read 2 Done Done Sent false
16796 5514 Acked Flush 0 Done Done Sent false
16797 5515 Acked Read 5 Done Done Sent false
16798 5516 Acked Read 10 Done Done Sent false
16799 5517 Acked Read 10 Done Done Sent false
16800 5518 Acked Read 7 Done Done Sent false
16801 5519 Acked Read 8 Done Done Sent false
16802 5520 Acked Read 5 Done Done Sent false
16803 5521 Acked Read 9 Done Done Sent false
16804 5522 Acked Read 3 Done Done Sent false
16805 5523 Acked Write 5 Done Done Sent false
16806 5524 Acked Flush 0 Done Done Sent false
16807 5525 Acked Read 8 Done Done Sent false
16808 5526 Acked Write 10 Done Done Sent false
16809 5527 Acked Write 7 Done Done Sent false
16810 STATES DS:0 DS:1 DS:2 TOTAL
16811 Sent 0 0 301 301
16812 Done 301 301 0 602
16813 Skipped 0 0 0 0
16814 Error 0 0 0 0
16815 Last Flush: Some(JobId(5524)) Some(JobId(5524)) Some(JobId(5226))
16816 Downstairs last five completed: 5226 5225 5224 5223 5222
16817 Upstairs last five completed: 5527 5526 5525 5524 5523
16818 CLIENT: Up:0 ds:301 act:2
168192025-03-17T22:19:55.938ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
168202025-03-17T22:19:56.763ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
168212025-03-17T22:19:56.764ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
168222025-03-17T22:19:56.764ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
168232025-03-17T22:19:56.764ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
168242025-03-17T22:19:56.764ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
168252025-03-17T22:19:57.264ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
168262025-03-17T22:19:57.942ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
16827 [8830] got start action
16828 [8830] I am going to restart
16829 Starting downstairs at port 8830
16830 Make output file at "/tmp/dsc/downstairs-8830.txt"
16831 [8830][2] reports Starting
16832 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1221)
16833 [8830][2] reports Running
168342025-03-17T22:19:58.944ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
16835 ----------------------------------------------------------------
16836 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16837 States: Active Active Offline
16838 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
16839 5227 Acked Flush 0 Done Done Sent false
16840 5228 Acked Flush 0 Done Done Sent false
16841 5229 Acked Read 10 Done Done Sent false
16842 5230 Acked Read 1 Done Done Sent false
16843 5231 Acked Read 1 Done Done Sent false
16844 5232 Acked Write 2 Done Done Sent false
16845 5233 Acked Write 8 Done Done Sent false
16846 5234 Acked Write 1 Done Done Sent false
16847 5235 Acked Write 9 Done Done Sent false
16848 5236 Acked Read 1 Done Done Sent false
16849 5237 Acked Read 7 Done Done Sent false
16850 5238 Acked Read 3 Done Done Sent false
16851 5239 Acked Write 6 Done Done Sent false
16852 5240 Acked Read 9 Done Done Sent false
16853 5241 Acked Write 6 Done Done Sent false
16854 5242 Acked Read 9 Done Done Sent false
16855 5243 Acked Write 3 Done Done Sent false
16856 5244 Acked Read 9 Done Done Sent false
16857 5245 Acked Write 2 Done Done Sent false
16858 5246 Acked Write 4 Done Done Sent false
16859 5247 Acked Write 5 Done Done Sent false
16860 5248 Acked Read 9 Done Done Sent false
16861 5249 Acked Read 2 Done Done Sent false
16862 5250 Acked Read 6 Done Done Sent false
16863 5251 Acked Write 3 Done Done Sent false
16864 5252 Acked Read 6 Done Done Sent false
16865 5253 Acked Read 2 Done Done Sent false
16866 5254 Acked Write 1 Done Done Sent false
16867 5255 Acked Read 6 Done Done Sent false
16868 5256 Acked Flush 0 Done Done Sent false
16869 5257 Acked Read 4 Done Done Sent false
16870 5258 Acked Read 6 Done Done Sent false
16871 5259 Acked Read 1 Done Done Sent false
16872 5260 Acked Write 9 Done Done Sent false
16873 5261 Acked Read 4 Done Done Sent false
16874 5262 Acked Write 5 Done Done Sent false
16875 5263 Acked Read 3 Done Done Sent false
16876 5264 Acked Flush 0 Done Done Sent false
16877 5265 Acked Read 10 Done Done Sent false
16878 5266 Acked Write 9 Done Done Sent false
16879 5267 Acked Write 8 Done Done Sent false
16880 5268 Acked Read 2 Done Done Sent false
16881 5269 Acked Read 10 Done Done Sent false
16882 5270 Acked Write 7 Done Done Sent false
16883 5271 Acked Read 9 Done Done Sent false
16884 5272 Acked Read 6 Done Done Sent false
16885 5273 Acked Write 3 Done Done Sent false
16886 5274 Acked Write 5 Done Done Sent false
16887 5275 Acked Read 3 Done Done Sent false
16888 5276 Acked Flush 0 Done Done Sent false
16889 5277 Acked Read 4 Done Done Sent false
16890 5278 Acked Flush 0 Done Done Sent false
16891 5279 Acked Read 1 Done Done Sent false
16892 5280 Acked Write 6 Done Done Sent false
16893 5281 Acked Write 9 Done Done Sent false
16894 5282 Acked Write 10 Done Done Sent false
16895 5283 Acked Write 1 Done Done Sent false
16896 5284 Acked Write 1 Done Done Sent false
16897 5285 Acked Read 10 Done Done Sent false
16898 5286 Acked Write 2 Done Done Sent false
16899 5287 Acked Write 4 Done Done Sent false
16900 5288 Acked Write 6 Done Done Sent false
16901 5289 Acked Write 4 Done Done Sent false
16902 5290 Acked Read 9 Done Done Sent false
16903 5291 Acked Flush 0 Done Done Sent false
16904 5292 Acked Write 3 Done Done Sent false
16905 5293 Acked Read 1 Done Done Sent false
16906 5294 Acked Write 6 Done Done Sent false
16907 5295 Acked Read 10 Done Done Sent false
16908 5296 Acked Flush 0 Done Done Sent false
16909 5297 Acked Read 2 Done Done Sent false
16910 5298 Acked Read 1 Done Done Sent false
16911 5299 Acked Read 8 Done Done Sent false
16912 5300 Acked Write 7 Done Done Sent false
16913 5301 Acked Write 4 Done Done Sent false
16914 5302 Acked Read 8 Done Done Sent false
16915 5303 Acked Write 8 Done Done Sent false
16916 5304 Acked Write 3 Done Done Sent false
16917 5305 Acked Read 6 Done Done Sent false
16918 5306 Acked Read 1 Done Done Sent false
16919 5307 Acked Write 9 Done Done Sent false
16920 5308 Acked Read 8 Done Done Sent false
16921 5309 Acked Read 9 Done Done Sent false
16922 5310 Acked Write 6 Done Done Sent false
16923 5311 Acked Read 5 Done Done Sent false
16924 5312 Acked Write 6 Done Done Sent false
16925 5313 Acked Write 7 Done Done Sent false
16926 5314 Acked Read 5 Done Done Sent false
16927 5315 Acked Write 9 Done Done Sent false
16928 5316 Acked Write 10 Done Done Sent false
16929 5317 Acked Read 2 Done Done Sent false
16930 5318 Acked Read 6 Done Done Sent false
16931 5319 Acked Flush 0 Done Done Sent false
16932 5320 Acked Write 3 Done Done Sent false
16933 5321 Acked Write 9 Done Done Sent false
16934 5322 Acked Read 6 Done Done Sent false
16935 5323 Acked Write 3 Done Done Sent false
16936 5324 Acked Write 8 Done Done Sent false
16937 5325 Acked Read 3 Done Done Sent false
16938 5326 Acked Write 5 Done Done Sent false
16939 5327 Acked Write 2 Done Done Sent false
16940 5328 Acked Write 9 Done Done Sent false
16941 5329 Acked Read 2 Done Done Sent false
16942 5330 Acked Read 3 Done Done Sent false
16943 5331 Acked Write 9 Done Done Sent false
16944 5332 Acked Write 10 Done Done Sent false
16945 5333 Acked Read 10 Done Done Sent false
16946 5334 Acked Write 4 Done Done Sent false
16947 5335 Acked Read 4 Done Done Sent false
16948 5336 Acked Write 4 Done Done Sent false
16949 5337 Acked Read 10 Done Done Sent false
16950 5338 Acked Read 1 Done Done Sent false
16951 5339 Acked Read 9 Done Done Sent false
16952 5340 Acked Read 4 Done Done Sent false
16953 5341 Acked Write 6 Done Done Sent false
16954 5342 Acked Write 4 Done Done Sent false
16955 5343 Acked Read 9 Done Done Sent false
16956 5344 Acked Read 8 Done Done Sent false
16957 5345 Acked Read 9 Done Done Sent false
16958 5346 Acked Write 6 Done Done Sent false
16959 5347 Acked Read 9 Done Done Sent false
16960 5348 Acked Write 1 Done Done Sent false
16961 5349 Acked Write 7 Done Done Sent false
16962 5350 Acked Read 9 Done Done Sent false
16963 5351 Acked Write 4 Done Done Sent false
16964 5352 Acked Flush 0 Done Done Sent false
16965 5353 Acked Read 2 Done Done Sent false
16966 5354 Acked Read 5 Done Done Sent false
16967 5355 Acked Write 9 Done Done Sent false
16968 5356 Acked Flush 0 Done Done Sent false
16969 5357 Acked Write 3 Done Done Sent false
16970 5358 Acked Read 5 Done Done Sent false
16971 5359 Acked Read 9 Done Done Sent false
16972 5360 Acked Write 2 Done Done Sent false
16973 5361 Acked Write 5 Done Done Sent false
16974 5362 Acked Read 7 Done Done Sent false
16975 5363 Acked Write 9 Done Done Sent false
16976 5364 Acked Read 8 Done Done Sent false
16977 5365 Acked Write 4 Done Done Sent false
16978 5366 Acked Read 8 Done Done Sent false
16979 5367 Acked Read 4 Done Done Sent false
16980 5368 Acked Read 8 Done Done Sent false
16981 5369 Acked Write 1 Done Done Sent false
16982 5370 Acked Write 9 Done Done Sent false
16983 5371 Acked Write 7 Done Done Sent false
16984 5372 Acked Read 8 Done Done Sent false
16985 5373 Acked Write 2 Done Done Sent false
16986 5374 Acked Read 8 Done Done Sent false
16987 5375 Acked Read 9 Done Done Sent false
16988 5376 Acked Write 2 Done Done Sent false
16989 5377 Acked Read 10 Done Done Sent false
16990 5378 Acked Read 2 Done Done Sent false
16991 5379 Acked Read 10 Done Done Sent false
16992 5380 Acked Read 8 Done Done Sent false
16993 5381 Acked Flush 0 Done Done Sent false
16994 5382 Acked Write 8 Done Done Sent false
16995 5383 Acked Flush 0 Done Done Sent false
16996 5384 Acked Read 10 Done Done Sent false
16997 5385 Acked Flush 0 Done Done Sent false
16998 5386 Acked Read 1 Done Done Sent false
16999 5387 Acked Flush 0 Done Done Sent false
17000 5388 Acked Read 8 Done Done Sent false
17001 5389 Acked Read 10 Done Done Sent false
17002 5390 Acked Read 9 Done Done Sent false
17003 5391 Acked Flush 0 Done Done Sent false
17004 5392 Acked Read 3 Done Done Sent false
17005 5393 Acked Read 10 Done Done Sent false
17006 5394 Acked Read 6 Done Done Sent false
17007 5395 Acked Write 7 Done Done Sent false
17008 5396 Acked Read 1 Done Done Sent false
17009 5397 Acked Write 10 Done Done Sent false
17010 5398 Acked Write 8 Done Done Sent false
17011 5399 Acked Write 8 Done Done Sent false
17012 5400 Acked Read 9 Done Done Sent false
17013 5401 Acked Read 3 Done Done Sent false
17014 5402 Acked Write 8 Done Done Sent false
17015 5403 Acked Write 6 Done Done Sent false
17016 5404 Acked Read 6 Done Done Sent false
17017 5405 Acked Read 8 Done Done Sent false
17018 5406 Acked Read 1 Done Done Sent false
17019 5407 Acked Write 8 Done Done Sent false
17020 5408 Acked Read 1 Done Done Sent false
17021 5409 Acked Read 7 Done Done Sent false
17022 5410 Acked Write 4 Done Done Sent false
17023 5411 Acked Write 8 Done Done Sent false
17024 5412 Acked Read 5 Done Done Sent false
17025 5413 Acked Flush 0 Done Done Sent false
17026 5414 Acked Read 8 Done Done Sent false
17027 5415 Acked Read 1 Done Done Sent false
17028 5416 Acked Write 2 Done Done Sent false
17029 5417 Acked Write 4 Done Done Sent false
17030 5418 Acked Write 7 Done Done Sent false
17031 5419 Acked Read 10 Done Done Sent false
17032 5420 Acked Flush 0 Done Done Sent false
17033 5421 Acked Write 4 Done Done Sent false
17034 5422 Acked Read 8 Done Done Sent false
17035 5423 Acked Read 9 Done Done Sent false
17036 5424 Acked Read 3 Done Done Sent false
17037 5425 Acked Read 9 Done Done Sent false
17038 5426 Acked Write 2 Done Done Sent false
17039 5427 Acked Flush 0 Done Done Sent false
17040 5428 Acked Read 3 Done Done Sent false
17041 5429 Acked Write 6 Done Done Sent false
17042 5430 Acked Read 9 Done Done Sent false
17043 5431 Acked Flush 0 Done Done Sent false
17044 5432 Acked Read 4 Done Done Sent false
17045 5433 Acked Flush 0 Done Done Sent false
17046 5434 Acked Read 5 Done Done Sent false
17047 5435 Acked Write 8 Done Done Sent false
17048 5436 Acked Write 10 Done Done Sent false
17049 5437 Acked Write 7 Done Done Sent false
17050 5438 Acked Read 1 Done Done Sent false
17051 5439 Acked Read 2 Done Done Sent false
17052 5440 Acked Read 1 Done Done Sent false
17053 5441 Acked Flush 0 Done Done Sent false
17054 5442 Acked Read 10 Done Done Sent false
17055 5443 Acked Read 8 Done Done Sent false
17056 5444 Acked Write 8 Done Done Sent false
17057 5445 Acked Read 3 Done Done Sent false
17058 5446 Acked Write 6 Done Done Sent false
17059 5447 Acked Write 1 Done Done Sent false
17060 5448 Acked Read 5 Done Done Sent false
17061 5449 Acked Read 4 Done Done Sent false
17062 5450 Acked Write 6 Done Done Sent false
17063 5451 Acked Write 4 Done Done Sent false
17064 5452 Acked Read 5 Done Done Sent false
17065 5453 Acked Flush 0 Done Done Sent false
17066 5454 Acked Write 2 Done Done Sent false
17067 5455 Acked Write 3 Done Done Sent false
17068 5456 Acked Read 10 Done Done Sent false
17069 5457 Acked Read 6 Done Done Sent false
17070 5458 Acked Write 1 Done Done Sent false
17071 5459 Acked Write 3 Done Done Sent false
17072 5460 Acked Read 3 Done Done Sent false
17073 5461 Acked Write 4 Done Done Sent false
17074 5462 Acked Write 6 Done Done Sent false
17075 5463 Acked Write 10 Done Done Sent false
17076 5464 Acked Write 2 Done Done Sent false
17077 5465 Acked Read 6 Done Done Sent false
17078 5466 Acked Read 4 Done Done Sent false
17079 5467 Acked Read 7 Done Done Sent false
17080 5468 Acked Write 3 Done Done Sent false
17081 5469 Acked Read 3 Done Done Sent false
17082 5470 Acked Read 5 Done Done Sent false
17083 5471 Acked Read 7 Done Done Sent false
17084 5472 Acked Flush 0 Done Done Sent false
17085 5473 Acked Read 9 Done Done Sent false
17086 5474 Acked Read 3 Done Done Sent false
17087 5475 Acked Read 4 Done Done Sent false
17088 5476 Acked Write 9 Done Done Sent false
17089 5477 Acked Flush 0 Done Done Sent false
17090 5478 Acked Read 10 Done Done Sent false
17091 5479 Acked Read 7 Done Done Sent false
17092 5480 Acked Write 3 Done Done Sent false
17093 5481 Acked Write 6 Done Done Sent false
17094 5482 Acked Write 2 Done Done Sent false
17095 5483 Acked Read 10 Done Done Sent false
17096 5484 Acked Read 2 Done Done Sent false
17097 5485 Acked Write 1 Done Done Sent false
17098 5486 Acked Write 2 Done Done Sent false
17099 5487 Acked Write 1 Done Done Sent false
17100 5488 Acked Write 10 Done Done Sent false
17101 5489 Acked Read 3 Done Done Sent false
17102 5490 Acked Read 7 Done Done Sent false
17103 5491 Acked Read 2 Done Done Sent false
17104 5492 Acked Write 3 Done Done Sent false
17105 5493 Acked Read 4 Done Done Sent false
17106 5494 Acked Write 6 Done Done Sent false
17107 5495 Acked Read 7 Done Done Sent false
17108 5496 Acked Read 9 Done Done Sent false
17109 5497 Acked Write 8 Done Done Sent false
17110 5498 Acked Read 8 Done Done Sent false
17111 5499 Acked Read 2 Done Done Sent false
17112 5500 Acked Write 8 Done Done Sent false
17113 5501 Acked Write 2 Done Done Sent false
17114 5502 Acked Write 1 Done Done Sent false
17115 5503 Acked Read 5 Done Done Sent false
17116 5504 Acked Read 2 Done Done Sent false
17117 5505 Acked Read 1 Done Done Sent false
17118 5506 Acked Write 7 Done Done Sent false
17119 5507 Acked Flush 0 Done Done Sent false
17120 5508 Acked Read 10 Done Done Sent false
17121 5509 Acked Write 6 Done Done Sent false
17122 5510 Acked Read 3 Done Done Sent false
17123 5511 Acked Read 5 Done Done Sent false
17124 5512 Acked Flush 0 Done Done Sent false
17125 5513 Acked Read 2 Done Done Sent false
17126 5514 Acked Flush 0 Done Done Sent false
17127 5515 Acked Read 5 Done Done Sent false
17128 5516 Acked Read 10 Done Done Sent false
17129 5517 Acked Read 10 Done Done Sent false
17130 5518 Acked Read 7 Done Done Sent false
17131 5519 Acked Read 8 Done Done Sent false
17132 5520 Acked Read 5 Done Done Sent false
17133 5521 Acked Read 9 Done Done Sent false
17134 5522 Acked Read 3 Done Done Sent false
17135 5523 Acked Write 5 Done Done Sent false
17136 5524 Acked Flush 0 Done Done Sent false
17137 5525 Acked Read 8 Done Done Sent false
17138 5526 Acked Write 10 Done Done Sent false
17139 5527 Acked Write 7 Done Done Sent false
17140 STATES DS:0 DS:1 DS:2 TOTAL
17141 Sent 0 0 301 301
17142 Done 301 301 0 602
17143 Skipped 0 0 0 0
17144 Error 0 0 0 0
17145 Last Flush: Some(JobId(5524)) Some(JobId(5524)) Some(JobId(5226))
17146 Downstairs last five completed: 5226 5225 5224 5223 5222
17147 Upstairs last five completed: 5527 5526 5525 5524 5523
17148 CLIENT: Up:0 ds:301 act:2
171492025-03-17T22:20:00.947ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
171502025-03-17T22:20:01.948ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
171512025-03-17T22:20:02.951ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
17152 ----------------------------------------------------------------
17153 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17154 States: Active Active Offline
17155 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17156 5227 Acked Flush 0 Done Done Sent false
17157 5228 Acked Flush 0 Done Done Sent false
17158 5229 Acked Read 10 Done Done Sent false
17159 5230 Acked Read 1 Done Done Sent false
17160 5231 Acked Read 1 Done Done Sent false
17161 5232 Acked Write 2 Done Done Sent false
17162 5233 Acked Write 8 Done Done Sent false
17163 5234 Acked Write 1 Done Done Sent false
17164 5235 Acked Write 9 Done Done Sent false
17165 5236 Acked Read 1 Done Done Sent false
17166 5237 Acked Read 7 Done Done Sent false
17167 5238 Acked Read 3 Done Done Sent false
17168 5239 Acked Write 6 Done Done Sent false
17169 5240 Acked Read 9 Done Done Sent false
17170 5241 Acked Write 6 Done Done Sent false
17171 5242 Acked Read 9 Done Done Sent false
17172 5243 Acked Write 3 Done Done Sent false
17173 5244 Acked Read 9 Done Done Sent false
17174 5245 Acked Write 2 Done Done Sent false
17175 5246 Acked Write 4 Done Done Sent false
17176 5247 Acked Write 5 Done Done Sent false
17177 5248 Acked Read 9 Done Done Sent false
17178 5249 Acked Read 2 Done Done Sent false
17179 5250 Acked Read 6 Done Done Sent false
17180 5251 Acked Write 3 Done Done Sent false
17181 5252 Acked Read 6 Done Done Sent false
17182 5253 Acked Read 2 Done Done Sent false
17183 5254 Acked Write 1 Done Done Sent false
17184 5255 Acked Read 6 Done Done Sent false
17185 5256 Acked Flush 0 Done Done Sent false
17186 5257 Acked Read 4 Done Done Sent false
17187 5258 Acked Read 6 Done Done Sent false
17188 5259 Acked Read 1 Done Done Sent false
17189 5260 Acked Write 9 Done Done Sent false
17190 5261 Acked Read 4 Done Done Sent false
17191 5262 Acked Write 5 Done Done Sent false
17192 5263 Acked Read 3 Done Done Sent false
17193 5264 Acked Flush 0 Done Done Sent false
17194 5265 Acked Read 10 Done Done Sent false
17195 5266 Acked Write 9 Done Done Sent false
17196 5267 Acked Write 8 Done Done Sent false
17197 5268 Acked Read 2 Done Done Sent false
17198 5269 Acked Read 10 Done Done Sent false
17199 5270 Acked Write 7 Done Done Sent false
17200 5271 Acked Read 9 Done Done Sent false
17201 5272 Acked Read 6 Done Done Sent false
17202 5273 Acked Write 3 Done Done Sent false
17203 5274 Acked Write 5 Done Done Sent false
17204 5275 Acked Read 3 Done Done Sent false
17205 5276 Acked Flush 0 Done Done Sent false
17206 5277 Acked Read 4 Done Done Sent false
17207 5278 Acked Flush 0 Done Done Sent false
17208 5279 Acked Read 1 Done Done Sent false
17209 5280 Acked Write 6 Done Done Sent false
17210 5281 Acked Write 9 Done Done Sent false
17211 5282 Acked Write 10 Done Done Sent false
17212 5283 Acked Write 1 Done Done Sent false
17213 5284 Acked Write 1 Done Done Sent false
17214 5285 Acked Read 10 Done Done Sent false
17215 5286 Acked Write 2 Done Done Sent false
17216 5287 Acked Write 4 Done Done Sent false
17217 5288 Acked Write 6 Done Done Sent false
17218 5289 Acked Write 4 Done Done Sent false
17219 5290 Acked Read 9 Done Done Sent false
17220 5291 Acked Flush 0 Done Done Sent false
17221 5292 Acked Write 3 Done Done Sent false
17222 5293 Acked Read 1 Done Done Sent false
17223 5294 Acked Write 6 Done Done Sent false
17224 5295 Acked Read 10 Done Done Sent false
17225 5296 Acked Flush 0 Done Done Sent false
17226 5297 Acked Read 2 Done Done Sent false
17227 5298 Acked Read 1 Done Done Sent false
17228 5299 Acked Read 8 Done Done Sent false
17229 5300 Acked Write 7 Done Done Sent false
17230 5301 Acked Write 4 Done Done Sent false
17231 5302 Acked Read 8 Done Done Sent false
17232 5303 Acked Write 8 Done Done Sent false
17233 5304 Acked Write 3 Done Done Sent false
17234 5305 Acked Read 6 Done Done Sent false
17235 5306 Acked Read 1 Done Done Sent false
17236 5307 Acked Write 9 Done Done Sent false
17237 5308 Acked Read 8 Done Done Sent false
17238 5309 Acked Read 9 Done Done Sent false
17239 5310 Acked Write 6 Done Done Sent false
17240 5311 Acked Read 5 Done Done Sent false
17241 5312 Acked Write 6 Done Done Sent false
17242 5313 Acked Write 7 Done Done Sent false
17243 5314 Acked Read 5 Done Done Sent false
17244 5315 Acked Write 9 Done Done Sent false
17245 5316 Acked Write 10 Done Done Sent false
17246 5317 Acked Read 2 Done Done Sent false
17247 5318 Acked Read 6 Done Done Sent false
17248 5319 Acked Flush 0 Done Done Sent false
17249 5320 Acked Write 3 Done Done Sent false
17250 5321 Acked Write 9 Done Done Sent false
17251 5322 Acked Read 6 Done Done Sent false
17252 5323 Acked Write 3 Done Done Sent false
17253 5324 Acked Write 8 Done Done Sent false
17254 5325 Acked Read 3 Done Done Sent false
17255 5326 Acked Write 5 Done Done Sent false
17256 5327 Acked Write 2 Done Done Sent false
17257 5328 Acked Write 9 Done Done Sent false
17258 5329 Acked Read 2 Done Done Sent false
17259 5330 Acked Read 3 Done Done Sent false
17260 5331 Acked Write 9 Done Done Sent false
17261 5332 Acked Write 10 Done Done Sent false
17262 5333 Acked Read 10 Done Done Sent false
17263 5334 Acked Write 4 Done Done Sent false
17264 5335 Acked Read 4 Done Done Sent false
17265 5336 Acked Write 4 Done Done Sent false
17266 5337 Acked Read 10 Done Done Sent false
17267 5338 Acked Read 1 Done Done Sent false
17268 5339 Acked Read 9 Done Done Sent false
17269 5340 Acked Read 4 Done Done Sent false
17270 5341 Acked Write 6 Done Done Sent false
17271 5342 Acked Write 4 Done Done Sent false
17272 5343 Acked Read 9 Done Done Sent false
17273 5344 Acked Read 8 Done Done Sent false
17274 5345 Acked Read 9 Done Done Sent false
17275 5346 Acked Write 6 Done Done Sent false
17276 5347 Acked Read 9 Done Done Sent false
17277 5348 Acked Write 1 Done Done Sent false
17278 5349 Acked Write 7 Done Done Sent false
17279 5350 Acked Read 9 Done Done Sent false
17280 5351 Acked Write 4 Done Done Sent false
17281 5352 Acked Flush 0 Done Done Sent false
17282 5353 Acked Read 2 Done Done Sent false
17283 5354 Acked Read 5 Done Done Sent false
17284 5355 Acked Write 9 Done Done Sent false
17285 5356 Acked Flush 0 Done Done Sent false
17286 5357 Acked Write 3 Done Done Sent false
17287 5358 Acked Read 5 Done Done Sent false
17288 5359 Acked Read 9 Done Done Sent false
17289 5360 Acked Write 2 Done Done Sent false
17290 5361 Acked Write 5 Done Done Sent false
17291 5362 Acked Read 7 Done Done Sent false
17292 5363 Acked Write 9 Done Done Sent false
17293 5364 Acked Read 8 Done Done Sent false
17294 5365 Acked Write 4 Done Done Sent false
17295 5366 Acked Read 8 Done Done Sent false
17296 5367 Acked Read 4 Done Done Sent false
17297 5368 Acked Read 8 Done Done Sent false
17298 5369 Acked Write 1 Done Done Sent false
17299 5370 Acked Write 9 Done Done Sent false
17300 5371 Acked Write 7 Done Done Sent false
17301 5372 Acked Read 8 Done Done Sent false
17302 5373 Acked Write 2 Done Done Sent false
17303 5374 Acked Read 8 Done Done Sent false
17304 5375 Acked Read 9 Done Done Sent false
17305 5376 Acked Write 2 Done Done Sent false
17306 5377 Acked Read 10 Done Done Sent false
17307 5378 Acked Read 2 Done Done Sent false
17308 5379 Acked Read 10 Done Done Sent false
17309 5380 Acked Read 8 Done Done Sent false
17310 5381 Acked Flush 0 Done Done Sent false
17311 5382 Acked Write 8 Done Done Sent false
17312 5383 Acked Flush 0 Done Done Sent false
17313 5384 Acked Read 10 Done Done Sent false
17314 5385 Acked Flush 0 Done Done Sent false
17315 5386 Acked Read 1 Done Done Sent false
17316 5387 Acked Flush 0 Done Done Sent false
17317 5388 Acked Read 8 Done Done Sent false
17318 5389 Acked Read 10 Done Done Sent false
17319 5390 Acked Read 9 Done Done Sent false
17320 5391 Acked Flush 0 Done Done Sent false
17321 5392 Acked Read 3 Done Done Sent false
17322 5393 Acked Read 10 Done Done Sent false
17323 5394 Acked Read 6 Done Done Sent false
17324 5395 Acked Write 7 Done Done Sent false
17325 5396 Acked Read 1 Done Done Sent false
17326 5397 Acked Write 10 Done Done Sent false
17327 5398 Acked Write 8 Done Done Sent false
17328 5399 Acked Write 8 Done Done Sent false
17329 5400 Acked Read 9 Done Done Sent false
17330 5401 Acked Read 3 Done Done Sent false
17331 5402 Acked Write 8 Done Done Sent false
17332 5403 Acked Write 6 Done Done Sent false
17333 5404 Acked Read 6 Done Done Sent false
17334 5405 Acked Read 8 Done Done Sent false
17335 5406 Acked Read 1 Done Done Sent false
17336 5407 Acked Write 8 Done Done Sent false
17337 5408 Acked Read 1 Done Done Sent false
17338 5409 Acked Read 7 Done Done Sent false
17339 5410 Acked Write 4 Done Done Sent false
17340 5411 Acked Write 8 Done Done Sent false
17341 5412 Acked Read 5 Done Done Sent false
17342 5413 Acked Flush 0 Done Done Sent false
17343 5414 Acked Read 8 Done Done Sent false
17344 5415 Acked Read 1 Done Done Sent false
17345 5416 Acked Write 2 Done Done Sent false
17346 5417 Acked Write 4 Done Done Sent false
17347 5418 Acked Write 7 Done Done Sent false
17348 5419 Acked Read 10 Done Done Sent false
17349 5420 Acked Flush 0 Done Done Sent false
17350 5421 Acked Write 4 Done Done Sent false
17351 5422 Acked Read 8 Done Done Sent false
17352 5423 Acked Read 9 Done Done Sent false
17353 5424 Acked Read 3 Done Done Sent false
17354 5425 Acked Read 9 Done Done Sent false
17355 5426 Acked Write 2 Done Done Sent false
17356 5427 Acked Flush 0 Done Done Sent false
17357 5428 Acked Read 3 Done Done Sent false
17358 5429 Acked Write 6 Done Done Sent false
17359 5430 Acked Read 9 Done Done Sent false
17360 5431 Acked Flush 0 Done Done Sent false
17361 5432 Acked Read 4 Done Done Sent false
17362 5433 Acked Flush 0 Done Done Sent false
17363 5434 Acked Read 5 Done Done Sent false
17364 5435 Acked Write 8 Done Done Sent false
17365 5436 Acked Write 10 Done Done Sent false
17366 5437 Acked Write 7 Done Done Sent false
17367 5438 Acked Read 1 Done Done Sent false
17368 5439 Acked Read 2 Done Done Sent false
17369 5440 Acked Read 1 Done Done Sent false
17370 5441 Acked Flush 0 Done Done Sent false
17371 5442 Acked Read 10 Done Done Sent false
17372 5443 Acked Read 8 Done Done Sent false
17373 5444 Acked Write 8 Done Done Sent false
17374 5445 Acked Read 3 Done Done Sent false
17375 5446 Acked Write 6 Done Done Sent false
17376 5447 Acked Write 1 Done Done Sent false
17377 5448 Acked Read 5 Done Done Sent false
17378 5449 Acked Read 4 Done Done Sent false
17379 5450 Acked Write 6 Done Done Sent false
17380 5451 Acked Write 4 Done Done Sent false
17381 5452 Acked Read 5 Done Done Sent false
17382 5453 Acked Flush 0 Done Done Sent false
17383 5454 Acked Write 2 Done Done Sent false
17384 5455 Acked Write 3 Done Done Sent false
17385 5456 Acked Read 10 Done Done Sent false
17386 5457 Acked Read 6 Done Done Sent false
17387 5458 Acked Write 1 Done Done Sent false
17388 5459 Acked Write 3 Done Done Sent false
17389 5460 Acked Read 3 Done Done Sent false
17390 5461 Acked Write 4 Done Done Sent false
17391 5462 Acked Write 6 Done Done Sent false
17392 5463 Acked Write 10 Done Done Sent false
17393 5464 Acked Write 2 Done Done Sent false
17394 5465 Acked Read 6 Done Done Sent false
17395 5466 Acked Read 4 Done Done Sent false
17396 5467 Acked Read 7 Done Done Sent false
17397 5468 Acked Write 3 Done Done Sent false
17398 5469 Acked Read 3 Done Done Sent false
17399 5470 Acked Read 5 Done Done Sent false
17400 5471 Acked Read 7 Done Done Sent false
17401 5472 Acked Flush 0 Done Done Sent false
17402 5473 Acked Read 9 Done Done Sent false
17403 5474 Acked Read 3 Done Done Sent false
17404 5475 Acked Read 4 Done Done Sent false
17405 5476 Acked Write 9 Done Done Sent false
17406 5477 Acked Flush 0 Done Done Sent false
17407 5478 Acked Read 10 Done Done Sent false
17408 5479 Acked Read 7 Done Done Sent false
17409 5480 Acked Write 3 Done Done Sent false
17410 5481 Acked Write 6 Done Done Sent false
17411 5482 Acked Write 2 Done Done Sent false
17412 5483 Acked Read 10 Done Done Sent false
17413 5484 Acked Read 2 Done Done Sent false
17414 5485 Acked Write 1 Done Done Sent false
17415 5486 Acked Write 2 Done Done Sent false
17416 5487 Acked Write 1 Done Done Sent false
17417 5488 Acked Write 10 Done Done Sent false
17418 5489 Acked Read 3 Done Done Sent false
17419 5490 Acked Read 7 Done Done Sent false
17420 5491 Acked Read 2 Done Done Sent false
17421 5492 Acked Write 3 Done Done Sent false
17422 5493 Acked Read 4 Done Done Sent false
17423 5494 Acked Write 6 Done Done Sent false
17424 5495 Acked Read 7 Done Done Sent false
17425 5496 Acked Read 9 Done Done Sent false
17426 5497 Acked Write 8 Done Done Sent false
17427 5498 Acked Read 8 Done Done Sent false
17428 5499 Acked Read 2 Done Done Sent false
17429 5500 Acked Write 8 Done Done Sent false
17430 5501 Acked Write 2 Done Done Sent false
17431 5502 Acked Write 1 Done Done Sent false
17432 5503 Acked Read 5 Done Done Sent false
17433 5504 Acked Read 2 Done Done Sent false
17434 5505 Acked Read 1 Done Done Sent false
17435 5506 Acked Write 7 Done Done Sent false
17436 5507 Acked Flush 0 Done Done Sent false
17437 5508 Acked Read 10 Done Done Sent false
17438 5509 Acked Write 6 Done Done Sent false
17439 5510 Acked Read 3 Done Done Sent false
17440 5511 Acked Read 5 Done Done Sent false
17441 5512 Acked Flush 0 Done Done Sent false
17442 5513 Acked Read 2 Done Done Sent false
17443 5514 Acked Flush 0 Done Done Sent false
17444 5515 Acked Read 5 Done Done Sent false
17445 5516 Acked Read 10 Done Done Sent false
17446 5517 Acked Read 10 Done Done Sent false
17447 5518 Acked Read 7 Done Done Sent false
17448 5519 Acked Read 8 Done Done Sent false
17449 5520 Acked Read 5 Done Done Sent false
17450 5521 Acked Read 9 Done Done Sent false
17451 5522 Acked Read 3 Done Done Sent false
17452 5523 Acked Write 5 Done Done Sent false
17453 5524 Acked Flush 0 Done Done Sent false
17454 5525 Acked Read 8 Done Done Sent false
17455 5526 Acked Write 10 Done Done Sent false
17456 5527 Acked Write 7 Done Done Sent false
17457 STATES DS:0 DS:1 DS:2 TOTAL
17458 Sent 0 0 301 301
17459 Done 301 301 0 602
17460 Skipped 0 0 0 0
17461 Error 0 0 0 0
17462 Last Flush: Some(JobId(5524)) Some(JobId(5524)) Some(JobId(5226))
17463 Downstairs last five completed: 5226 5225 5224 5223 5222
17464 Upstairs last five completed: 5527 5526 5525 5524 5523
17465 CLIENT: Up:0 ds:301 act:2
174662025-03-17T22:20:03.952ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
174672025-03-17T22:20:05.955ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
174682025-03-17T22:20:06.765ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
174692025-03-17T22:20:06.765ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:48747) = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
174702025-03-17T22:20:06.766ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
174712025-03-17T22:20:06.767ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
174722025-03-17T22:20:06.767ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
174732025-03-17T22:20:06.767ZINFOcrucible: send last flush ID to this DS: Some(JobId(5226)) = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
174742025-03-17T22:20:06.767ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
174752025-03-17T22:20:06.767ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(5226)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
174762025-03-17T22:20:06.767ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(5226)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
17477 ----------------------------------------------------------------
17478 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:3
17479 States: Active Active Active
17480 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17481 5525 Acked Read 8 Done Done Done true
17482 5526 Acked Write 10 Done Done Done true
17483 5527 Acked Write 7 Done Done Done true
17484 STATES DS:0 DS:1 DS:2 TOTAL
17485 Sent 0 0 0 0
17486 Done 3 3 3 9
17487 Skipped 0 0 0 0
17488 Error 0 0 0 0
17489 Last Flush: Some(JobId(5524)) Some(JobId(5524)) Some(JobId(5524))
17490 Downstairs last five completed: 5524 5523 5522 5521 5520
17491 Upstairs last five completed: 5527 5526 5525 5524 5523
17492 CLIENT: Up:0 ds:3 act:3
17493 ----------------------------------------------------------------
17494 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
17495 Downstairs last five completed: 5528 5527 5526 5525 5524
17496 Upstairs last five completed: 5528 5527 5526 5525 5524
17497 CLIENT: Up:0 ds:0 act:3
17498 Replay: All jobs finished
17499 Main task has work to do, go find it
17500 got dsc Stop(2)
17501 stop 2
17502 Mar 17 22:20:11.030 INFO request completed, latency_us: 382, response_code: 204[8830] Got stop action so:false kr:false
17503 , uri: /stop/cid/2, method: POST, req_id: 71e135c2-e8d3-4880-beca-4a42d465c47f, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
17504 Mar 17 22:20:11.031 INFO request completed, latency_us: 153, response_code: 200, uri: /state/cid/2, method: GET, req_id: e2a9c60f-1a77-47ba-a532-c080542138bd, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
175052025-03-17T22:20:11.031ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
175062025-03-17T22:20:11.031ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
175072025-03-17T22:20:11.031ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
175082025-03-17T22:20:11.031ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
17509 [8830] Exited with: ExitStatus(unix_wait_status(9))
17510 [8830][2] reports Exit
17511 Mar 17 22:20:15.034 INFO request completed, latency_us: 300, response_code: 200, uri: /state/cid/2, method: GET, req_id: cbdd1715-48c2-40ad-b515-29d2f1b42324, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
17512 001/300 Flush
17513 002/300 Read block 2086 len 12288
17514 003/300 Read block 497 len 24576
17515 004/300 Read block 261 len 20480
17516 005/300 Flush
17517 006/300 Read block 2541 len 20480
17518 007/300 Write block 2833 len 40960 data: 3 3 4 4 4 5 4 4 3 3
17519 008/300 Write block 4366 len 28672 data: 5 6 5 4 4 5 5
17520 009/300 Flush
17521 010/300 Read block 1519 len 40960
17522 011/300 Write block 2386 len 4096 data: 3
17523 012/300 Read block 2109 len 4096
17524 013/300 Write block 1136 len 4096 data: 5
17525 014/300 Read block 4607 len 28672
17526 015/300 Write block 611 len 4096 data: 4
17527 016/300 Write block 49 len 8192 data: 4 4
17528 017/300 Read block 4653 len 28672
17529 018/300 Read block 4007 len 16384
17530 019/300 Write block 1234 len 28672 data: 4 4 4 4 4 3 2
17531 020/300 Write block 3444 len 20480 data: 6 6 6 5 5
17532 021/300 Read block 1030 len 40960
17533 022/300 Read block 3218 len 8192
17534 023/300 Write block 1298 len 24576 data: 4 4 4 4 3 3
17535 024/300 Read block 745 len 32768
17536 025/300 Read block 1318 len 36864
17537 026/300 Write block 2549 len 8192 data: 6 6
17538 027/300 Read block 3383 len 12288
17539 028/300 Read block 3492 len 28672
17540 029/300 Read block 1052 len 12288
17541 030/300 Write block 266 len 4096 data: 4
17542 031/300 Read block 4590 len 28672
17543 032/300 Read block 1314 len 20480
17544 033/300 Read block 1711 len 36864
17545 034/300 Write block 3273 len 24576 data: 4 4 5 5 2 3
17546 035/300 Flush
17547 036/300 Write block 1079 len 16384 data: 5 4 3 4
17548 037/300 Read block 750 len 20480
17549 038/300 Write block 2476 len 8192 data: 6 5
17550 039/300 Write block 3927 len 16384 data: 2 3 3 3
17551 040/300 Read block 1352 len 20480
17552 041/300 Write block 1717 len 28672 data: 4 4 4 3 2 3 3
17553 042/300 Write block 2298 len 20480 data: 5 5 5 3 2
17554 043/300 Write block 649 len 8192 data: 4 4
17555 044/300 Read block 981 len 4096
17556 045/300 Write block 3420 len 12288 data: 5 4 6
17557 046/300 Read block 4947 len 20480
17558 047/300 Read block 1411 len 24576
17559 048/300 Write block 1847 len 28672 data: 4 4 4 4 5 5 5
17560 049/300 Read block 1144 len 16384
17561 050/300 Write block 2552 len 16384 data: 4 4 4 5
17562 051/300 Write block 4545 len 16384 data: 6 4 6 5
17563 052/300 Write block 2492 len 16384 data: 3 2 2 3
17564 053/300 Write block 4468 len 4096 data: 7
17565 054/300 Write block 4347 len 32768 data: 3 3 4 2 2 3 2 2
17566 055/300 Write block 2079 len 8192 data: 2 2
17567 056/300 Flush
17568 057/300 Write block 144 len 24576 data: 3 3 2 2 2 2
17569 058/300 Write block 2474 len 28672 data: 4 4 7 6 6 6 4
17570 059/300 Read block 3405 len 16384
17571 060/300 Read block 4138 len 16384
17572 061/300 Write block 1165 len 16384 data: 5 7 7 7
17573 062/300 Flush
17574 063/300 Write block 4508 len 40960 data: 4 4 4 4 3 3 4 4 5 4
17575 064/300 Read block 4640 len 24576
17576 065/300 Write block 1043 len 28672 data: 3 3 2 3 3 4 5
17577 066/300 Flush
17578 067/300 Read block 1675 len 28672
17579 068/300 Read block 4610 len 16384
17580 069/300 Write block 1617 len 4096 data: 4
17581 070/300 Read block 4978 len 20480
17582 071/300 Write block 120 len 20480 data: 5 5 5 5 5
17583 072/300 Write block 4131 len 28672 data: 2 2 2 3 3 2 3
17584 073/300 Read block 4680 len 36864
17585 074/300 Write block 310 len 40960 data: 4 4 3 4 4 4 6 6 7 7
17586 075/300 Read block 3410 len 28672
17587 076/300 Read block 877 len 8192
17588 077/300 Flush
17589 078/300 Flush
17590 079/300 Write block 1039 len 12288 data: 5 6 7
17591 080/300 Write block 4266 len 20480 data: 3 2 2 3 3
17592 081/300 Write block 4806 len 24576 data: 3 5 5 5 7 7
17593 082/300 Write block 951 len 16384 data: 3 4 4 6
17594 083/300 Read block 3305 len 12288
17595 084/300 Read block 43 len 12288
17596 085/300 Flush
17597 086/300 Write block 3730 len 24576 data: 7 6 5 5 5 2
17598 087/300 Write block 1921 len 32768 data: 4 3 3 3 3 4 3 4
17599 088/300 Write block 1073 len 20480 data: 4 5 6 6 6
17600 089/300 Write block 3922 len 16384 data: 4 4 3 2
17601 090/300 Read block 4405 len 24576
17602 091/300 Read block 3807 len 36864
17603 092/300 Flush
17604 093/300 Write block 4146 len 32768 data: 5 6 6 7 5 4 2 2
17605 094/300 Read block 1179 len 24576
17606 095/300 Write block 2078 len 40960 data: 4 3 3 2 2 2 2 3 4 3
17607 096/300 Write block 53 len 4096 data: 3
17608 097/300 Read block 1174 len 28672
17609 098/300 Write block 2786 len 32768 data: 4 3 3 3 3 4 4 4
17610 099/300 Write block 673 len 16384 data: 3 2 2 2
17611 100/300 Read block 3934 len 4096
17612 101/300 Read block 4243 len 40960
17613 102/300 Read block 4844 len 36864
17614 103/300 Write block 921 len 20480 data: 2 3 3 4 4
17615 104/300 Read block 4342 len 16384
17616 105/300 Read block 387 len 4096
17617 106/300 Flush
17618 107/300 Read block 3230 len 4096
17619 108/300 Write block 4197 len 16384 data: 2 3 4 3
17620 109/300 Read block 3962 len 32768
17621 110/300 Read block 2821 len 36864
17622 111/300 Write block 737 len 32768 data: 3 3 3 3 2 2 2 3
17623 112/300 Write block 1259 len 32768 data: 4 2 2 2 2 2 2 2
17624 113/300 Read block 4379 len 4096
17625 114/300 Write block 2301 len 36864 data: 4 3 2 2 2 2 2 3 4
17626 115/300 Write block 3437 len 24576 data: 5 3 3 3 4 3
17627 116/300 Read block 3221 len 8192
17628 117/300 Read block 1944 len 4096
17629 118/300 Write block 1238 len 36864 data: 5 4 3 2 2 3 3 3 3
17630 119/300 Read block 316 len 28672
17631 120/300 Write block 1580 len 12288 data: 6 5 5
17632 121/300 Flush
17633 122/300 Flush
17634 123/300 Read block 3173 len 28672
17635 124/300 Write block 3146 len 28672 data: 4 5 5 5 5 3 3
17636 125/300 Read block 2646 len 24576
17637 126/300 Flush
17638 127/300 Write block 2912 len 4096 data: 4
17639 128/300 Read block 4568 len 32768
17640 129/300 Write block 973 len 12288 data: 2 2 2
17641 130/300 Flush
17642 131/300 Write block 2174 len 40960 data: 3 4 5 6 8 6 5 6 5 5
17643 132/300 Read block 4427 len 16384
17644 133/300 Read block 3905 len 8192
17645 134/300 Flush
17646 135/300 Read block 2221 len 20480
17647 136/300 Read block 4382 len 28672
17648 137/300 Read block 4632 len 4096
17649 138/300 Read block 882 len 36864
17650 139/300 Write block 1936 len 40960 data: 2 2 2 2 3 3 3 3 3 3
17651 140/300 Read block 2167 len 8192
17652 141/300 Read block 702 len 16384
17653 142/300 Write block 1693 len 8192 data: 4 5
17654 143/300 Read block 4551 len 8192
17655 144/300 Write block 761 len 4096 data: 2
17656 145/300 Read block 3771 len 40960
17657 146/300 Read block 2819 len 12288
17658 147/300 Read block 3036 len 8192
17659 148/300 Read block 4984 len 36864
17660 149/300 Read block 2525 len 12288
17661 150/300 Read block 4325 len 32768
17662 151/300 Write block 1874 len 28672 data: 4 3 3 3 2 2 3
17663 152/300 Read block 3974 len 32768
17664 153/300 Write block 695 len 4096 data: 2
17665 154/300 Read block 3712 len 4096
17666 155/300 Write block 358 len 36864 data: 4 5 5 5 6 5 4 3 2
17667 156/300 Write block 2469 len 8192 data: 3 3
17668 157/300 Read block 734 len 40960
17669 158/300 Write block 948 len 32768 data: 3 2 2 4 5 5 7 6
17670 159/300 Write block 2064 len 24576 data: 5 4 4 4 4 5
17671 160/300 Write block 1276 len 8192 data: 5 5
17672 161/300 Write block 1366 len 40960 data: 5 4 3 3 3 3 3 5 5 5
17673 162/300 Read block 4509 len 12288
17674 163/300 Write block 2494 len 24576 data: 3 4 3 3 3 3
17675 164/300 Write block 3080 len 32768 data: 2 2 2 2 3 3 3 3
17676 165/300 Read block 2445 len 8192
17677 166/300 Write block 889 len 24576 data: 5 5 4 4 4 3
17678 167/300 Write block 220 len 40960 data: 2 2 3 2 2 2 3 2 3 3
17679 168/300 Write block 4766 len 40960 data: 3 3 3 3 3 4 4 4 5 6
17680 169/300 Write block 4541 len 32768 data: 2 3 3 3 7 5 7 6
17681 170/300 Write block 1305 len 36864 data: 2 2 3 5 5 5 5 5 5
17682 171/300 Read block 3919 len 8192
17683 172/300 Read block 3530 len 4096
17684 173/300 Write block 2558 len 12288 data: 4 3 2
17685 174/300 Read block 1325 len 40960
17686 175/300 Write block 3830 len 36864 data: 6 4 4 4 4 4 4 4 3
17687 176/300 Read block 3762 len 8192
17688 177/300 Write block 1399 len 20480 data: 6 6 6 6 6
17689 178/300 Flush
17690 179/300 Flush
17691 180/300 Write block 502 len 4096 data: 5
17692 181/300 Read block 100 len 8192
17693 182/300 Read block 118 len 4096
17694 183/300 Write block 2341 len 20480 data: 2 2 2 2 2
17695 184/300 Write block 4099 len 4096 data: 5
17696 185/300 Read block 460 len 32768
17697 186/300 Read block 3663 len 32768
17698 187/300 Write block 3402 len 36864 data: 4 5 4 4 4 3 3 3 2
17699 188/300 Write block 2833 len 36864 data: 4 4 5 5 5 6 5 5 4
17700 189/300 Read block 623 len 20480
17701 190/300 Flush
17702 191/300 Flush
17703 192/300 Write block 504 len 28672 data: 6 6 6 5 4 4 3
17704 193/300 Read block 1419 len 8192
17705 194/300 Read block 4308 len 4096
17706 195/300 Write block 4398 len 32768 data: 5 5 5 3 3 4 6 5
17707 196/300 Write block 2510 len 40960 data: 3 3 3 3 3 4 3 2 2 2
17708 197/300 Write block 4575 len 4096 data: 4
17709 198/300 Write block 3372 len 8192 data: 4 4
17710 199/300 Flush
17711 200/300 Write block 4364 len 8192 data: 3 3
17712 201/300 Write block 2929 len 16384 data: 6 9 8 7
17713 202/300 Flush
17714 203/300 Write block 4949 len 20480 data: 5 6 5 4 4
17715 204/300 Flush
17716 205/300 Read block 2111 len 12288
17717 206/300 Write block 3864 len 32768 data: 4 4 3 3 3 3 3 4
17718 207/300 Write block 4330 len 12288 data: 4 4 4
17719 208/300 Read block 3122 len 32768
17720 209/300 Write block 2041 len 40960 data: 6 6 5 5 4 5 5 6 6 5
17721 210/300 Read block 4661 len 8192
17722 211/300 Read block 386 len 12288
17723 212/300 Read block 1047 len 28672
17724 213/300 Write block 4734 len 36864 data: 4 4 4 3 3 3 3 3 3
17725 214/300 Write block 403 len 8192 data: 5 5
17726 215/300 Read block 3429 len 28672
17727 216/300 Read block 2241 len 4096
17728 217/300 Read block 4621 len 20480
17729 218/300 Write block 2008 len 40960 data: 3 2 2 3 4 4 3 4 4 4
17730 219/300 Read block 198 len 20480
17731 220/300 Read block 1421 len 12288
17732 221/300 Flush
17733 222/300 Read block 3507 len 24576
17734 223/300 Read block 2182 len 36864
17735 224/300 Flush
17736 225/300 Read block 2142 len 28672
17737 226/300 Write block 106 len 32768 data: 3 3 3 3 3 2 2 2
17738 227/300 Write block 1212 len 40960 data: 5 4 4 5 5 5 4 6 5 6
17739 228/300 Write block 4485 len 8192 data: 3 4
17740 229/300 Flush
17741 230/300 Read block 1703 len 8192
17742 231/300 Write block 1479 len 8192 data: 6 5
17743 232/300 Read block 791 len 20480
17744 233/300 Read block 1795 len 12288
17745 234/300 Write block 4737 len 4096 data: 4
17746 235/300 Write block 1156 len 40960 data: 5 5 4 3 3 4 6 5 4 6
17747 236/300 Read block 145 len 32768
17748 237/300 Write block 1055 len 40960 data: 3 3 3 3 3 3 4 3 4 4
17749 238/300 Write block 2459 len 12288 data: 3 3 2
17750 239/300 Write block 739 len 8192 data: 4 4
17751 240/300 Write block 3510 len 20480 data: 2 2 4 4 5
17752 241/300 Read block 2403 len 12288
17753 242/300 Write block 1792 len 28672 data: 2 2 3 3 3 5 5
17754 243/300 Write block 223 len 12288 data: 3 3 3
17755 244/300 Read block 668 len 36864
17756 245/300 Read block 3362 len 20480
17757 246/300 Read block 1092 len 8192
17758 247/300 Read block 1840 len 32768
17759 248/300 Flush
17760 249/300 Write block 898 len 36864 data: 3 3 3 2 2 2 3 4 5
17761 250/300 Flush
17762 251/300 Read block 1489 len 40960
17763 252/300 Write block 3094 len 36864 data: 3 3 3 5 5 4 4 4 5
17764 253/300 Read block 4065 len 24576
17765 254/300 Write block 860 len 32768 data: 3 3 3 3 3 4 3 2
17766 255/300 Read block 4409 len 4096
17767 256/300 Read block 3262 len 32768
17768 257/300 Read block 1458 len 8192
17769 258/300 Read block 1532 len 40960
17770 259/300 Write block 464 len 24576 data: 4 5 5 4 4 4
17771 260/300 Read block 1520 len 36864
17772 261/300 Write block 4765 len 36864 data: 4 4 4 4 4 4 5 5 5
17773 262/300 Read block 4876 len 4096
17774 263/300 Write block 1968 len 4096 data: 5
17775 264/300 Write block 4105 len 16384 data: 4 5 7 6
17776 265/300 Read block 4040 len 28672
17777 266/300 Write block 3296 len 36864 data: 4 4 3 4 3 3 3 3 4
17778 267/300 Write block 1864 len 8192 data: 5 4
17779 268/300 Write block 603 len 24576 data: 6 6 7 6 6 6
17780 269/300 Write block 809 len 32768 data: 4 3 4 4 5 4 4 3
17781 270/300 Write block 199 len 36864 data: 3 4 4 4 4 2 2 2 2
17782 271/300 Read block 20 len 32768
17783 272/300 Read block 4788 len 20480
17784 273/300 Flush
17785 274/300 Read block 1641 len 20480
17786 275/300 Write block 4802 len 28672 data: 3 3 3 4 4 6 6
17787 276/300 Write block 1682 len 40960 data: 5 5 5 5 3 3 3 4 4 4
17788 277/300 Write block 4873 len 24576 data: 6 5 5 5 5 5
17789 278/300 Read block 2149 len 32768
17790 279/300 Read block 1231 len 28672
17791 280/300 Read block 559 len 40960
17792 281/300 Write block 3033 len 24576 data: 3 3 3 5 5 5
17793 282/300 Read block 4762 len 40960
17794 283/300 Read block 3539 len 4096
17795 284/300 Read block 240 len 20480
17796 285/300 Read block 3997 len 32768
17797 286/300 Flush
17798 287/300 Read block 4825 len 28672
17799 288/300 Write block 4427 len 28672 data: 4 3 3 4 3 3 4
17800 289/300 Write block 3461 len 12288 data: 7 5 5
17801 290/300 Write block 4181 len 20480 data: 3 4 3 4 3
17802 291/300 Read block 2309 len 20480
17803 292/300 Write block 2054 len 36864 data: 6 5 4 3 6 8 8 8 6
17804 293/300 Read block 1232 len 36864
17805 294/300 Write block 1108 len 4096 data: 2
17806 295/300 Read block 3611 len 40960
17807 296/300 Write block 4302 len 8192 data: 6 5
17808 297/300 Read block 1519 len 16384
17809 298/300 Read block 2232 len 12288
17810 299/300 Write block 170 len 40960 data: 4 4 4 4 4 2 2 2 2 2
17811 300/300 Write block 2106 len 12288 data: 5 6 5
17812 301/300 Read block 4587 len 28672
17813 Main task has work to do, go find it
17814 got dsc Start(2)
17815 start 2
17816 Mar 17 22:20:15.270 INFO request completed, latency_us: 223, response_code: 204, uri: /start/cid/2, method: POST, req_id: bd91b1f4-b86a-49c7-af9a-6a430cade6a6, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
17817 Replay: started 2, returned:Ok(())
17818 ----------------------------------------------------------------
17819 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17820 States: Active Active Offline
17821 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
17822 5529 Acked Flush 0 Done Done Sent false
17823 5530 Acked Read 3 Done Done Sent false
17824 5531 Acked Read 6 Done Done Sent false
17825 5532 Acked Read 5 Done Done Sent false
17826 5533 Acked Flush 0 Done Done Sent false
17827 5534 Acked Read 5 Done Done Sent false
17828 5535 Acked Write 10 Done Done Sent false
17829 5536 Acked Write 7 Done Done Sent false
17830 5537 Acked Flush 0 Done Done Sent false
17831 5538 Acked Read 10 Done Done Sent false
17832 5539 Acked Write 1 Done Done Sent false
17833 5540 Acked Read 1 Done Done Sent false
17834 5541 Acked Write 1 Done Done Sent false
17835 5542 Acked Read 7 Done Done Sent false
17836 5543 Acked Write 1 Done Done Sent false
17837 5544 Acked Write 2 Done Done Sent false
17838 5545 Acked Read 7 Done Done Sent false
17839 5546 Acked Read 4 Done Done Sent false
17840 5547 Acked Write 7 Done Done Sent false
17841 5548 Acked Write 5 Done Done Sent false
17842 5549 Acked Read 10 Done Done Sent false
17843 5550 Acked Read 2 Done Done Sent false
17844 5551 Acked Write 6 Done Done Sent false
17845 5552 Acked Read 8 Done Done Sent false
17846 5553 Acked Read 9 Done Done Sent false
17847 5554 Acked Write 2 Done Done Sent false
17848 5555 Acked Read 3 Done Done Sent false
17849 5556 Acked Read 7 Done Done Sent false
17850 5557 Acked Read 3 Done Done Sent false
17851 5558 Acked Write 1 Done Done Sent false
17852 5559 Acked Read 7 Done Done Sent false
17853 5560 Acked Read 5 Done Done Sent false
17854 5561 Acked Read 9 Done Done Sent false
17855 5562 Acked Write 6 Done Done Sent false
17856 5563 Acked Flush 0 Done Done Sent false
17857 5564 Acked Write 4 Done Done Sent false
17858 5565 Acked Read 5 Done Done Sent false
17859 5566 Acked Write 2 Done Done Sent false
17860 5567 Acked Write 4 Done Done Sent false
17861 5568 Acked Read 5 Done Done Sent false
17862 5569 Acked Write 7 Done Done Sent false
17863 5570 Acked Write 5 Done Done Sent false
17864 5571 Acked Write 2 Done Done Sent false
17865 5572 Acked Read 1 Done Done Sent false
17866 5573 Acked Write 3 Done Done Sent false
17867 5574 Acked Read 5 Done Done Sent false
17868 5575 Acked Read 6 Done Done Sent false
17869 5576 Acked Write 7 Done Done Sent false
17870 5577 Acked Read 4 Done Done Sent false
17871 5578 Acked Write 4 Done Done Sent false
17872 5579 Acked Write 4 Done Done Sent false
17873 5580 Acked Write 4 Done Done Sent false
17874 5581 Acked Write 1 Done Done Sent false
17875 5582 Acked Write 8 Done Done Sent false
17876 5583 Acked Write 2 Done Done Sent false
17877 5584 Acked Flush 0 Done Done Sent false
17878 5585 Acked Write 6 Done Done Sent false
17879 5586 Acked Write 7 Done Done Sent false
17880 5587 Acked Read 4 Done Done Sent false
17881 5588 Acked Read 4 Done Done Sent false
17882 5589 Acked Write 4 Done Done Sent false
17883 5590 Acked Flush 0 Done Done Sent false
17884 5591 Acked Write 10 Done Done Sent false
17885 5592 Acked Read 6 Done Done Sent false
17886 5593 Acked Write 7 Done Done Sent false
17887 5594 Acked Flush 0 Done Done Sent false
17888 5595 Acked Read 7 Done Done Sent false
17889 5596 Acked Read 4 Done Done Sent false
17890 5597 Acked Write 1 Done Done Sent false
17891 5598 Acked Read 5 Done Done Sent false
17892 5599 Acked Write 5 Done Done Sent false
17893 5600 Acked Write 7 Done Done Sent false
17894 5601 Acked Read 9 Done Done Sent false
17895 5602 Acked Write 10 Done Done Sent false
17896 5603 Acked Read 7 Done Done Sent false
17897 5604 Acked Read 2 Done Done Sent false
17898 5605 Acked Flush 0 Done Done Sent false
17899 5606 Acked Flush 0 Done Done Sent false
17900 5607 Acked Write 3 Done Done Sent false
17901 5608 Acked Write 5 Done Done Sent false
17902 5609 Acked Write 6 Done Done Sent false
17903 5610 Acked Write 4 Done Done Sent false
17904 5611 Acked Read 3 Done Done Sent false
17905 5612 Acked Read 3 Done Done Sent false
17906 5613 Acked Flush 0 Done Done Sent false
17907 5614 Acked Write 6 Done Done Sent false
17908 5615 Acked Write 8 Done Done Sent false
17909 5616 Acked Write 5 Done Done Sent false
17910 5617 Acked Write 4 Done Done Sent false
17911 5618 Acked Read 6 Done Done Sent false
17912 5619 Acked Read 9 Done Done Sent false
17913 5620 Acked Flush 0 Done Done Sent false
17914 5621 Acked Write 8 Done Done Sent false
17915 5622 Acked Read 6 Done Done Sent false
17916 5623 Acked Write 10 Done Done Sent false
17917 5624 Acked Write 1 Done Done Sent false
17918 5625 Acked Read 7 Done Done Sent false
17919 5626 Acked Write 8 Done Done Sent false
17920 5627 Acked Write 4 Done Done Sent false
17921 5628 Acked Read 1 Done Done Sent false
17922 5629 Acked Read 10 Done Done Sent false
17923 5630 Acked Read 9 Done Done Sent false
17924 5631 Acked Write 5 Done Done Sent false
17925 5632 Acked Read 4 Done Done Sent false
17926 5633 Acked Read 1 Done Done Sent false
17927 5634 Acked Flush 0 Done Done Sent false
17928 5635 Acked Read 1 Done Done Sent false
17929 5636 Acked Write 4 Done Done Sent false
17930 5637 Acked Read 8 Done Done Sent false
17931 5638 Acked Read 9 Done Done Sent false
17932 5639 Acked Write 8 Done Done Sent false
17933 5640 Acked Write 8 Done Done Sent false
17934 5641 Acked Read 1 Done Done Sent false
17935 5642 Acked Write 9 Done Done Sent false
17936 5643 Acked Write 6 Done Done Sent false
17937 5644 Acked Read 2 Done Done Sent false
17938 5645 Acked Read 1 Done Done Sent false
17939 5646 Acked Write 9 Done Done Sent false
17940 5647 Acked Read 7 Done Done Sent false
17941 5648 Acked Write 3 Done Done Sent false
17942 5649 Acked Flush 0 Done Done Sent false
17943 5650 Acked Flush 0 Done Done Sent false
17944 5651 Acked Read 7 Done Done Sent false
17945 5652 Acked Write 7 Done Done Sent false
17946 5653 Acked Read 6 Done Done Sent false
17947 5654 Acked Flush 0 Done Done Sent false
17948 5655 Acked Write 1 Done Done Sent false
17949 5656 Acked Read 8 Done Done Sent false
17950 5657 Acked Write 3 Done Done Sent false
17951 5658 Acked Flush 0 Done Done Sent false
17952 5659 Acked Write 10 Done Done Sent false
17953 5660 Acked Read 4 Done Done Sent false
17954 5661 Acked Read 2 Done Done Sent false
17955 5662 Acked Flush 0 Done Done Sent false
17956 5663 Acked Read 5 Done Done Sent false
17957 5664 Acked Read 7 Done Done Sent false
17958 5665 Acked Read 1 Done Done Sent false
17959 5666 Acked Read 9 Done Done Sent false
17960 5667 Acked Write 10 Done Done Sent false
17961 5668 Acked Read 2 Done Done Sent false
17962 5669 Acked Read 4 Done Done Sent false
17963 5670 Acked Write 2 Done Done Sent false
17964 5671 Acked Read 2 Done Done Sent false
17965 5672 Acked Write 1 Done Done Sent false
17966 5673 Acked Read 10 Done Done Sent false
17967 5674 Acked Read 3 Done Done Sent false
17968 5675 Acked Read 2 Done Done Sent false
17969 5676 Acked Read 9 Done Done Sent false
17970 5677 Acked Read 3 Done Done Sent false
17971 5678 Acked Read 8 Done Done Sent false
17972 5679 Acked Write 7 Done Done Sent false
17973 5680 Acked Read 8 Done Done Sent false
17974 5681 Acked Write 1 Done Done Sent false
17975 5682 Acked Read 1 Done Done Sent false
17976 5683 Acked Write 9 Done Done Sent false
17977 5684 Acked Write 2 Done Done Sent false
17978 5685 Acked Read 10 Done Done Sent false
17979 5686 Acked Write 8 Done Done Sent false
17980 5687 Acked Write 6 Done Done Sent false
17981 5688 Acked Write 2 Done Done Sent false
17982 5689 Acked Write 10 Done Done Sent false
17983 5690 Acked Read 3 Done Done Sent false
17984 5691 Acked Write 6 Done Done Sent false
17985 5692 Acked Write 8 Done Done Sent false
17986 5693 Acked Read 2 Done Done Sent false
17987 5694 Acked Write 6 Done Done Sent false
17988 5695 Acked Write 10 Done Done Sent false
17989 5696 Acked Write 10 Done Done Sent false
17990 5697 Acked Write 8 Done Done Sent false
17991 5698 Acked Write 9 Done Done Sent false
17992 5699 Acked Read 2 Done Done Sent false
17993 5700 Acked Read 1 Done Done Sent false
17994 5701 Acked Write 3 Done Done Sent false
17995 5702 Acked Read 10 Done Done Sent false
17996 5703 Acked Write 9 Done Done Sent false
17997 5704 Acked Read 2 Done Done Sent false
17998 5705 Acked Write 5 Done Done Sent false
17999 5706 Acked Flush 0 Done Done Sent false
18000 5707 Acked Flush 0 Done Done Sent false
18001 5708 Acked Write 1 Done Done Sent false
18002 5709 Acked Read 2 Done Done Sent false
18003 5710 Acked Read 1 Done Done Sent false
18004 5711 Acked Write 5 Done Done Sent false
18005 5712 Acked Write 1 Done Done Sent false
18006 5713 Acked Read 8 Done Done Sent false
18007 5714 Acked Read 8 Done Done Sent false
18008 5715 Acked Write 9 Done Done Sent false
18009 5716 Acked Write 9 Done Done Sent false
18010 5717 Acked Read 5 Done Done Sent false
18011 5718 Acked Flush 0 Done Done Sent false
18012 5719 Acked Flush 0 Done Done Sent false
18013 5720 Acked Write 7 Done Done Sent false
18014 5721 Acked Read 2 Done Done Sent false
18015 5722 Acked Read 1 Done Done Sent false
18016 5723 Acked Write 8 Done Done Sent false
18017 5724 Acked Write 10 Done Done Sent false
18018 5725 Acked Write 1 Done Done Sent false
18019 5726 Acked Write 2 Done Done Sent false
18020 5727 Acked Flush 0 Done Done Sent false
18021 5728 Acked Write 2 Done Done Sent false
18022 5729 Acked Write 4 Done Done Sent false
18023 5730 Acked Flush 0 Done Done Sent false
18024 5731 Acked Write 5 Done Done Sent false
18025 5732 Acked Flush 0 Done Done Sent false
18026 5733 Acked Read 3 Done Done Sent false
18027 5734 Acked Write 8 Done Done Sent false
18028 5735 Acked Write 3 Done Done Sent false
18029 5736 Acked Read 8 Done Done Sent false
18030 5737 Acked Write 10 Done Done Sent false
18031 5738 Acked Read 2 Done Done Sent false
18032 5739 Acked Read 3 Done Done Sent false
18033 5740 Acked Read 7 Done Done Sent false
18034 5741 Acked Write 9 Done Done Sent false
18035 5742 Acked Write 2 Done Done Sent false
18036 5743 Acked Read 7 Done Done Sent false
18037 5744 Acked Read 1 Done Done Sent false
18038 5745 Acked Read 5 Done Done Sent false
18039 5746 Acked Write 10 Done Done Sent false
18040 5747 Acked Read 5 Done Done Sent false
18041 5748 Acked Read 3 Done Done Sent false
18042 5749 Acked Flush 0 Done Done Sent false
18043 5750 Acked Read 6 Done Done Sent false
18044 5751 Acked Read 9 Done Done Sent false
18045 5752 Acked Flush 0 Done Done Sent false
18046 5753 Acked Read 7 Done Done Sent false
18047 5754 Acked Write 8 Done Done Sent false
18048 5755 Acked Write 10 Done Done Sent false
18049 5756 Acked Write 2 Done Done Sent false
18050 5757 Acked Flush 0 Done Done Sent false
18051 5758 Acked Read 2 Done Done Sent false
18052 5759 Acked Write 2 Done Done Sent false
18053 5760 Acked Read 5 Done Done Sent false
18054 5761 Acked Read 3 Done Done Sent false
18055 5762 Acked Write 1 Done Done Sent false
18056 5763 Acked Write 10 Done Done Sent false
18057 5764 Acked Read 8 Done Done Sent false
18058 5765 Acked Write 10 Done Done Sent false
18059 5766 Acked Write 3 Done Done Sent false
18060 5767 Acked Write 2 Done Done Sent false
18061 5768 Acked Write 5 Done Done Sent false
18062 5769 Acked Read 3 Done Done Sent false
18063 5770 Acked Write 7 Done Done Sent false
18064 5771 Acked Write 3 Done Done Sent false
18065 5772 Acked Read 9 Done Done Sent false
18066 5773 Acked Read 5 Done Done Sent false
18067 5774 Acked Read 2 Done Done Sent false
18068 5775 Acked Read 8 Done Done Sent false
18069 5776 Acked Flush 0 Done Done Sent false
18070 5777 Acked Write 9 Done Done Sent false
18071 5778 Acked Flush 0 Done Done Sent false
18072 5779 Acked Read 10 Done Done Sent false
18073 5780 Acked Write 9 Done Done Sent false
18074 5781 Acked Read 6 Done Done Sent false
18075 5782 Acked Write 8 Done Done Sent false
18076 5783 Acked Read 1 Done Done Sent false
18077 5784 Acked Read 8 Done Done Sent false
18078 5785 Acked Read 2 Done Done Sent false
18079 5786 Acked Read 10 Done Done Sent false
18080 5787 Acked Write 6 Done Done Sent false
18081 5788 Acked Read 9 Done Done Sent false
18082 5789 Acked Write 9 Done Done Sent false
18083 5790 Acked Read 1 Done Done Sent false
18084 5791 Acked Write 1 Done Done Sent false
18085 5792 Acked Write 4 Done Done Sent false
18086 5793 Acked Read 7 Done Done Sent false
18087 5794 Acked Write 9 Done Done Sent false
18088 5795 Acked Write 2 Done Done Sent false
18089 5796 Acked Write 6 Done Done Sent false
18090 5797 Acked Write 8 Done Done Sent false
18091 5798 Acked Write 9 Done Done Sent false
18092 5799 Acked Read 8 Done Done Sent false
18093 5800 Acked Read 5 Done Done Sent false
18094 5801 Acked Flush 0 Done Done Sent false
18095 5802 Acked Read 5 Done Done Sent false
18096 5803 Acked Write 7 Done Done Sent false
18097 5804 Acked Write 10 Done Done Sent false
18098 5805 Acked Write 6 Done Done Sent false
18099 5806 Acked Read 8 Done Done Sent false
18100 5807 Acked Read 7 Done Done Sent false
18101 5808 Acked Read 10 Done Done Sent false
18102 5809 Acked Write 6 Done Done Sent false
18103 5810 Acked Read 10 Done Done Sent false
18104 5811 Acked Read 1 Done Done Sent false
18105 5812 Acked Read 5 Done Done Sent false
18106 5813 Acked Read 8 Done Done Sent false
18107 5814 Acked Flush 0 Done Done Sent false
18108 5815 Acked Read 7 Done Done Sent false
18109 5816 Acked Write 7 Done Done Sent false
18110 5817 Acked Write 3 Done Done Sent false
18111 5818 Acked Write 5 Done Done Sent false
18112 5819 Acked Read 5 Done Done Sent false
18113 5820 Acked Write 9 Done Done Sent false
18114 5821 Acked Read 9 Done Done Sent false
18115 5822 Acked Write 1 Done Done Sent false
18116 5823 Acked Read 10 Done Done Sent false
18117 5824 Acked Write 2 Done Done Sent false
18118 5825 Acked Read 4 Done Done Sent false
18119 5826 Acked Read 3 Done Done Sent false
18120 5827 Acked Write 10 Done Done Sent false
18121 5828 Acked Write 3 Done Done Sent false
18122 5829 Acked Read 7 Done Done Sent false
18123 STATES DS:0 DS:1 DS:2 TOTAL
18124 Sent 0 0 301 301
18125 Done 301 301 0 602
18126 Skipped 0 0 0 0
18127 Error 0 0 0 0
18128 Last Flush: Some(JobId(5814)) Some(JobId(5814)) Some(JobId(5528))
18129 Downstairs last five completed: 5528 5527 5526 5525 5524
18130 Upstairs last five completed: 5829 5828 5827 5826 5825
18131 CLIENT: Up:0 ds:301 act:2
181322025-03-17T22:20:15.972ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
181332025-03-17T22:20:16.974ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
181342025-03-17T22:20:17.976ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
181352025-03-17T22:20:18.979ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
18136 [8830] got start action
18137 [8830] I am going to restart
18138 Starting downstairs at port 8830
18139 Make output file at "/tmp/dsc/downstairs-8830.txt"
18140 [8830][2] reports Starting
18141 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1222)
18142 [8830][2] reports Running
18143 ----------------------------------------------------------------
18144 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18145 States: Active Active Offline
18146 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18147 5529 Acked Flush 0 Done Done Sent false
18148 5530 Acked Read 3 Done Done Sent false
18149 5531 Acked Read 6 Done Done Sent false
18150 5532 Acked Read 5 Done Done Sent false
18151 5533 Acked Flush 0 Done Done Sent false
18152 5534 Acked Read 5 Done Done Sent false
18153 5535 Acked Write 10 Done Done Sent false
18154 5536 Acked Write 7 Done Done Sent false
18155 5537 Acked Flush 0 Done Done Sent false
18156 5538 Acked Read 10 Done Done Sent false
18157 5539 Acked Write 1 Done Done Sent false
18158 5540 Acked Read 1 Done Done Sent false
18159 5541 Acked Write 1 Done Done Sent false
18160 5542 Acked Read 7 Done Done Sent false
18161 5543 Acked Write 1 Done Done Sent false
18162 5544 Acked Write 2 Done Done Sent false
18163 5545 Acked Read 7 Done Done Sent false
18164 5546 Acked Read 4 Done Done Sent false
18165 5547 Acked Write 7 Done Done Sent false
18166 5548 Acked Write 5 Done Done Sent false
18167 5549 Acked Read 10 Done Done Sent false
18168 5550 Acked Read 2 Done Done Sent false
18169 5551 Acked Write 6 Done Done Sent false
18170 5552 Acked Read 8 Done Done Sent false
18171 5553 Acked Read 9 Done Done Sent false
18172 5554 Acked Write 2 Done Done Sent false
18173 5555 Acked Read 3 Done Done Sent false
18174 5556 Acked Read 7 Done Done Sent false
18175 5557 Acked Read 3 Done Done Sent false
18176 5558 Acked Write 1 Done Done Sent false
18177 5559 Acked Read 7 Done Done Sent false
18178 5560 Acked Read 5 Done Done Sent false
18179 5561 Acked Read 9 Done Done Sent false
18180 5562 Acked Write 6 Done Done Sent false
18181 5563 Acked Flush 0 Done Done Sent false
18182 5564 Acked Write 4 Done Done Sent false
18183 5565 Acked Read 5 Done Done Sent false
18184 5566 Acked Write 2 Done Done Sent false
18185 5567 Acked Write 4 Done Done Sent false
18186 5568 Acked Read 5 Done Done Sent false
18187 5569 Acked Write 7 Done Done Sent false
18188 5570 Acked Write 5 Done Done Sent false
18189 5571 Acked Write 2 Done Done Sent false
18190 5572 Acked Read 1 Done Done Sent false
18191 5573 Acked Write 3 Done Done Sent false
18192 5574 Acked Read 5 Done Done Sent false
18193 5575 Acked Read 6 Done Done Sent false
18194 5576 Acked Write 7 Done Done Sent false
18195 5577 Acked Read 4 Done Done Sent false
18196 5578 Acked Write 4 Done Done Sent false
18197 5579 Acked Write 4 Done Done Sent false
18198 5580 Acked Write 4 Done Done Sent false
18199 5581 Acked Write 1 Done Done Sent false
18200 5582 Acked Write 8 Done Done Sent false
18201 5583 Acked Write 2 Done Done Sent false
18202 5584 Acked Flush 0 Done Done Sent false
18203 5585 Acked Write 6 Done Done Sent false
18204 5586 Acked Write 7 Done Done Sent false
18205 5587 Acked Read 4 Done Done Sent false
18206 5588 Acked Read 4 Done Done Sent false
18207 5589 Acked Write 4 Done Done Sent false
18208 5590 Acked Flush 0 Done Done Sent false
18209 5591 Acked Write 10 Done Done Sent false
18210 5592 Acked Read 6 Done Done Sent false
18211 5593 Acked Write 7 Done Done Sent false
18212 5594 Acked Flush 0 Done Done Sent false
18213 5595 Acked Read 7 Done Done Sent false
18214 5596 Acked Read 4 Done Done Sent false
18215 5597 Acked Write 1 Done Done Sent false
18216 5598 Acked Read 5 Done Done Sent false
18217 5599 Acked Write 5 Done Done Sent false
18218 5600 Acked Write 7 Done Done Sent false
18219 5601 Acked Read 9 Done Done Sent false
18220 5602 Acked Write 10 Done Done Sent false
18221 5603 Acked Read 7 Done Done Sent false
18222 5604 Acked Read 2 Done Done Sent false
18223 5605 Acked Flush 0 Done Done Sent false
18224 5606 Acked Flush 0 Done Done Sent false
18225 5607 Acked Write 3 Done Done Sent false
18226 5608 Acked Write 5 Done Done Sent false
18227 5609 Acked Write 6 Done Done Sent false
18228 5610 Acked Write 4 Done Done Sent false
18229 5611 Acked Read 3 Done Done Sent false
18230 5612 Acked Read 3 Done Done Sent false
18231 5613 Acked Flush 0 Done Done Sent false
18232 5614 Acked Write 6 Done Done Sent false
18233 5615 Acked Write 8 Done Done Sent false
18234 5616 Acked Write 5 Done Done Sent false
18235 5617 Acked Write 4 Done Done Sent false
18236 5618 Acked Read 6 Done Done Sent false
18237 5619 Acked Read 9 Done Done Sent false
18238 5620 Acked Flush 0 Done Done Sent false
18239 5621 Acked Write 8 Done Done Sent false
18240 5622 Acked Read 6 Done Done Sent false
18241 5623 Acked Write 10 Done Done Sent false
18242 5624 Acked Write 1 Done Done Sent false
18243 5625 Acked Read 7 Done Done Sent false
18244 5626 Acked Write 8 Done Done Sent false
18245 5627 Acked Write 4 Done Done Sent false
18246 5628 Acked Read 1 Done Done Sent false
18247 5629 Acked Read 10 Done Done Sent false
18248 5630 Acked Read 9 Done Done Sent false
18249 5631 Acked Write 5 Done Done Sent false
18250 5632 Acked Read 4 Done Done Sent false
18251 5633 Acked Read 1 Done Done Sent false
18252 5634 Acked Flush 0 Done Done Sent false
18253 5635 Acked Read 1 Done Done Sent false
18254 5636 Acked Write 4 Done Done Sent false
18255 5637 Acked Read 8 Done Done Sent false
18256 5638 Acked Read 9 Done Done Sent false
18257 5639 Acked Write 8 Done Done Sent false
18258 5640 Acked Write 8 Done Done Sent false
18259 5641 Acked Read 1 Done Done Sent false
18260 5642 Acked Write 9 Done Done Sent false
18261 5643 Acked Write 6 Done Done Sent false
18262 5644 Acked Read 2 Done Done Sent false
18263 5645 Acked Read 1 Done Done Sent false
18264 5646 Acked Write 9 Done Done Sent false
18265 5647 Acked Read 7 Done Done Sent false
18266 5648 Acked Write 3 Done Done Sent false
18267 5649 Acked Flush 0 Done Done Sent false
18268 5650 Acked Flush 0 Done Done Sent false
18269 5651 Acked Read 7 Done Done Sent false
18270 5652 Acked Write 7 Done Done Sent false
18271 5653 Acked Read 6 Done Done Sent false
18272 5654 Acked Flush 0 Done Done Sent false
18273 5655 Acked Write 1 Done Done Sent false
18274 5656 Acked Read 8 Done Done Sent false
18275 5657 Acked Write 3 Done Done Sent false
18276 5658 Acked Flush 0 Done Done Sent false
18277 5659 Acked Write 10 Done Done Sent false
18278 5660 Acked Read 4 Done Done Sent false
18279 5661 Acked Read 2 Done Done Sent false
18280 5662 Acked Flush 0 Done Done Sent false
18281 5663 Acked Read 5 Done Done Sent false
18282 5664 Acked Read 7 Done Done Sent false
18283 5665 Acked Read 1 Done Done Sent false
18284 5666 Acked Read 9 Done Done Sent false
18285 5667 Acked Write 10 Done Done Sent false
18286 5668 Acked Read 2 Done Done Sent false
18287 5669 Acked Read 4 Done Done Sent false
18288 5670 Acked Write 2 Done Done Sent false
18289 5671 Acked Read 2 Done Done Sent false
18290 5672 Acked Write 1 Done Done Sent false
18291 5673 Acked Read 10 Done Done Sent false
18292 5674 Acked Read 3 Done Done Sent false
18293 5675 Acked Read 2 Done Done Sent false
18294 5676 Acked Read 9 Done Done Sent false
18295 5677 Acked Read 3 Done Done Sent false
18296 5678 Acked Read 8 Done Done Sent false
18297 5679 Acked Write 7 Done Done Sent false
18298 5680 Acked Read 8 Done Done Sent false
18299 5681 Acked Write 1 Done Done Sent false
18300 5682 Acked Read 1 Done Done Sent false
18301 5683 Acked Write 9 Done Done Sent false
18302 5684 Acked Write 2 Done Done Sent false
18303 5685 Acked Read 10 Done Done Sent false
18304 5686 Acked Write 8 Done Done Sent false
18305 5687 Acked Write 6 Done Done Sent false
18306 5688 Acked Write 2 Done Done Sent false
18307 5689 Acked Write 10 Done Done Sent false
18308 5690 Acked Read 3 Done Done Sent false
18309 5691 Acked Write 6 Done Done Sent false
18310 5692 Acked Write 8 Done Done Sent false
18311 5693 Acked Read 2 Done Done Sent false
18312 5694 Acked Write 6 Done Done Sent false
18313 5695 Acked Write 10 Done Done Sent false
18314 5696 Acked Write 10 Done Done Sent false
18315 5697 Acked Write 8 Done Done Sent false
18316 5698 Acked Write 9 Done Done Sent false
18317 5699 Acked Read 2 Done Done Sent false
18318 5700 Acked Read 1 Done Done Sent false
18319 5701 Acked Write 3 Done Done Sent false
18320 5702 Acked Read 10 Done Done Sent false
18321 5703 Acked Write 9 Done Done Sent false
18322 5704 Acked Read 2 Done Done Sent false
18323 5705 Acked Write 5 Done Done Sent false
18324 5706 Acked Flush 0 Done Done Sent false
18325 5707 Acked Flush 0 Done Done Sent false
18326 5708 Acked Write 1 Done Done Sent false
18327 5709 Acked Read 2 Done Done Sent false
18328 5710 Acked Read 1 Done Done Sent false
18329 5711 Acked Write 5 Done Done Sent false
18330 5712 Acked Write 1 Done Done Sent false
18331 5713 Acked Read 8 Done Done Sent false
18332 5714 Acked Read 8 Done Done Sent false
18333 5715 Acked Write 9 Done Done Sent false
18334 5716 Acked Write 9 Done Done Sent false
18335 5717 Acked Read 5 Done Done Sent false
18336 5718 Acked Flush 0 Done Done Sent false
18337 5719 Acked Flush 0 Done Done Sent false
18338 5720 Acked Write 7 Done Done Sent false
18339 5721 Acked Read 2 Done Done Sent false
18340 5722 Acked Read 1 Done Done Sent false
18341 5723 Acked Write 8 Done Done Sent false
18342 5724 Acked Write 10 Done Done Sent false
18343 5725 Acked Write 1 Done Done Sent false
18344 5726 Acked Write 2 Done Done Sent false
18345 5727 Acked Flush 0 Done Done Sent false
18346 5728 Acked Write 2 Done Done Sent false
18347 5729 Acked Write 4 Done Done Sent false
18348 5730 Acked Flush 0 Done Done Sent false
18349 5731 Acked Write 5 Done Done Sent false
18350 5732 Acked Flush 0 Done Done Sent false
18351 5733 Acked Read 3 Done Done Sent false
18352 5734 Acked Write 8 Done Done Sent false
18353 5735 Acked Write 3 Done Done Sent false
18354 5736 Acked Read 8 Done Done Sent false
18355 5737 Acked Write 10 Done Done Sent false
18356 5738 Acked Read 2 Done Done Sent false
18357 5739 Acked Read 3 Done Done Sent false
18358 5740 Acked Read 7 Done Done Sent false
18359 5741 Acked Write 9 Done Done Sent false
18360 5742 Acked Write 2 Done Done Sent false
18361 5743 Acked Read 7 Done Done Sent false
18362 5744 Acked Read 1 Done Done Sent false
18363 5745 Acked Read 5 Done Done Sent false
18364 5746 Acked Write 10 Done Done Sent false
18365 5747 Acked Read 5 Done Done Sent false
18366 5748 Acked Read 3 Done Done Sent false
18367 5749 Acked Flush 0 Done Done Sent false
18368 5750 Acked Read 6 Done Done Sent false
18369 5751 Acked Read 9 Done Done Sent false
18370 5752 Acked Flush 0 Done Done Sent false
18371 5753 Acked Read 7 Done Done Sent false
18372 5754 Acked Write 8 Done Done Sent false
18373 5755 Acked Write 10 Done Done Sent false
18374 5756 Acked Write 2 Done Done Sent false
18375 5757 Acked Flush 0 Done Done Sent false
18376 5758 Acked Read 2 Done Done Sent false
18377 5759 Acked Write 2 Done Done Sent false
18378 5760 Acked Read 5 Done Done Sent false
18379 5761 Acked Read 3 Done Done Sent false
18380 5762 Acked Write 1 Done Done Sent false
18381 5763 Acked Write 10 Done Done Sent false
18382 5764 Acked Read 8 Done Done Sent false
18383 5765 Acked Write 10 Done Done Sent false
18384 5766 Acked Write 3 Done Done Sent false
18385 5767 Acked Write 2 Done Done Sent false
18386 5768 Acked Write 5 Done Done Sent false
18387 5769 Acked Read 3 Done Done Sent false
18388 5770 Acked Write 7 Done Done Sent false
18389 5771 Acked Write 3 Done Done Sent false
18390 5772 Acked Read 9 Done Done Sent false
18391 5773 Acked Read 5 Done Done Sent false
18392 5774 Acked Read 2 Done Done Sent false
18393 5775 Acked Read 8 Done Done Sent false
18394 5776 Acked Flush 0 Done Done Sent false
18395 5777 Acked Write 9 Done Done Sent false
18396 5778 Acked Flush 0 Done Done Sent false
18397 5779 Acked Read 10 Done Done Sent false
18398 5780 Acked Write 9 Done Done Sent false
18399 5781 Acked Read 6 Done Done Sent false
18400 5782 Acked Write 8 Done Done Sent false
18401 5783 Acked Read 1 Done Done Sent false
18402 5784 Acked Read 8 Done Done Sent false
18403 5785 Acked Read 2 Done Done Sent false
18404 5786 Acked Read 10 Done Done Sent false
18405 5787 Acked Write 6 Done Done Sent false
18406 5788 Acked Read 9 Done Done Sent false
18407 5789 Acked Write 9 Done Done Sent false
18408 5790 Acked Read 1 Done Done Sent false
18409 5791 Acked Write 1 Done Done Sent false
18410 5792 Acked Write 4 Done Done Sent false
18411 5793 Acked Read 7 Done Done Sent false
18412 5794 Acked Write 9 Done Done Sent false
18413 5795 Acked Write 2 Done Done Sent false
18414 5796 Acked Write 6 Done Done Sent false
18415 5797 Acked Write 8 Done Done Sent false
18416 5798 Acked Write 9 Done Done Sent false
18417 5799 Acked Read 8 Done Done Sent false
18418 5800 Acked Read 5 Done Done Sent false
18419 5801 Acked Flush 0 Done Done Sent false
18420 5802 Acked Read 5 Done Done Sent false
18421 5803 Acked Write 7 Done Done Sent false
18422 5804 Acked Write 10 Done Done Sent false
18423 5805 Acked Write 6 Done Done Sent false
18424 5806 Acked Read 8 Done Done Sent false
18425 5807 Acked Read 7 Done Done Sent false
18426 5808 Acked Read 10 Done Done Sent false
18427 5809 Acked Write 6 Done Done Sent false
18428 5810 Acked Read 10 Done Done Sent false
18429 5811 Acked Read 1 Done Done Sent false
18430 5812 Acked Read 5 Done Done Sent false
18431 5813 Acked Read 8 Done Done Sent false
18432 5814 Acked Flush 0 Done Done Sent false
18433 5815 Acked Read 7 Done Done Sent false
18434 5816 Acked Write 7 Done Done Sent false
18435 5817 Acked Write 3 Done Done Sent false
18436 5818 Acked Write 5 Done Done Sent false
18437 5819 Acked Read 5 Done Done Sent false
18438 5820 Acked Write 9 Done Done Sent false
18439 5821 Acked Read 9 Done Done Sent false
18440 5822 Acked Write 1 Done Done Sent false
18441 5823 Acked Read 10 Done Done Sent false
18442 5824 Acked Write 2 Done Done Sent false
18443 5825 Acked Read 4 Done Done Sent false
18444 5826 Acked Read 3 Done Done Sent false
18445 5827 Acked Write 10 Done Done Sent false
18446 5828 Acked Write 3 Done Done Sent false
18447 5829 Acked Read 7 Done Done Sent false
18448 STATES DS:0 DS:1 DS:2 TOTAL
18449 Sent 0 0 301 301
18450 Done 301 301 0 602
18451 Skipped 0 0 0 0
18452 Error 0 0 0 0
18453 Last Flush: Some(JobId(5814)) Some(JobId(5814)) Some(JobId(5528))
18454 Downstairs last five completed: 5528 5527 5526 5525 5524
18455 Upstairs last five completed: 5829 5828 5827 5826 5825
18456 CLIENT: Up:0 ds:301 act:2
184572025-03-17T22:20:20.982ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
184582025-03-17T22:20:21.031ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
184592025-03-17T22:20:21.031ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:36083) = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
184602025-03-17T22:20:21.032ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
184612025-03-17T22:20:21.033ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
184622025-03-17T22:20:21.033ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
184632025-03-17T22:20:21.033ZINFOcrucible: send last flush ID to this DS: Some(JobId(5528)) = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
184642025-03-17T22:20:21.033ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
184652025-03-17T22:20:21.033ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(5528)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
184662025-03-17T22:20:21.033ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(5528)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
18467 ----------------------------------------------------------------
18468 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
18469 Downstairs last five completed: 5830 5829 5828 5827 5826
18470 Upstairs last five completed: 5830 5829 5828 5827 5826
18471 CLIENT: Up:0 ds:0 act:3
18472 Replay: All jobs finished
18473 Main task has work to do, go find it
18474 got dsc Stop(1)
18475 stop 1
18476 Mar 17 22:20:23.277 [8820] Got stop action so:false kr:false
18477 INFO request completed, latency_us: 365, response_code: 204, uri: /stop/cid/1, method: POST, req_id: b4d1988d-bffe-4ce7-ab6e-be5f7ba24313, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
18478 Mar 17 22:20:23.278 INFO request completed, latency_us: 158, response_code: 200, uri: /state/cid/1, method: GET{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:20:23.278317865Z","hostname":"w-01JPK2S810SFAA9787XEG8GFX5","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"0f8310d7-099e-4ead-8484-91ddc162d307"}
18479 , req_id: e81e69bd-5ca6-479f-a37c-b6f7430d09b5, remote_addr: 127.{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:20:23.278401318Z","hostname":"w-01JPK2S810SFAA9787XEG8GFX5","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"0f8310d7-099e-4ead-8484-91ddc162d307"}
18480 0.0.1:60423, local_addr: 127.0.0{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:20:23.278441564Z","hostname":"w-01JPK2S810SFAA9787XEG8GFX5","pid":1206,"session_id":"0f8310d7-099e-4ead-8484-91ddc162d307"}
18481 .1:9998
184822025-03-17T22:20:23.278ZINFOcrucible: client task is exiting = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
18483 [8820] Exited with: ExitStatus(unix_wait_status(9))
18484 [8820][1] reports Exit
18485 Mar 17 22:20:27.280 INFO request completed, latency_us: 200, response_code: 200, uri: /state/cid/1, method: GET, req_id: 765d35c8-9b4e-4344-aef2-b7bb171272b3, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
18486 001/300 Read block 1593 len 32768
18487 002/300 Read block 2761 len 20480
18488 003/300 Write block 3824 len 28672 data: 4 5 7 7 8 7 7
18489 004/300 Write block 4875 len 8192 data: 6 6
18490 005/300 Read block 4844 len 28672
18491 006/300 Read block 3525 len 16384
18492 007/300 Read block 281 len 40960
18493 008/300 Read block 506 len 36864
18494 009/300 Read block 2585 len 16384
18495 010/300 Read block 474 len 24576
18496 011/300 Read block 4623 len 24576
18497 012/300 Read block 3889 len 8192
18498 013/300 Read block 701 len 24576
18499 014/300 Read block 2504 len 32768
18500 015/300 Read block 4406 len 16384
18501 016/300 Read block 2928 len 28672
18502 017/300 Read block 4249 len 32768
18503 018/300 Write block 124 len 24576 data: 6 4 5 5 5 5
18504 019/300 Read block 4684 len 16384
18505 020/300 Read block 2254 len 40960
18506 021/300 Read block 251 len 12288
18507 022/300 Write block 1782 len 32768 data: 3 3 2 3 3 2 2 2
18508 023/300 Write block 2159 len 8192 data: 3 3
18509 024/300 Write block 1842 len 16384 data: 4 4 4 5
18510 025/300 Read block 3331 len 4096
18511 026/300 Flush
18512 027/300 Flush
18513 028/300 Write block 4530 len 36864 data: 4 5 5 5 4 4 4 4 4
18514 029/300 Read block 622 len 32768
18515 030/300 Read block 919 len 12288
18516 031/300 Write block 3140 len 28672 data: 5 4 4 5 5 5 5
18517 032/300 Write block 4152 len 24576 data: 3 3 2 2 3 3
18518 033/300 Read block 4430 len 20480
18519 034/300 Read block 1748 len 32768
18520 035/300 Read block 16 len 32768
18521 036/300 Write block 1925 len 32768 data: 4 5 4 5 3 3 4 5
18522 037/300 Read block 3041 len 16384
18523 038/300 Write block 3076 len 4096 data: 3
18524 039/300 Flush
18525 040/300 Flush
18526 041/300 Write block 2072 len 8192 data: 7 7
18527 042/300 Write block 826 len 36864 data: 2 2 2 5 4 5 5 5 5
18528 043/300 Write block 4575 len 32768 data: 5 3 3 3 3 3 3 2
18529 044/300 Write block 54 len 20480 data: 3 3 4 4 4
18530 045/300 Read block 3517 len 12288
18531 046/300 Read block 2567 len 8192
18532 047/300 Flush
18533 048/300 Read block 1392 len 32768
18534 049/300 Write block 3765 len 16384 data: 2 2 2 3
18535 050/300 Write block 2939 len 32768 data: 5 5 6 5 6 5 4 6
18536 051/300 Read block 4379 len 32768
18537 052/300 Write block 2217 len 36864 data: 5 5 7 7 6 7 6 6 6
18538 053/300 Read block 1660 len 12288
18539 054/300 Read block 2437 len 4096
18540 055/300 Read block 404 len 36864
18541 056/300 Read block 328 len 12288
18542 057/300 Read block 3176 len 20480
18543 058/300 Write block 3243 len 28672 data: 4 4 3 3 5 5 5
18544 059/300 Write block 4159 len 24576 data: 4 5 5 5 5 4
18545 060/300 Write block 3187 len 36864 data: 2 2 2 2 2 2 2 2 2
18546 061/300 Read block 3058 len 36864
18547 062/300 Read block 286 len 4096
18548 063/300 Write block 2391 len 32768 data: 4 3 3 3 3 3 2 2
18549 064/300 Write block 3997 len 28672 data: 3 4 4 4 4 5 4
18550 065/300 Write block 901 len 16384 data: 3 3 3 4
18551 066/300 Read block 405 len 36864
18552 067/300 Read block 2995 len 4096
18553 068/300 Write block 3155 len 20480 data: 3 3 3 3 4
18554 069/300 Write block 1228 len 16384 data: 4 2 2 2
18555 070/300 Write block 3570 len 20480 data: 3 2 2 2 2
18556 071/300 Read block 3302 len 36864
18557 072/300 Write block 2617 len 20480 data: 4 4 4 4 4
18558 073/300 Read block 2099 len 24576
18559 074/300 Write block 3548 len 24576 data: 4 3 3 3 3 2
18560 075/300 Write block 4109 len 40960 data: 5 3 2 3 3 3 2 2 2 2
18561 076/300 Write block 1558 len 20480 data: 3 3 4 4 4
18562 077/300 Read block 111 len 4096
18563 078/300 Read block 723 len 4096
18564 079/300 Write block 3352 len 12288 data: 5 6 7
18565 080/300 Flush
18566 081/300 Read block 2914 len 4096
18567 082/300 Read block 4441 len 20480
18568 083/300 Write block 3226 len 12288 data: 5 6 5
18569 084/300 Read block 3749 len 16384
18570 085/300 Write block 762 len 16384 data: 2 2 2 3
18571 086/300 Write block 1079 len 36864 data: 6 5 4 5 4 3 4 5 4
18572 087/300 Read block 2851 len 4096
18573 088/300 Write block 1459 len 32768 data: 6 5 5 4 4 3 5 5
18574 089/300 Write block 562 len 28672 data: 8 7 7 7 6 3 3
18575 090/300 Flush
18576 091/300 Write block 3685 len 16384 data: 4 4 4 3
18577 092/300 Write block 1493 len 12288 data: 3 2 3
18578 093/300 Read block 4088 len 4096
18579 094/300 Read block 2842 len 20480
18580 095/300 Read block 2372 len 36864
18581 096/300 Read block 4424 len 16384
18582 097/300 Write block 2482 len 4096 data: 4
18583 098/300 Read block 482 len 16384
18584 099/300 Read block 4435 len 36864
18585 100/300 Read block 3957 len 36864
18586 101/300 Write block 1709 len 36864 data: 2 2 3 3 3 4 4 4 5
18587 102/300 Write block 4129 len 8192 data: 3 3
18588 103/300 Read block 3815 len 24576
18589 104/300 Read block 2255 len 12288
18590 105/300 Read block 1889 len 24576
18591 106/300 Read block 3201 len 24576
18592 107/300 Read block 2362 len 12288
18593 108/300 Write block 698 len 28672 data: 5 4 4 4 3 7 5
18594 109/300 Write block 4087 len 4096 data: 8
18595 110/300 Read block 281 len 40960
18596 111/300 Write block 457 len 36864 data: 2 3 3 4 4 4 3 5 6
18597 112/300 Read block 1466 len 8192
18598 113/300 Flush
18599 114/300 Read block 886 len 36864
18600 115/300 Flush
18601 116/300 Write block 2863 len 12288 data: 5 5 4
18602 117/300 Write block 4084 len 40960 data: 6 7 7 9 5 5 5 4 4 5
18603 118/300 Read block 2300 len 28672
18604 119/300 Write block 4793 len 32768 data: 3 4 3 3 3 3 3 3
18605 120/300 Write block 1284 len 32768 data: 5 4 4 4 5 5 3 4
18606 121/300 Read block 4385 len 8192
18607 122/300 Write block 2862 len 40960 data: 4 6 6 5 4 3 5 5 5 6
18608 123/300 Write block 1989 len 8192 data: 7 7
18609 124/300 Write block 2413 len 28672 data: 4 2 3 3 5 3 4
18610 125/300 Flush
18611 126/300 Write block 2495 len 20480 data: 5 4 4 4 4
18612 127/300 Write block 3976 len 4096 data: 3
18613 128/300 Read block 4294 len 20480
18614 129/300 Write block 383 len 28672 data: 2 2 3 5 5 5 4
18615 130/300 Write block 33 len 36864 data: 6 5 5 5 3 3 2 2 2
18616 131/300 Read block 524 len 4096
18617 132/300 Read block 3555 len 36864
18618 133/300 Read block 855 len 12288
18619 134/300 Read block 2876 len 16384
18620 135/300 Write block 1392 len 20480 data: 4 5 4 4 4
18621 136/300 Read block 4699 len 24576
18622 137/300 Write block 147 len 20480 data: 3 3 3 2 2
18623 138/300 Read block 3629 len 28672
18624 139/300 Read block 2895 len 40960
18625 140/300 Read block 237 len 36864
18626 141/300 Write block 239 len 24576 data: 6 6 5 3 3 3
18627 142/300 Read block 4500 len 36864
18628 143/300 Read block 4999 len 4096
18629 144/300 Write block 1201 len 4096 data: 4
18630 145/300 Read block 1314 len 8192
18631 146/300 Write block 4440 len 24576 data: 6 5 5 5 5 5
18632 147/300 Write block 3945 len 4096 data: 4
18633 148/300 Read block 4291 len 40960
18634 149/300 Read block 4604 len 40960
18635 150/300 Flush
18636 151/300 Read block 3230 len 24576
18637 152/300 Write block 2534 len 12288 data: 4 3 2
18638 153/300 Flush
18639 154/300 Read block 724 len 16384
18640 155/300 Write block 3977 len 28672 data: 3 3 2 2 2 2 2
18641 156/300 Read block 319 len 4096
18642 157/300 Read block 3499 len 36864
18643 158/300 Write block 3230 len 8192 data: 6 8
18644 159/300 Write block 4531 len 40960 data: 6 6 6 5 5 5 5 5 3 3
18645 160/300 Write block 4000 len 12288 data: 5 5 6
18646 161/300 Read block 1485 len 12288
18647 162/300 Write block 1558 len 8192 data: 4 4
18648 163/300 Write block 1796 len 36864 data: 4 6 6 4 3 5 4 4 5
18649 164/300 Write block 4194 len 32768 data: 2 2 2 3 4 5 4 3
18650 165/300 Write block 4898 len 24576 data: 4 6 7 6 6 6
18651 166/300 Read block 4883 len 40960
18652 167/300 Flush
18653 168/300 Read block 2570 len 8192
18654 169/300 Read block 577 len 12288
18655 170/300 Read block 475 len 36864
18656 171/300 Read block 110 len 16384
18657 172/300 Read block 302 len 24576
18658 173/300 Write block 2134 len 40960 data: 6 6 6 6 5 5 4 3 5 5
18659 174/300 Read block 1381 len 12288
18660 175/300 Read block 2675 len 28672
18661 176/300 Read block 3987 len 32768
18662 177/300 Write block 1525 len 28672 data: 9 9 10 9 8 8 8
18663 178/300 Read block 1672 len 32768
18664 179/300 Write block 2483 len 32768 data: 4 4 4 4 4 4 4 4
18665 180/300 Write block 2500 len 36864 data: 3 2 3 3 2 2 2 2 3
18666 181/300 Write block 1422 len 16384 data: 4 4 4 4
18667 182/300 Write block 639 len 12288 data: 3 3 4
18668 183/300 Read block 499 len 32768
18669 184/300 Read block 2099 len 32768
18670 185/300 Read block 865 len 32768
18671 186/300 Write block 2945 len 8192 data: 5 7
18672 187/300 Read block 4655 len 16384
18673 188/300 Write block 170 len 36864 data: 5 5 5 5 5 3 3 3 3
18674 189/300 Write block 951 len 40960 data: 5 6 6 8 7 5 5 4 6 5
18675 190/300 Read block 4926 len 24576
18676 191/300 Flush
18677 192/300 Write block 4228 len 12288 data: 4 4 4
18678 193/300 Write block 1352 len 20480 data: 3 3 4 4 6
18679 194/300 Read block 3038 len 36864
18680 195/300 Flush
18681 196/300 Write block 113 len 28672 data: 3 3 4 4 5 5 5
18682 197/300 Read block 4431 len 36864
18683 198/300 Write block 97 len 16384 data: 3 3 4 3
18684 199/300 Read block 4395 len 36864
18685 200/300 Read block 2228 len 4096
18686 201/300 Write block 4613 len 28672 data: 4 4 5 4 4 2 2
18687 202/300 Read block 3909 len 20480
18688 203/300 Read block 275 len 32768
18689 204/300 Read block 1864 len 20480
18690 205/300 Write block 4975 len 16384 data: 3 3 3 3
18691 206/300 Read block 497 len 32768
18692 207/300 Read block 4575 len 20480
18693 208/300 Flush
18694 209/300 Write block 2168 len 28672 data: 2 2 3 3 3 3 4
18695 210/300 Flush
18696 211/300 Write block 754 len 36864 data: 2 2 3 4 3 3 2 3 3
18697 212/300 Read block 557 len 24576
18698 213/300 Write block 3899 len 24576 data: 8 7 8 7 4 4
18699 214/300 Write block 1406 len 24576 data: 5 4 3 3 5 5
18700 215/300 Read block 3134 len 40960
18701 216/300 Read block 4490 len 16384
18702 217/300 Write block 3501 len 40960 data: 4 2 2 2 2 2 2 2 2 3
18703 218/300 Read block 3075 len 16384
18704 219/300 Write block 2144 len 4096 data: 4
18705 220/300 Read block 17 len 4096
18706 221/300 Read block 4821 len 36864
18707 222/300 Read block 1580 len 20480
18708 223/300 Write block 3530 len 32768 data: 3 3 3 4 4 5 5 5
18709 224/300 Read block 1108 len 28672
18710 225/300 Read block 2194 len 12288
18711 226/300 Write block 4375 len 24576 data: 4 4 5 4 3 3
18712 227/300 Read block 3379 len 24576
18713 228/300 Read block 2727 len 16384
18714 229/300 Write block 4368 len 20480 data: 6 5 5 6 6
18715 230/300 Flush
18716 231/300 Read block 2527 len 12288
18717 232/300 Read block 3797 len 16384
18718 233/300 Read block 1219 len 40960
18719 234/300 Write block 2516 len 8192 data: 4 3
18720 235/300 Write block 4862 len 16384 data: 8 8 8 8
18721 236/300 Write block 2657 len 40960 data: 4 4 5 4 4 3 3 4 4 4
18722 237/300 Read block 117 len 40960
18723 238/300 Read block 4573 len 28672
18724 239/300 Read block 616 len 8192
18725 240/300 Write block 747 len 36864 data: 3 2 2 2 2 2 2 3 3
18726 241/300 Read block 1832 len 4096
18727 242/300 Write block 2956 len 20480 data: 8 8 8 9 9
18728 243/300 Write block 3636 len 36864 data: 4 4 4 3 3 5 6 6 6
18729 244/300 Read block 4763 len 36864
18730 245/300 Read block 3618 len 8192
18731 246/300 Flush
18732 247/300 Read block 1809 len 12288
18733 248/300 Read block 2489 len 20480
18734 249/300 Write block 64 len 12288 data: 6 6 5
18735 250/300 Write block 613 len 4096 data: 4
18736 251/300 Read block 4379 len 32768
18737 252/300 Read block 2878 len 24576
18738 253/300 Write block 3081 len 8192 data: 3 3
18739 254/300 Write block 1305 len 36864 data: 3 3 4 6 6 6 6 6 6
18740 255/300 Read block 979 len 12288
18741 256/300 Flush
18742 257/300 Write block 1924 len 20480 data: 4 5 6 5 6
18743 258/300 Write block 3597 len 24576 data: 5 3 3 3 2 2
18744 259/300 Read block 765 len 24576
18745 260/300 Read block 3422 len 28672
18746 261/300 Write block 4934 len 12288 data: 2 2 2
18747 262/300 Read block 984 len 32768
18748 263/300 Write block 403 len 28672 data: 6 6 4 4 5 5 3
18749 264/300 Read block 592 len 32768
18750 265/300 Flush
18751 266/300 Write block 3265 len 32768 data: 4 3 3 3 5 5 4 4
18752 267/300 Read block 2925 len 16384
18753 268/300 Read block 2066 len 8192
18754 269/300 Flush
18755 270/300 Flush
18756 271/300 Read block 1581 len 20480
18757 272/300 Read block 1785 len 28672
18758 273/300 Read block 4329 len 40960
18759 274/300 Read block 3323 len 12288
18760 275/300 Write block 341 len 12288 data: 2 3 3
18761 276/300 Write block 2389 len 28672 data: 4 4 5 4 4 4 4
18762 277/300 Write block 1742 len 32768 data: 3 3 2 3 3 5 4 4
18763 278/300 Flush
18764 279/300 Write block 2671 len 12288 data: 3 4 4
18765 280/300 Read block 3950 len 4096
18766 281/300 Write block 107 len 8192 data: 4 4
18767 282/300 Read block 3271 len 8192
18768 283/300 Read block 2157 len 12288
18769 284/300 Write block 1014 len 28672 data: 3 3 3 3 3 3 4
18770 285/300 Read block 3089 len 40960
18771 286/300 Read block 872 len 12288
18772 287/300 Read block 530 len 8192
18773 288/300 Write block 2166 len 8192 data: 3 2
18774 289/300 Read block 3359 len 12288
18775 290/300 Read block 3254 len 28672
18776 291/300 Write block 1795 len 24576 data: 4 5 7 7 5 4
18777 292/300 Read block 1196 len 36864
18778 293/300 Write block 2296 len 40960 data: 5 5 6 6 6 5 4 3 3 3
18779 294/300 Write block 2826 len 20480 data: 8 8 5 5 6
18780 295/300 Read block 1397 len 40960
18781 296/300 Write block 2638 len 8192 data: 2 2
18782 297/300 Read block 3162 len 24576
18783 298/300 Write block 2945 len 12288 data: 6 8 6
18784 299/300 Read block 3964 len 16384
18785 300/300 Read block 4515 len 8192
18786 301/300 Read block 4383 len 28672
18787 Main task has work to do, go find it
18788 got dsc Start(1)
18789 start 1
18790 Mar 17 22:20:27.533 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/1, method: POST, req_id: e9026a24-38a7-48a7-9e49-3508e10c1c83, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
18791 Replay: started 1, returned:Ok(())
18792 ----------------------------------------------------------------
18793 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18794 States: Active Offline Active
18795 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
18796 5831 Acked Read 8 Done Sent Done false
18797 5832 Acked Read 5 Done Sent Done false
18798 5833 Acked Write 7 Done Sent Done false
18799 5834 Acked Write 2 Done Sent Done false
18800 5835 Acked Read 7 Done Sent Done false
18801 5836 Acked Read 4 Done Sent Done false
18802 5837 Acked Read 10 Done Sent Done false
18803 5838 Acked Read 9 Done Sent Done false
18804 5839 Acked Read 4 Done Sent Done false
18805 5840 Acked Read 6 Done Sent Done false
18806 5841 Acked Read 6 Done Sent Done false
18807 5842 Acked Read 2 Done Sent Done false
18808 5843 Acked Read 6 Done Sent Done false
18809 5844 Acked Read 8 Done Sent Done false
18810 5845 Acked Read 4 Done Sent Done false
18811 5846 Acked Read 7 Done Sent Done false
18812 5847 Acked Read 8 Done Sent Done false
18813 5848 Acked Write 6 Done Sent Done false
18814 5849 Acked Read 4 Done Sent Done false
18815 5850 Acked Read 10 Done Sent Done false
18816 5851 Acked Read 3 Done Sent Done false
18817 5852 Acked Write 8 Done Sent Done false
18818 5853 Acked Write 2 Done Sent Done false
18819 5854 Acked Write 4 Done Sent Done false
18820 5855 Acked Read 1 Done Sent Done false
18821 5856 Acked Flush 0 Done Sent Done false
18822 5857 Acked Flush 0 Done Sent Done false
18823 5858 Acked Write 9 Done Sent Done false
18824 5859 Acked Read 8 Done Sent Done false
18825 5860 Acked Read 3 Done Sent Done false
18826 5861 Acked Write 7 Done Sent Done false
18827 5862 Acked Write 6 Done Sent Done false
18828 5863 Acked Read 5 Done Sent Done false
18829 5864 Acked Read 8 Done Sent Done false
18830 5865 Acked Read 8 Done Sent Done false
18831 5866 Acked Write 8 Done Sent Done false
18832 5867 Acked Read 4 Done Sent Done false
18833 5868 Acked Write 1 Done Sent Done false
18834 5869 Acked Flush 0 Done Sent Done false
18835 5870 Acked Flush 0 Done Sent Done false
18836 5871 Acked Write 2 Done Sent Done false
18837 5872 Acked Write 9 Done Sent Done false
18838 5873 Acked Write 8 Done Sent Done false
18839 5874 Acked Write 5 Done Sent Done false
18840 5875 Acked Read 3 Done Sent Done false
18841 5876 Acked Read 2 Done Sent Done false
18842 5877 Acked Flush 0 Done Sent Done false
18843 5878 Acked Read 8 Done Sent Done false
18844 5879 Acked Write 4 Done Sent Done false
18845 5880 Acked Write 8 Done Sent Done false
18846 5881 Acked Read 8 Done Sent Done false
18847 5882 Acked Write 9 Done Sent Done false
18848 5883 Acked Read 3 Done Sent Done false
18849 5884 Acked Read 1 Done Sent Done false
18850 5885 Acked Read 9 Done Sent Done false
18851 5886 Acked Read 3 Done Sent Done false
18852 5887 Acked Read 5 Done Sent Done false
18853 5888 Acked Write 7 Done Sent Done false
18854 5889 Acked Write 6 Done Sent Done false
18855 5890 Acked Write 9 Done Sent Done false
18856 5891 Acked Read 9 Done Sent Done false
18857 5892 Acked Read 1 Done Sent Done false
18858 5893 Acked Write 8 Done Sent Done false
18859 5894 Acked Write 7 Done Sent Done false
18860 5895 Acked Write 4 Done Sent Done false
18861 5896 Acked Read 9 Done Sent Done false
18862 5897 Acked Read 1 Done Sent Done false
18863 5898 Acked Write 5 Done Sent Done false
18864 5899 Acked Write 4 Done Sent Done false
18865 5900 Acked Write 5 Done Sent Done false
18866 5901 Acked Read 9 Done Sent Done false
18867 5902 Acked Write 5 Done Sent Done false
18868 5903 Acked Read 6 Done Sent Done false
18869 5904 Acked Write 6 Done Sent Done false
18870 5905 Acked Write 10 Done Sent Done false
18871 5906 Acked Write 5 Done Sent Done false
18872 5907 Acked Read 1 Done Sent Done false
18873 5908 Acked Read 1 Done Sent Done false
18874 5909 Acked Write 3 Done Sent Done false
18875 5910 Acked Flush 0 Done Sent Done false
18876 5911 Acked Read 1 Done Sent Done false
18877 5912 Acked Read 5 Done Sent Done false
18878 5913 Acked Write 3 Done Sent Done false
18879 5914 Acked Read 4 Done Sent Done false
18880 5915 Acked Write 4 Done Sent Done false
18881 5916 Acked Write 9 Done Sent Done false
18882 5917 Acked Read 1 Done Sent Done false
18883 5918 Acked Write 8 Done Sent Done false
18884 5919 Acked Write 7 Done Sent Done false
18885 5920 Acked Flush 0 Done Sent Done false
18886 5921 Acked Write 4 Done Sent Done false
18887 5922 Acked Write 3 Done Sent Done false
18888 5923 Acked Read 1 Done Sent Done false
18889 5924 Acked Read 5 Done Sent Done false
18890 5925 Acked Read 9 Done Sent Done false
18891 5926 Acked Read 4 Done Sent Done false
18892 5927 Acked Write 1 Done Sent Done false
18893 5928 Acked Read 4 Done Sent Done false
18894 5929 Acked Read 9 Done Sent Done false
18895 5930 Acked Read 9 Done Sent Done false
18896 5931 Acked Write 9 Done Sent Done false
18897 5932 Acked Write 2 Done Sent Done false
18898 5933 Acked Read 6 Done Sent Done false
18899 5934 Acked Read 3 Done Sent Done false
18900 5935 Acked Read 6 Done Sent Done false
18901 5936 Acked Read 6 Done Sent Done false
18902 5937 Acked Read 3 Done Sent Done false
18903 5938 Acked Write 7 Done Sent Done false
18904 5939 Acked Write 1 Done Sent Done false
18905 5940 Acked Read 10 Done Sent Done false
18906 5941 Acked Write 9 Done Sent Done false
18907 5942 Acked Read 2 Done Sent Done false
18908 5943 Acked Flush 0 Done Sent Done false
18909 5944 Acked Read 9 Done Sent Done false
18910 5945 Acked Flush 0 Done Sent Done false
18911 5946 Acked Write 3 Done Sent Done false
18912 5947 Acked Write 10 Done Sent Done false
18913 5948 Acked Read 7 Done Sent Done false
18914 5949 Acked Write 8 Done Sent Done false
18915 5950 Acked Write 8 Done Sent Done false
18916 5951 Acked Read 2 Done Sent Done false
18917 5952 Acked Write 10 Done Sent Done false
18918 5953 Acked Write 2 Done Sent Done false
18919 5954 Acked Write 7 Done Sent Done false
18920 5955 Acked Flush 0 Done Sent Done false
18921 5956 Acked Write 5 Done Sent Done false
18922 5957 Acked Write 1 Done Sent Done false
18923 5958 Acked Read 5 Done Sent Done false
18924 5959 Acked Write 7 Done Sent Done false
18925 5960 Acked Write 9 Done Sent Done false
18926 5961 Acked Read 1 Done Sent Done false
18927 5962 Acked Read 9 Done Sent Done false
18928 5963 Acked Read 3 Done Sent Done false
18929 5964 Acked Read 4 Done Sent Done false
18930 5965 Acked Write 5 Done Sent Done false
18931 5966 Acked Read 6 Done Sent Done false
18932 5967 Acked Write 5 Done Sent Done false
18933 5968 Acked Read 7 Done Sent Done false
18934 5969 Acked Read 10 Done Sent Done false
18935 5970 Acked Read 9 Done Sent Done false
18936 5971 Acked Write 6 Done Sent Done false
18937 5972 Acked Read 9 Done Sent Done false
18938 5973 Acked Read 1 Done Sent Done false
18939 5974 Acked Write 1 Done Sent Done false
18940 5975 Acked Read 2 Done Sent Done false
18941 5976 Acked Write 6 Done Sent Done false
18942 5977 Acked Write 1 Done Sent Done false
18943 5978 Acked Read 10 Done Sent Done false
18944 5979 Acked Read 10 Done Sent Done false
18945 5980 Acked Flush 0 Done Sent Done false
18946 5981 Acked Read 6 Done Sent Done false
18947 5982 Acked Write 3 Done Sent Done false
18948 5983 Acked Flush 0 Done Sent Done false
18949 5984 Acked Read 4 Done Sent Done false
18950 5985 Acked Write 7 Done Sent Done false
18951 5986 Acked Read 1 Done Sent Done false
18952 5987 Acked Read 9 Done Sent Done false
18953 5988 Acked Write 2 Done Sent Done false
18954 5989 Acked Write 10 Done Sent Done false
18955 5990 Acked Write 3 Done Sent Done false
18956 5991 Acked Read 3 Done Sent Done false
18957 5992 Acked Write 2 Done Sent Done false
18958 5993 Acked Write 9 Done Sent Done false
18959 5994 Acked Write 8 Done Sent Done false
18960 5995 Acked Write 6 Done Sent Done false
18961 5996 Acked Read 10 Done Sent Done false
18962 5997 Acked Flush 0 Done Sent Done false
18963 5998 Acked Read 2 Done Sent Done false
18964 5999 Acked Read 3 Done Sent Done false
18965 6000 Acked Read 9 Done Sent Done false
18966 6001 Acked Read 4 Done Sent Done false
18967 6002 Acked Read 6 Done Sent Done false
18968 6003 Acked Write 10 Done Sent Done false
18969 6004 Acked Read 3 Done Sent Done false
18970 6005 Acked Read 7 Done Sent Done false
18971 6006 Acked Read 8 Done Sent Done false
18972 6007 Acked Write 7 Done Sent Done false
18973 6008 Acked Read 8 Done Sent Done false
18974 6009 Acked Write 8 Done Sent Done false
18975 6010 Acked Write 9 Done Sent Done false
18976 6011 Acked Write 4 Done Sent Done false
18977 6012 Acked Write 3 Done Sent Done false
18978 6013 Acked Read 8 Done Sent Done false
18979 6014 Acked Read 8 Done Sent Done false
18980 6015 Acked Read 8 Done Sent Done false
18981 6016 Acked Write 2 Done Sent Done false
18982 6017 Acked Read 4 Done Sent Done false
18983 6018 Acked Write 9 Done Sent Done false
18984 6019 Acked Write 10 Done Sent Done false
18985 6020 Acked Read 6 Done Sent Done false
18986 6021 Acked Flush 0 Done Sent Done false
18987 6022 Acked Write 3 Done Sent Done false
18988 6023 Acked Write 5 Done Sent Done false
18989 6024 Acked Read 9 Done Sent Done false
18990 6025 Acked Flush 0 Done Sent Done false
18991 6026 Acked Write 7 Done Sent Done false
18992 6027 Acked Read 9 Done Sent Done false
18993 6028 Acked Write 4 Done Sent Done false
18994 6029 Acked Read 9 Done Sent Done false
18995 6030 Acked Read 1 Done Sent Done false
18996 6031 Acked Write 7 Done Sent Done false
18997 6032 Acked Read 5 Done Sent Done false
18998 6033 Acked Read 8 Done Sent Done false
18999 6034 Acked Read 5 Done Sent Done false
19000 6035 Acked Write 4 Done Sent Done false
19001 6036 Acked Read 8 Done Sent Done false
19002 6037 Acked Read 5 Done Sent Done false
19003 6038 Acked Flush 0 Done Sent Done false
19004 6039 Acked Write 7 Done Sent Done false
19005 6040 Acked Flush 0 Done Sent Done false
19006 6041 Acked Write 9 Done Sent Done false
19007 6042 Acked Read 6 Done Sent Done false
19008 6043 Acked Write 6 Done Sent Done false
19009 6044 Acked Write 6 Done Sent Done false
19010 6045 Acked Read 10 Done Sent Done false
19011 6046 Acked Read 4 Done Sent Done false
19012 6047 Acked Write 10 Done Sent Done false
19013 6048 Acked Read 4 Done Sent Done false
19014 6049 Acked Write 1 Done Sent Done false
19015 6050 Acked Read 1 Done Sent Done false
19016 6051 Acked Read 9 Done Sent Done false
19017 6052 Acked Read 5 Done Sent Done false
19018 6053 Acked Write 8 Done Sent Done false
19019 6054 Acked Read 7 Done Sent Done false
19020 6055 Acked Read 3 Done Sent Done false
19021 6056 Acked Write 6 Done Sent Done false
19022 6057 Acked Read 6 Done Sent Done false
19023 6058 Acked Read 4 Done Sent Done false
19024 6059 Acked Write 5 Done Sent Done false
19025 6060 Acked Flush 0 Done Sent Done false
19026 6061 Acked Read 3 Done Sent Done false
19027 6062 Acked Read 4 Done Sent Done false
19028 6063 Acked Read 10 Done Sent Done false
19029 6064 Acked Write 2 Done Sent Done false
19030 6065 Acked Write 4 Done Sent Done false
19031 6066 Acked Write 10 Done Sent Done false
19032 6067 Acked Read 10 Done Sent Done false
19033 6068 Acked Read 7 Done Sent Done false
19034 6069 Acked Read 2 Done Sent Done false
19035 6070 Acked Write 9 Done Sent Done false
19036 6071 Acked Read 1 Done Sent Done false
19037 6072 Acked Write 5 Done Sent Done false
19038 6073 Acked Write 9 Done Sent Done false
19039 6074 Acked Read 9 Done Sent Done false
19040 6075 Acked Read 2 Done Sent Done false
19041 6076 Acked Flush 0 Done Sent Done false
19042 6077 Acked Read 3 Done Sent Done false
19043 6078 Acked Read 5 Done Sent Done false
19044 6079 Acked Write 3 Done Sent Done false
19045 6080 Acked Write 1 Done Sent Done false
19046 6081 Acked Read 8 Done Sent Done false
19047 6082 Acked Read 6 Done Sent Done false
19048 6083 Acked Write 2 Done Sent Done false
19049 6084 Acked Write 9 Done Sent Done false
19050 6085 Acked Read 3 Done Sent Done false
19051 6086 Acked Flush 0 Done Sent Done false
19052 6087 Acked Write 5 Done Sent Done false
19053 6088 Acked Write 6 Done Sent Done false
19054 6089 Acked Read 6 Done Sent Done false
19055 6090 Acked Read 7 Done Sent Done false
19056 6091 Acked Write 3 Done Sent Done false
19057 6092 Acked Read 8 Done Sent Done false
19058 6093 Acked Write 7 Done Sent Done false
19059 6094 Acked Read 8 Done Sent Done false
19060 6095 Acked Flush 0 Done Sent Done false
19061 6096 Acked Write 8 Done Sent Done false
19062 6097 Acked Read 4 Done Sent Done false
19063 6098 Acked Read 2 Done Sent Done false
19064 6099 Acked Flush 0 Done Sent Done false
19065 6100 Acked Flush 0 Done Sent Done false
19066 6101 Acked Read 5 Done Sent Done false
19067 6102 Acked Read 7 Done Sent Done false
19068 6103 Acked Read 10 Done Sent Done false
19069 6104 Acked Read 3 Done Sent Done false
19070 6105 Acked Write 3 Done Sent Done false
19071 6106 Acked Write 7 Done Sent Done false
19072 6107 Acked Write 8 Done Sent Done false
19073 6108 Acked Flush 0 Done Sent Done false
19074 6109 Acked Write 3 Done Sent Done false
19075 6110 Acked Read 1 Done Sent Done false
19076 6111 Acked Write 2 Done Sent Done false
19077 6112 Acked Read 2 Done Sent Done false
19078 6113 Acked Read 3 Done Sent Done false
19079 6114 Acked Write 7 Done Sent Done false
19080 6115 Acked Read 10 Done Sent Done false
19081 6116 Acked Read 3 Done Sent Done false
19082 6117 Acked Read 2 Done Sent Done false
19083 6118 Acked Write 2 Done Sent Done false
19084 6119 Acked Read 3 Done Sent Done false
19085 6120 Acked Read 7 Done Sent Done false
19086 6121 Acked Write 6 Done Sent Done false
19087 6122 Acked Read 9 Done Sent Done false
19088 6123 Acked Write 10 Done Sent Done false
19089 6124 Acked Write 5 Done Sent Done false
19090 6125 Acked Read 10 Done Sent Done false
19091 6126 Acked Write 2 Done Sent Done false
19092 6127 Acked Read 6 Done Sent Done false
19093 6128 Acked Write 3 Done Sent Done false
19094 6129 Acked Read 4 Done Sent Done false
19095 6130 Acked Read 2 Done Sent Done false
19096 6131 Acked Read 7 Done Sent Done false
19097 STATES DS:0 DS:1 DS:2 TOTAL
19098 Sent 0 301 0 301
19099 Done 301 0 301 602
19100 Skipped 0 0 0 0
19101 Error 0 0 0 0
19102 Last Flush: Some(JobId(6108)) Some(JobId(5830)) Some(JobId(6108))
19103 Downstairs last five completed: 5830 5829 5828 5827 5826
19104 Upstairs last five completed: 6131 6130 6129 6128 6127
19105 CLIENT: Up:0 ds:301 act:2
191062025-03-17T22:20:28.035ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
191072025-03-17T22:20:28.995ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
191082025-03-17T22:20:29.495ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
191092025-03-17T22:20:30.015ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
191102025-03-17T22:20:30.999ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
19111 ----------------------------------------------------------------
19112 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19113 States: Active Offline Active
19114 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19115 5831 Acked Read 8 Done Sent Done false
19116 5832 Acked Read 5 Done Sent Done false
19117 5833 Acked Write 7 Done Sent Done false
19118 5834 Acked Write 2 Done Sent Done false
19119 5835 Acked Read 7 Done Sent Done false
19120 5836 Acked Read 4 Done Sent Done false
19121 5837 Acked Read 10 Done Sent Done false
19122 5838 Acked Read 9 Done Sent Done false
19123 5839 Acked Read 4 Done Sent Done false
19124 5840 Acked Read 6 Done Sent Done false
19125 5841 Acked Read 6 Done Sent Done false
19126 5842 Acked Read 2 Done Sent Done false
19127 5843 Acked Read 6 Done Sent Done false
19128 5844 Acked Read 8 Done Sent Done false
19129 5845 Acked Read 4 Done Sent Done false
19130 5846 Acked Read 7 Done Sent Done false
19131 5847 Acked Read 8 Done Sent Done false
19132 5848 Acked Write 6 Done Sent Done false
19133 5849 Acked Read 4 Done Sent Done false
19134 5850 Acked Read 10 Done Sent Done false
19135 5851 Acked Read 3 Done Sent Done false
19136 5852 Acked Write 8 Done Sent Done false
19137 5853 Acked Write 2 Done Sent Done false
19138 5854 Acked Write 4 Done Sent Done false
19139 5855 Acked Read 1 Done Sent Done false
19140 5856 Acked Flush 0 Done Sent Done false
19141 5857 Acked Flush 0 Done Sent Done false
19142 5858 Acked Write 9 Done Sent Done false
19143 5859 Acked Read 8 Done Sent Done false
19144 5860 Acked Read 3 Done Sent Done false
19145 5861 Acked Write 7 Done Sent Done false
19146 5862 Acked Write 6 Done Sent Done false
19147 5863 Acked Read 5 Done Sent Done false
19148 5864 Acked Read 8 Done Sent Done false
19149 5865 Acked Read 8 Done Sent Done false
19150 5866 Acked Write 8 Done Sent Done false
19151 5867 Acked Read 4 Done Sent Done false
19152 5868 Acked Write 1 Done Sent Done false
19153 5869 Acked Flush 0 Done Sent Done false
19154 5870 Acked Flush 0 Done Sent Done false
19155 5871 Acked Write 2 Done Sent Done false
19156 5872 Acked Write 9 Done Sent Done false
19157 5873 Acked Write 8 Done Sent Done false
19158 5874 Acked Write 5 Done Sent Done false
19159 5875 Acked Read 3 Done Sent Done false
19160 5876 Acked Read 2 Done Sent Done false
19161 5877 Acked Flush 0 Done Sent Done false
19162 5878 Acked Read 8 Done Sent Done false
19163 5879 Acked Write 4 Done Sent Done false
19164 5880 Acked Write 8 Done Sent Done false
19165 5881 Acked Read 8 Done Sent Done false
19166 5882 Acked Write 9 Done Sent Done false
19167 5883 Acked Read 3 Done Sent Done false
19168 5884 Acked Read 1 Done Sent Done false
19169 5885 Acked Read 9 Done Sent Done false
19170 5886 Acked Read 3 Done Sent Done false
19171 5887 Acked Read 5 Done Sent Done false
19172 5888 Acked Write 7 Done Sent Done false
19173 5889 Acked Write 6 Done Sent Done false
19174 5890 Acked Write 9 Done Sent Done false
19175 5891 Acked Read 9 Done Sent Done false
19176 5892 Acked Read 1 Done Sent Done false
19177 5893 Acked Write 8 Done Sent Done false
19178 5894 Acked Write 7 Done Sent Done false
19179 5895 Acked Write 4 Done Sent Done false
19180 5896 Acked Read 9 Done Sent Done false
19181 5897 Acked Read 1 Done Sent Done false
19182 5898 Acked Write 5 Done Sent Done false
19183 5899 Acked Write 4 Done Sent Done false
19184 5900 Acked Write 5 Done Sent Done false
19185 5901 Acked Read 9 Done Sent Done false
19186 5902 Acked Write 5 Done Sent Done false
19187 5903 Acked Read 6 Done Sent Done false
19188 5904 Acked Write 6 Done Sent Done false
19189 5905 Acked Write 10 Done Sent Done false
19190 5906 Acked Write 5 Done Sent Done false
19191 5907 Acked Read 1 Done Sent Done false
19192 5908 Acked Read 1 Done Sent Done false
19193 5909 Acked Write 3 Done Sent Done false
19194 5910 Acked Flush 0 Done Sent Done false
19195 5911 Acked Read 1 Done Sent Done false
19196 5912 Acked Read 5 Done Sent Done false
19197 5913 Acked Write 3 Done Sent Done false
19198 5914 Acked Read 4 Done Sent Done false
19199 5915 Acked Write 4 Done Sent Done false
19200 5916 Acked Write 9 Done Sent Done false
19201 5917 Acked Read 1 Done Sent Done false
19202 5918 Acked Write 8 Done Sent Done false
19203 5919 Acked Write 7 Done Sent Done false
19204 5920 Acked Flush 0 Done Sent Done false
19205 5921 Acked Write 4 Done Sent Done false
19206 5922 Acked Write 3 Done Sent Done false
19207 5923 Acked Read 1 Done Sent Done false
19208 5924 Acked Read 5 Done Sent Done false
19209 5925 Acked Read 9 Done Sent Done false
19210 5926 Acked Read 4 Done Sent Done false
19211 5927 Acked Write 1 Done Sent Done false
19212 5928 Acked Read 4 Done Sent Done false
19213 5929 Acked Read 9 Done Sent Done false
19214 5930 Acked Read 9 Done Sent Done false
19215 5931 Acked Write 9 Done Sent Done false
19216 5932 Acked Write 2 Done Sent Done false
19217 5933 Acked Read 6 Done Sent Done false
19218 5934 Acked Read 3 Done Sent Done false
19219 5935 Acked Read 6 Done Sent Done false
19220 5936 Acked Read 6 Done Sent Done false
19221 5937 Acked Read 3 Done Sent Done false
19222 5938 Acked Write 7 Done Sent Done false
19223 5939 Acked Write 1 Done Sent Done false
19224 5940 Acked Read 10 Done Sent Done false
19225 5941 Acked Write 9 Done Sent Done false
19226 5942 Acked Read 2 Done Sent Done false
19227 5943 Acked Flush 0 Done Sent Done false
19228 5944 Acked Read 9 Done Sent Done false
19229 5945 Acked Flush 0 Done Sent Done false
19230 5946 Acked Write 3 Done Sent Done false
19231 5947 Acked Write 10 Done Sent Done false
19232 5948 Acked Read 7 Done Sent Done false
19233 5949 Acked Write 8 Done Sent Done false
19234 5950 Acked Write 8 Done Sent Done false
19235 5951 Acked Read 2 Done Sent Done false
19236 5952 Acked Write 10 Done Sent Done false
19237 5953 Acked Write 2 Done Sent Done false
19238 5954 Acked Write 7 Done Sent Done false
19239 5955 Acked Flush 0 Done Sent Done false
19240 5956 Acked Write 5 Done Sent Done false
19241 5957 Acked Write 1 Done Sent Done false
19242 5958 Acked Read 5 Done Sent Done false
19243 5959 Acked Write 7 Done Sent Done false
19244 5960 Acked Write 9 Done Sent Done false
19245 5961 Acked Read 1 Done Sent Done false
19246 5962 Acked Read 9 Done Sent Done false
19247 5963 Acked Read 3 Done Sent Done false
19248 5964 Acked Read 4 Done Sent Done false
19249 5965 Acked Write 5 Done Sent Done false
19250 5966 Acked Read 6 Done Sent Done false
19251 5967 Acked Write 5 Done Sent Done false
19252 5968 Acked Read 7 Done Sent Done false
19253 5969 Acked Read 10 Done Sent Done false
19254 5970 Acked Read 9 Done Sent Done false
19255 5971 Acked Write 6 Done Sent Done false
19256 5972 Acked Read 9 Done Sent Done false
19257 5973 Acked Read 1 Done Sent Done false
19258 5974 Acked Write 1 Done Sent Done false
19259 5975 Acked Read 2 Done Sent Done false
19260 5976 Acked Write 6 Done Sent Done false
19261 5977 Acked Write 1 Done Sent Done false
19262 5978 Acked Read 10 Done Sent Done false
19263 5979 Acked Read 10 Done Sent Done false
19264 5980 Acked Flush 0 Done Sent Done false
19265 5981 Acked Read 6 Done Sent Done false
19266 5982 Acked Write 3 Done Sent Done false
19267 5983 Acked Flush 0 Done Sent Done false
19268 5984 Acked Read 4 Done Sent Done false
19269 5985 Acked Write 7 Done Sent Done false
19270 5986 Acked Read 1 Done Sent Done false
19271 5987 Acked Read 9 Done Sent Done false
19272 5988 Acked Write 2 Done Sent Done false
19273 5989 Acked Write 10 Done Sent Done false
19274 5990 Acked Write 3 Done Sent Done false
19275 5991 Acked Read 3 Done Sent Done false
19276 5992 Acked Write 2 Done Sent Done false
19277 5993 Acked Write 9 Done Sent Done false
19278 5994 Acked Write 8 Done Sent Done false
19279 5995 Acked Write 6 Done Sent Done false
19280 5996 Acked Read 10 Done Sent Done false
19281 5997 Acked Flush 0 Done Sent Done false
19282 5998 Acked Read 2 Done Sent Done false
19283 5999 Acked Read 3 Done Sent Done false
19284 6000 Acked Read 9 Done Sent Done false
19285 6001 Acked Read 4 Done Sent Done false
19286 6002 Acked Read 6 Done Sent Done false
19287 6003 Acked Write 10 Done Sent Done false
19288 6004 Acked Read 3 Done Sent Done false
19289 6005 Acked Read 7 Done Sent Done false
19290 6006 Acked Read 8 Done Sent Done false
19291 6007 Acked Write 7 Done Sent Done false
19292 6008 Acked Read 8 Done Sent Done false
19293 6009 Acked Write 8 Done Sent Done false
19294 6010 Acked Write 9 Done Sent Done false
19295 6011 Acked Write 4 Done Sent Done false
19296 6012 Acked Write 3 Done Sent Done false
19297 6013 Acked Read 8 Done Sent Done false
19298 6014 Acked Read 8 Done Sent Done false
19299 6015 Acked Read 8 Done Sent Done false
19300 6016 Acked Write 2 Done Sent Done false
19301 6017 Acked Read 4 Done Sent Done false
19302 6018 Acked Write 9 Done Sent Done false
19303 6019 Acked Write 10 Done Sent Done false
19304 6020 Acked Read 6 Done Sent Done false
19305 6021 Acked Flush 0 Done Sent Done false
19306 6022 Acked Write 3 Done Sent Done false
19307 6023 Acked Write 5 Done Sent Done false
19308 6024 Acked Read 9 Done Sent Done false
19309 6025 Acked Flush 0 Done Sent Done false
19310 6026 Acked Write 7 Done Sent Done false
19311 6027 Acked Read 9 Done Sent Done false
19312 6028 Acked Write 4 Done Sent Done false
19313 6029 Acked Read 9 Done Sent Done false
19314 6030 Acked Read 1 Done Sent Done false
19315 6031 Acked Write 7 Done Sent Done false
19316 6032 Acked Read 5 Done Sent Done false
19317 6033 Acked Read 8 Done Sent Done false
19318 6034 Acked Read 5 Done Sent Done false
19319 6035 Acked Write 4 Done Sent Done false
19320 6036 Acked Read 8 Done Sent Done false
19321 6037 Acked Read 5 Done Sent Done false
19322 6038 Acked Flush 0 Done Sent Done false
19323 6039 Acked Write 7 Done Sent Done false
19324 6040 Acked Flush 0 Done Sent Done false
19325 6041 Acked Write 9 Done Sent Done false
19326 6042 Acked Read 6 Done Sent Done false
19327 6043 Acked Write 6 Done Sent Done false
19328 6044 Acked Write 6 Done Sent Done false
19329 6045 Acked Read 10 Done Sent Done false
19330 6046 Acked Read 4 Done Sent Done false
19331 6047 Acked Write 10 Done Sent Done false
19332 6048 Acked Read 4 Done Sent Done false
19333 6049 Acked Write 1 Done Sent Done false
19334 6050 Acked Read 1 Done Sent Done false
19335 6051 Acked Read 9 Done Sent Done false
19336 6052 Acked Read 5 Done Sent Done false
19337 6053 Acked Write 8 Done Sent Done false
19338 6054 Acked Read 7 Done Sent Done false
19339 6055 Acked Read 3 Done Sent Done false
19340 6056 Acked Write 6 Done Sent Done false
19341 6057 Acked Read 6 Done Sent Done false
19342 6058 Acked Read 4 Done Sent Done false
19343 6059 Acked Write 5 Done Sent Done false
19344 6060 Acked Flush 0 Done Sent Done false
19345 6061 Acked Read 3 Done Sent Done false
19346 6062 Acked Read 4 Done Sent Done false
19347 6063 Acked Read 10 Done Sent Done false
19348 6064 Acked Write 2 Done Sent Done false
19349 6065 Acked Write 4 Done Sent Done false
19350 6066 Acked Write 10 Done Sent Done false
19351 6067 Acked Read 10 Done Sent Done false
19352 6068 Acked Read 7 Done Sent Done false
19353 6069 Acked Read 2 Done Sent Done false
19354 6070 Acked Write 9 Done Sent Done false
19355 6071 Acked Read 1 Done Sent Done false
19356 6072 Acked Write 5 Done Sent Done false
19357 6073 Acked Write 9 Done Sent Done false
19358 6074 Acked Read 9 Done Sent Done false
19359 6075 Acked Read 2 Done Sent Done false
19360 6076 Acked Flush 0 Done Sent Done false
19361 6077 Acked Read 3 Done Sent Done false
19362 6078 Acked Read 5 Done Sent Done false
19363 6079 Acked Write 3 Done Sent Done false
19364 6080 Acked Write 1 Done Sent Done false
19365 6081 Acked Read 8 Done Sent Done false
19366 6082 Acked Read 6 Done Sent Done false
19367 6083 Acked Write 2 Done Sent Done false
19368 6084 Acked Write 9 Done Sent Done false
19369 6085 Acked Read 3 Done Sent Done false
19370 6086 Acked Flush 0 Done Sent Done false
19371 6087 Acked Write 5 Done Sent Done false
19372 6088 Acked Write 6 Done Sent Done false
19373 6089 Acked Read 6 Done Sent Done false
19374 6090 Acked Read 7 Done Sent Done false
19375 6091 Acked Write 3 Done Sent Done false
19376 6092 Acked Read 8 Done Sent Done false
19377 6093 Acked Write 7 Done Sent Done false
19378 6094 Acked Read 8 Done Sent Done false
19379 6095 Acked Flush 0 Done Sent Done false
19380 6096 Acked Write 8 Done Sent Done false
19381 6097 Acked Read 4 Done Sent Done false
19382 6098 Acked Read 2 Done Sent Done false
19383 6099 Acked Flush 0 Done Sent Done false
19384 6100 Acked Flush 0 Done Sent Done false
19385 6101 Acked Read 5 Done Sent Done false
19386 6102 Acked Read 7 Done Sent Done false
19387 6103 Acked Read 10 Done Sent Done false
19388 6104 Acked Read 3 Done Sent Done false
19389 6105 Acked Write 3 Done Sent Done false
19390 6106 Acked Write 7 Done Sent Done false
19391 6107 Acked Write 8 Done Sent Done false
19392 6108 Acked Flush 0 Done Sent Done false
19393 6109 Acked Write 3 Done Sent Done false
19394 6110 Acked Read 1 Done Sent Done false
19395 6111 Acked Write 2 Done Sent Done false
19396 6112 Acked Read 2 Done Sent Done false
19397 6113 Acked Read 3 Done Sent Done false
19398 6114 Acked Write 7 Done Sent Done false
19399 6115 Acked Read 10 Done Sent Done false
19400 6116 Acked Read 3 Done Sent Done false
19401 6117 Acked Read 2 Done Sent Done false
19402 6118 Acked Write 2 Done Sent Done false
19403 6119 Acked Read 3 Done Sent Done false
19404 6120 Acked Read 7 Done Sent Done false
19405 6121 Acked Write 6 Done Sent Done false
19406 6122 Acked Read 9 Done Sent Done false
19407 6123 Acked Write 10 Done Sent Done false
19408 6124 Acked Write 5 Done Sent Done false
19409 6125 Acked Read 10 Done Sent Done false
19410 6126 Acked Write 2 Done Sent Done false
19411 6127 Acked Read 6 Done Sent Done false
19412 6128 Acked Write 3 Done Sent Done false
19413 6129 Acked Read 4 Done Sent Done false
19414 6130 Acked Read 2 Done Sent Done false
19415 6131 Acked Read 7 Done Sent Done false
19416 STATES DS:0 DS:1 DS:2 TOTAL
19417 Sent 0 301 0 301
19418 Done 301 0 301 602
19419 Skipped 0 0 0 0
19420 Error 0 0 0 0
19421 Last Flush: Some(JobId(6108)) Some(JobId(5830)) Some(JobId(6108))
19422 Downstairs last five completed: 5830 5829 5828 5827 5826
19423 Upstairs last five completed: 6131 6130 6129 6128 6127
19424 CLIENT: Up:0 ds:301 act:2
194252025-03-17T22:20:32.038ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
194262025-03-17T22:20:33.003ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
194272025-03-17T22:20:33.280ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
194282025-03-17T22:20:33.280ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
194292025-03-17T22:20:33.280ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
194302025-03-17T22:20:33.280ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
194312025-03-17T22:20:33.280ZINFOcrucible: client task is exiting = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
19432 [8820] got start action
19433 [8820] I am going to restart
19434 Starting downstairs at port 8820
19435 Make output file at "/tmp/dsc/downstairs-8820.txt"
19436 [8820][1] reports Starting
19437 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1223)
19438 [8820][1] reports Running
194392025-03-17T22:20:34.004ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
194402025-03-17T22:20:35.015ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
19441 ----------------------------------------------------------------
19442 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19443 States: Active Offline Active
19444 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19445 5831 Acked Read 8 Done Sent Done false
19446 5832 Acked Read 5 Done Sent Done false
19447 5833 Acked Write 7 Done Sent Done false
19448 5834 Acked Write 2 Done Sent Done false
19449 5835 Acked Read 7 Done Sent Done false
19450 5836 Acked Read 4 Done Sent Done false
19451 5837 Acked Read 10 Done Sent Done false
19452 5838 Acked Read 9 Done Sent Done false
19453 5839 Acked Read 4 Done Sent Done false
19454 5840 Acked Read 6 Done Sent Done false
19455 5841 Acked Read 6 Done Sent Done false
19456 5842 Acked Read 2 Done Sent Done false
19457 5843 Acked Read 6 Done Sent Done false
19458 5844 Acked Read 8 Done Sent Done false
19459 5845 Acked Read 4 Done Sent Done false
19460 5846 Acked Read 7 Done Sent Done false
19461 5847 Acked Read 8 Done Sent Done false
19462 5848 Acked Write 6 Done Sent Done false
19463 5849 Acked Read 4 Done Sent Done false
19464 5850 Acked Read 10 Done Sent Done false
19465 5851 Acked Read 3 Done Sent Done false
19466 5852 Acked Write 8 Done Sent Done false
19467 5853 Acked Write 2 Done Sent Done false
19468 5854 Acked Write 4 Done Sent Done false
19469 5855 Acked Read 1 Done Sent Done false
19470 5856 Acked Flush 0 Done Sent Done false
19471 5857 Acked Flush 0 Done Sent Done false
19472 5858 Acked Write 9 Done Sent Done false
19473 5859 Acked Read 8 Done Sent Done false
19474 5860 Acked Read 3 Done Sent Done false
19475 5861 Acked Write 7 Done Sent Done false
19476 5862 Acked Write 6 Done Sent Done false
19477 5863 Acked Read 5 Done Sent Done false
19478 5864 Acked Read 8 Done Sent Done false
19479 5865 Acked Read 8 Done Sent Done false
19480 5866 Acked Write 8 Done Sent Done false
19481 5867 Acked Read 4 Done Sent Done false
19482 5868 Acked Write 1 Done Sent Done false
19483 5869 Acked Flush 0 Done Sent Done false
19484 5870 Acked Flush 0 Done Sent Done false
19485 5871 Acked Write 2 Done Sent Done false
19486 5872 Acked Write 9 Done Sent Done false
19487 5873 Acked Write 8 Done Sent Done false
19488 5874 Acked Write 5 Done Sent Done false
19489 5875 Acked Read 3 Done Sent Done false
19490 5876 Acked Read 2 Done Sent Done false
19491 5877 Acked Flush 0 Done Sent Done false
19492 5878 Acked Read 8 Done Sent Done false
19493 5879 Acked Write 4 Done Sent Done false
19494 5880 Acked Write 8 Done Sent Done false
19495 5881 Acked Read 8 Done Sent Done false
19496 5882 Acked Write 9 Done Sent Done false
19497 5883 Acked Read 3 Done Sent Done false
19498 5884 Acked Read 1 Done Sent Done false
19499 5885 Acked Read 9 Done Sent Done false
19500 5886 Acked Read 3 Done Sent Done false
19501 5887 Acked Read 5 Done Sent Done false
19502 5888 Acked Write 7 Done Sent Done false
19503 5889 Acked Write 6 Done Sent Done false
19504 5890 Acked Write 9 Done Sent Done false
19505 5891 Acked Read 9 Done Sent Done false
19506 5892 Acked Read 1 Done Sent Done false
19507 5893 Acked Write 8 Done Sent Done false
19508 5894 Acked Write 7 Done Sent Done false
19509 5895 Acked Write 4 Done Sent Done false
19510 5896 Acked Read 9 Done Sent Done false
19511 5897 Acked Read 1 Done Sent Done false
19512 5898 Acked Write 5 Done Sent Done false
19513 5899 Acked Write 4 Done Sent Done false
19514 5900 Acked Write 5 Done Sent Done false
19515 5901 Acked Read 9 Done Sent Done false
19516 5902 Acked Write 5 Done Sent Done false
19517 5903 Acked Read 6 Done Sent Done false
19518 5904 Acked Write 6 Done Sent Done false
19519 5905 Acked Write 10 Done Sent Done false
19520 5906 Acked Write 5 Done Sent Done false
19521 5907 Acked Read 1 Done Sent Done false
19522 5908 Acked Read 1 Done Sent Done false
19523 5909 Acked Write 3 Done Sent Done false
19524 5910 Acked Flush 0 Done Sent Done false
19525 5911 Acked Read 1 Done Sent Done false
19526 5912 Acked Read 5 Done Sent Done false
19527 5913 Acked Write 3 Done Sent Done false
19528 5914 Acked Read 4 Done Sent Done false
19529 5915 Acked Write 4 Done Sent Done false
19530 5916 Acked Write 9 Done Sent Done false
19531 5917 Acked Read 1 Done Sent Done false
19532 5918 Acked Write 8 Done Sent Done false
19533 5919 Acked Write 7 Done Sent Done false
19534 5920 Acked Flush 0 Done Sent Done false
19535 5921 Acked Write 4 Done Sent Done false
19536 5922 Acked Write 3 Done Sent Done false
19537 5923 Acked Read 1 Done Sent Done false
19538 5924 Acked Read 5 Done Sent Done false
19539 5925 Acked Read 9 Done Sent Done false
19540 5926 Acked Read 4 Done Sent Done false
19541 5927 Acked Write 1 Done Sent Done false
19542 5928 Acked Read 4 Done Sent Done false
19543 5929 Acked Read 9 Done Sent Done false
19544 5930 Acked Read 9 Done Sent Done false
19545 5931 Acked Write 9 Done Sent Done false
19546 5932 Acked Write 2 Done Sent Done false
19547 5933 Acked Read 6 Done Sent Done false
19548 5934 Acked Read 3 Done Sent Done false
19549 5935 Acked Read 6 Done Sent Done false
19550 5936 Acked Read 6 Done Sent Done false
19551 5937 Acked Read 3 Done Sent Done false
19552 5938 Acked Write 7 Done Sent Done false
19553 5939 Acked Write 1 Done Sent Done false
19554 5940 Acked Read 10 Done Sent Done false
19555 5941 Acked Write 9 Done Sent Done false
19556 5942 Acked Read 2 Done Sent Done false
19557 5943 Acked Flush 0 Done Sent Done false
19558 5944 Acked Read 9 Done Sent Done false
19559 5945 Acked Flush 0 Done Sent Done false
19560 5946 Acked Write 3 Done Sent Done false
19561 5947 Acked Write 10 Done Sent Done false
19562 5948 Acked Read 7 Done Sent Done false
19563 5949 Acked Write 8 Done Sent Done false
19564 5950 Acked Write 8 Done Sent Done false
19565 5951 Acked Read 2 Done Sent Done false
19566 5952 Acked Write 10 Done Sent Done false
19567 5953 Acked Write 2 Done Sent Done false
19568 5954 Acked Write 7 Done Sent Done false
19569 5955 Acked Flush 0 Done Sent Done false
19570 5956 Acked Write 5 Done Sent Done false
19571 5957 Acked Write 1 Done Sent Done false
19572 5958 Acked Read 5 Done Sent Done false
19573 5959 Acked Write 7 Done Sent Done false
19574 5960 Acked Write 9 Done Sent Done false
19575 5961 Acked Read 1 Done Sent Done false
19576 5962 Acked Read 9 Done Sent Done false
19577 5963 Acked Read 3 Done Sent Done false
19578 5964 Acked Read 4 Done Sent Done false
19579 5965 Acked Write 5 Done Sent Done false
19580 5966 Acked Read 6 Done Sent Done false
19581 5967 Acked Write 5 Done Sent Done false
19582 5968 Acked Read 7 Done Sent Done false
19583 5969 Acked Read 10 Done Sent Done false
19584 5970 Acked Read 9 Done Sent Done false
19585 5971 Acked Write 6 Done Sent Done false
19586 5972 Acked Read 9 Done Sent Done false
19587 5973 Acked Read 1 Done Sent Done false
19588 5974 Acked Write 1 Done Sent Done false
19589 5975 Acked Read 2 Done Sent Done false
19590 5976 Acked Write 6 Done Sent Done false
19591 5977 Acked Write 1 Done Sent Done false
19592 5978 Acked Read 10 Done Sent Done false
19593 5979 Acked Read 10 Done Sent Done false
19594 5980 Acked Flush 0 Done Sent Done false
19595 5981 Acked Read 6 Done Sent Done false
19596 5982 Acked Write 3 Done Sent Done false
19597 5983 Acked Flush 0 Done Sent Done false
19598 5984 Acked Read 4 Done Sent Done false
19599 5985 Acked Write 7 Done Sent Done false
19600 5986 Acked Read 1 Done Sent Done false
19601 5987 Acked Read 9 Done Sent Done false
19602 5988 Acked Write 2 Done Sent Done false
19603 5989 Acked Write 10 Done Sent Done false
19604 5990 Acked Write 3 Done Sent Done false
19605 5991 Acked Read 3 Done Sent Done false
19606 5992 Acked Write 2 Done Sent Done false
19607 5993 Acked Write 9 Done Sent Done false
19608 5994 Acked Write 8 Done Sent Done false
19609 5995 Acked Write 6 Done Sent Done false
19610 5996 Acked Read 10 Done Sent Done false
19611 5997 Acked Flush 0 Done Sent Done false
19612 5998 Acked Read 2 Done Sent Done false
19613 5999 Acked Read 3 Done Sent Done false
19614 6000 Acked Read 9 Done Sent Done false
19615 6001 Acked Read 4 Done Sent Done false
19616 6002 Acked Read 6 Done Sent Done false
19617 6003 Acked Write 10 Done Sent Done false
19618 6004 Acked Read 3 Done Sent Done false
19619 6005 Acked Read 7 Done Sent Done false
19620 6006 Acked Read 8 Done Sent Done false
19621 6007 Acked Write 7 Done Sent Done false
19622 6008 Acked Read 8 Done Sent Done false
19623 6009 Acked Write 8 Done Sent Done false
19624 6010 Acked Write 9 Done Sent Done false
19625 6011 Acked Write 4 Done Sent Done false
19626 6012 Acked Write 3 Done Sent Done false
19627 6013 Acked Read 8 Done Sent Done false
19628 6014 Acked Read 8 Done Sent Done false
19629 6015 Acked Read 8 Done Sent Done false
19630 6016 Acked Write 2 Done Sent Done false
19631 6017 Acked Read 4 Done Sent Done false
19632 6018 Acked Write 9 Done Sent Done false
19633 6019 Acked Write 10 Done Sent Done false
19634 6020 Acked Read 6 Done Sent Done false
19635 6021 Acked Flush 0 Done Sent Done false
19636 6022 Acked Write 3 Done Sent Done false
19637 6023 Acked Write 5 Done Sent Done false
19638 6024 Acked Read 9 Done Sent Done false
19639 6025 Acked Flush 0 Done Sent Done false
19640 6026 Acked Write 7 Done Sent Done false
19641 6027 Acked Read 9 Done Sent Done false
19642 6028 Acked Write 4 Done Sent Done false
19643 6029 Acked Read 9 Done Sent Done false
19644 6030 Acked Read 1 Done Sent Done false
19645 6031 Acked Write 7 Done Sent Done false
19646 6032 Acked Read 5 Done Sent Done false
19647 6033 Acked Read 8 Done Sent Done false
19648 6034 Acked Read 5 Done Sent Done false
19649 6035 Acked Write 4 Done Sent Done false
19650 6036 Acked Read 8 Done Sent Done false
19651 6037 Acked Read 5 Done Sent Done false
19652 6038 Acked Flush 0 Done Sent Done false
19653 6039 Acked Write 7 Done Sent Done false
19654 6040 Acked Flush 0 Done Sent Done false
19655 6041 Acked Write 9 Done Sent Done false
19656 6042 Acked Read 6 Done Sent Done false
19657 6043 Acked Write 6 Done Sent Done false
19658 6044 Acked Write 6 Done Sent Done false
19659 6045 Acked Read 10 Done Sent Done false
19660 6046 Acked Read 4 Done Sent Done false
19661 6047 Acked Write 10 Done Sent Done false
19662 6048 Acked Read 4 Done Sent Done false
19663 6049 Acked Write 1 Done Sent Done false
19664 6050 Acked Read 1 Done Sent Done false
19665 6051 Acked Read 9 Done Sent Done false
19666 6052 Acked Read 5 Done Sent Done false
19667 6053 Acked Write 8 Done Sent Done false
19668 6054 Acked Read 7 Done Sent Done false
19669 6055 Acked Read 3 Done Sent Done false
19670 6056 Acked Write 6 Done Sent Done false
19671 6057 Acked Read 6 Done Sent Done false
19672 6058 Acked Read 4 Done Sent Done false
19673 6059 Acked Write 5 Done Sent Done false
19674 6060 Acked Flush 0 Done Sent Done false
19675 6061 Acked Read 3 Done Sent Done false
19676 6062 Acked Read 4 Done Sent Done false
19677 6063 Acked Read 10 Done Sent Done false
19678 6064 Acked Write 2 Done Sent Done false
19679 6065 Acked Write 4 Done Sent Done false
19680 6066 Acked Write 10 Done Sent Done false
19681 6067 Acked Read 10 Done Sent Done false
19682 6068 Acked Read 7 Done Sent Done false
19683 6069 Acked Read 2 Done Sent Done false
19684 6070 Acked Write 9 Done Sent Done false
19685 6071 Acked Read 1 Done Sent Done false
19686 6072 Acked Write 5 Done Sent Done false
19687 6073 Acked Write 9 Done Sent Done false
19688 6074 Acked Read 9 Done Sent Done false
19689 6075 Acked Read 2 Done Sent Done false
19690 6076 Acked Flush 0 Done Sent Done false
19691 6077 Acked Read 3 Done Sent Done false
19692 6078 Acked Read 5 Done Sent Done false
19693 6079 Acked Write 3 Done Sent Done false
19694 6080 Acked Write 1 Done Sent Done false
19695 6081 Acked Read 8 Done Sent Done false
19696 6082 Acked Read 6 Done Sent Done false
19697 6083 Acked Write 2 Done Sent Done false
19698 6084 Acked Write 9 Done Sent Done false
19699 6085 Acked Read 3 Done Sent Done false
19700 6086 Acked Flush 0 Done Sent Done false
19701 6087 Acked Write 5 Done Sent Done false
19702 6088 Acked Write 6 Done Sent Done false
19703 6089 Acked Read 6 Done Sent Done false
19704 6090 Acked Read 7 Done Sent Done false
19705 6091 Acked Write 3 Done Sent Done false
19706 6092 Acked Read 8 Done Sent Done false
19707 6093 Acked Write 7 Done Sent Done false
19708 6094 Acked Read 8 Done Sent Done false
19709 6095 Acked Flush 0 Done Sent Done false
19710 6096 Acked Write 8 Done Sent Done false
19711 6097 Acked Read 4 Done Sent Done false
19712 6098 Acked Read 2 Done Sent Done false
19713 6099 Acked Flush 0 Done Sent Done false
19714 6100 Acked Flush 0 Done Sent Done false
19715 6101 Acked Read 5 Done Sent Done false
19716 6102 Acked Read 7 Done Sent Done false
19717 6103 Acked Read 10 Done Sent Done false
19718 6104 Acked Read 3 Done Sent Done false
19719 6105 Acked Write 3 Done Sent Done false
19720 6106 Acked Write 7 Done Sent Done false
19721 6107 Acked Write 8 Done Sent Done false
19722 6108 Acked Flush 0 Done Sent Done false
19723 6109 Acked Write 3 Done Sent Done false
19724 6110 Acked Read 1 Done Sent Done false
19725 6111 Acked Write 2 Done Sent Done false
19726 6112 Acked Read 2 Done Sent Done false
19727 6113 Acked Read 3 Done Sent Done false
19728 6114 Acked Write 7 Done Sent Done false
19729 6115 Acked Read 10 Done Sent Done false
19730 6116 Acked Read 3 Done Sent Done false
19731 6117 Acked Read 2 Done Sent Done false
19732 6118 Acked Write 2 Done Sent Done false
19733 6119 Acked Read 3 Done Sent Done false
19734 6120 Acked Read 7 Done Sent Done false
19735 6121 Acked Write 6 Done Sent Done false
19736 6122 Acked Read 9 Done Sent Done false
19737 6123 Acked Write 10 Done Sent Done false
19738 6124 Acked Write 5 Done Sent Done false
19739 6125 Acked Read 10 Done Sent Done false
19740 6126 Acked Write 2 Done Sent Done false
19741 6127 Acked Read 6 Done Sent Done false
19742 6128 Acked Write 3 Done Sent Done false
19743 6129 Acked Read 4 Done Sent Done false
19744 6130 Acked Read 2 Done Sent Done false
19745 6131 Acked Read 7 Done Sent Done false
19746 STATES DS:0 DS:1 DS:2 TOTAL
19747 Sent 0 301 0 301
19748 Done 301 0 301 602
19749 Skipped 0 0 0 0
19750 Error 0 0 0 0
19751 Last Flush: Some(JobId(6108)) Some(JobId(5830)) Some(JobId(6108))
19752 Downstairs last five completed: 5830 5829 5828 5827 5826
19753 Upstairs last five completed: 6131 6130 6129 6128 6127
19754 CLIENT: Up:0 ds:301 act:2
197552025-03-17T22:20:37.009ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
197562025-03-17T22:20:38.012ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
197572025-03-17T22:20:39.012ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
19758 ----------------------------------------------------------------
19759 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19760 States: Active Offline Active
19761 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
19762 5831 Acked Read 8 Done Sent Done false
19763 5832 Acked Read 5 Done Sent Done false
19764 5833 Acked Write 7 Done Sent Done false
19765 5834 Acked Write 2 Done Sent Done false
19766 5835 Acked Read 7 Done Sent Done false
19767 5836 Acked Read 4 Done Sent Done false
19768 5837 Acked Read 10 Done Sent Done false
19769 5838 Acked Read 9 Done Sent Done false
19770 5839 Acked Read 4 Done Sent Done false
19771 5840 Acked Read 6 Done Sent Done false
19772 5841 Acked Read 6 Done Sent Done false
19773 5842 Acked Read 2 Done Sent Done false
19774 5843 Acked Read 6 Done Sent Done false
19775 5844 Acked Read 8 Done Sent Done false
19776 5845 Acked Read 4 Done Sent Done false
19777 5846 Acked Read 7 Done Sent Done false
19778 5847 Acked Read 8 Done Sent Done false
19779 5848 Acked Write 6 Done Sent Done false
19780 5849 Acked Read 4 Done Sent Done false
19781 5850 Acked Read 10 Done Sent Done false
19782 5851 Acked Read 3 Done Sent Done false
19783 5852 Acked Write 8 Done Sent Done false
19784 5853 Acked Write 2 Done Sent Done false
19785 5854 Acked Write 4 Done Sent Done false
19786 5855 Acked Read 1 Done Sent Done false
19787 5856 Acked Flush 0 Done Sent Done false
19788 5857 Acked Flush 0 Done Sent Done false
19789 5858 Acked Write 9 Done Sent Done false
19790 5859 Acked Read 8 Done Sent Done false
19791 5860 Acked Read 3 Done Sent Done false
19792 5861 Acked Write 7 Done Sent Done false
19793 5862 Acked Write 6 Done Sent Done false
19794 5863 Acked Read 5 Done Sent Done false
19795 5864 Acked Read 8 Done Sent Done false
19796 5865 Acked Read 8 Done Sent Done false
19797 5866 Acked Write 8 Done Sent Done false
19798 5867 Acked Read 4 Done Sent Done false
19799 5868 Acked Write 1 Done Sent Done false
19800 5869 Acked Flush 0 Done Sent Done false
19801 5870 Acked Flush 0 Done Sent Done false
19802 5871 Acked Write 2 Done Sent Done false
19803 5872 Acked Write 9 Done Sent Done false
19804 5873 Acked Write 8 Done Sent Done false
19805 5874 Acked Write 5 Done Sent Done false
19806 5875 Acked Read 3 Done Sent Done false
19807 5876 Acked Read 2 Done Sent Done false
19808 5877 Acked Flush 0 Done Sent Done false
19809 5878 Acked Read 8 Done Sent Done false
19810 5879 Acked Write 4 Done Sent Done false
19811 5880 Acked Write 8 Done Sent Done false
19812 5881 Acked Read 8 Done Sent Done false
19813 5882 Acked Write 9 Done Sent Done false
19814 5883 Acked Read 3 Done Sent Done false
19815 5884 Acked Read 1 Done Sent Done false
19816 5885 Acked Read 9 Done Sent Done false
19817 5886 Acked Read 3 Done Sent Done false
19818 5887 Acked Read 5 Done Sent Done false
19819 5888 Acked Write 7 Done Sent Done false
19820 5889 Acked Write 6 Done Sent Done false
19821 5890 Acked Write 9 Done Sent Done false
19822 5891 Acked Read 9 Done Sent Done false
19823 5892 Acked Read 1 Done Sent Done false
19824 5893 Acked Write 8 Done Sent Done false
19825 5894 Acked Write 7 Done Sent Done false
19826 5895 Acked Write 4 Done Sent Done false
19827 5896 Acked Read 9 Done Sent Done false
19828 5897 Acked Read 1 Done Sent Done false
19829 5898 Acked Write 5 Done Sent Done false
19830 5899 Acked Write 4 Done Sent Done false
19831 5900 Acked Write 5 Done Sent Done false
19832 5901 Acked Read 9 Done Sent Done false
19833 5902 Acked Write 5 Done Sent Done false
19834 5903 Acked Read 6 Done Sent Done false
19835 5904 Acked Write 6 Done Sent Done false
19836 5905 Acked Write 10 Done Sent Done false
19837 5906 Acked Write 5 Done Sent Done false
19838 5907 Acked Read 1 Done Sent Done false
19839 5908 Acked Read 1 Done Sent Done false
19840 5909 Acked Write 3 Done Sent Done false
19841 5910 Acked Flush 0 Done Sent Done false
19842 5911 Acked Read 1 Done Sent Done false
19843 5912 Acked Read 5 Done Sent Done false
19844 5913 Acked Write 3 Done Sent Done false
19845 5914 Acked Read 4 Done Sent Done false
19846 5915 Acked Write 4 Done Sent Done false
19847 5916 Acked Write 9 Done Sent Done false
19848 5917 Acked Read 1 Done Sent Done false
19849 5918 Acked Write 8 Done Sent Done false
19850 5919 Acked Write 7 Done Sent Done false
19851 5920 Acked Flush 0 Done Sent Done false
19852 5921 Acked Write 4 Done Sent Done false
19853 5922 Acked Write 3 Done Sent Done false
19854 5923 Acked Read 1 Done Sent Done false
19855 5924 Acked Read 5 Done Sent Done false
19856 5925 Acked Read 9 Done Sent Done false
19857 5926 Acked Read 4 Done Sent Done false
19858 5927 Acked Write 1 Done Sent Done false
19859 5928 Acked Read 4 Done Sent Done false
19860 5929 Acked Read 9 Done Sent Done false
19861 5930 Acked Read 9 Done Sent Done false
19862 5931 Acked Write 9 Done Sent Done false
19863 5932 Acked Write 2 Done Sent Done false
19864 5933 Acked Read 6 Done Sent Done false
19865 5934 Acked Read 3 Done Sent Done false
19866 5935 Acked Read 6 Done Sent Done false
19867 5936 Acked Read 6 Done Sent Done false
19868 5937 Acked Read 3 Done Sent Done false
19869 5938 Acked Write 7 Done Sent Done false
19870 5939 Acked Write 1 Done Sent Done false
19871 5940 Acked Read 10 Done Sent Done false
19872 5941 Acked Write 9 Done Sent Done false
19873 5942 Acked Read 2 Done Sent Done false
19874 5943 Acked Flush 0 Done Sent Done false
19875 5944 Acked Read 9 Done Sent Done false
19876 5945 Acked Flush 0 Done Sent Done false
19877 5946 Acked Write 3 Done Sent Done false
19878 5947 Acked Write 10 Done Sent Done false
19879 5948 Acked Read 7 Done Sent Done false
19880 5949 Acked Write 8 Done Sent Done false
19881 5950 Acked Write 8 Done Sent Done false
19882 5951 Acked Read 2 Done Sent Done false
19883 5952 Acked Write 10 Done Sent Done false
19884 5953 Acked Write 2 Done Sent Done false
19885 5954 Acked Write 7 Done Sent Done false
19886 5955 Acked Flush 0 Done Sent Done false
19887 5956 Acked Write 5 Done Sent Done false
19888 5957 Acked Write 1 Done Sent Done false
19889 5958 Acked Read 5 Done Sent Done false
19890 5959 Acked Write 7 Done Sent Done false
19891 5960 Acked Write 9 Done Sent Done false
19892 5961 Acked Read 1 Done Sent Done false
19893 5962 Acked Read 9 Done Sent Done false
19894 5963 Acked Read 3 Done Sent Done false
19895 5964 Acked Read 4 Done Sent Done false
19896 5965 Acked Write 5 Done Sent Done false
19897 5966 Acked Read 6 Done Sent Done false
19898 5967 Acked Write 5 Done Sent Done false
19899 5968 Acked Read 7 Done Sent Done false
19900 5969 Acked Read 10 Done Sent Done false
19901 5970 Acked Read 9 Done Sent Done false
19902 5971 Acked Write 6 Done Sent Done false
19903 5972 Acked Read 9 Done Sent Done false
19904 5973 Acked Read 1 Done Sent Done false
19905 5974 Acked Write 1 Done Sent Done false
19906 5975 Acked Read 2 Done Sent Done false
19907 5976 Acked Write 6 Done Sent Done false
19908 5977 Acked Write 1 Done Sent Done false
19909 5978 Acked Read 10 Done Sent Done false
19910 5979 Acked Read 10 Done Sent Done false
19911 5980 Acked Flush 0 Done Sent Done false
19912 5981 Acked Read 6 Done Sent Done false
19913 5982 Acked Write 3 Done Sent Done false
19914 5983 Acked Flush 0 Done Sent Done false
19915 5984 Acked Read 4 Done Sent Done false
19916 5985 Acked Write 7 Done Sent Done false
19917 5986 Acked Read 1 Done Sent Done false
19918 5987 Acked Read 9 Done Sent Done false
19919 5988 Acked Write 2 Done Sent Done false
19920 5989 Acked Write 10 Done Sent Done false
19921 5990 Acked Write 3 Done Sent Done false
19922 5991 Acked Read 3 Done Sent Done false
19923 5992 Acked Write 2 Done Sent Done false
19924 5993 Acked Write 9 Done Sent Done false
19925 5994 Acked Write 8 Done Sent Done false
19926 5995 Acked Write 6 Done Sent Done false
19927 5996 Acked Read 10 Done Sent Done false
19928 5997 Acked Flush 0 Done Sent Done false
19929 5998 Acked Read 2 Done Sent Done false
19930 5999 Acked Read 3 Done Sent Done false
19931 6000 Acked Read 9 Done Sent Done false
19932 6001 Acked Read 4 Done Sent Done false
19933 6002 Acked Read 6 Done Sent Done false
19934 6003 Acked Write 10 Done Sent Done false
19935 6004 Acked Read 3 Done Sent Done false
19936 6005 Acked Read 7 Done Sent Done false
19937 6006 Acked Read 8 Done Sent Done false
19938 6007 Acked Write 7 Done Sent Done false
19939 6008 Acked Read 8 Done Sent Done false
19940 6009 Acked Write 8 Done Sent Done false
19941 6010 Acked Write 9 Done Sent Done false
19942 6011 Acked Write 4 Done Sent Done false
19943 6012 Acked Write 3 Done Sent Done false
19944 6013 Acked Read 8 Done Sent Done false
19945 6014 Acked Read 8 Done Sent Done false
19946 6015 Acked Read 8 Done Sent Done false
19947 6016 Acked Write 2 Done Sent Done false
19948 6017 Acked Read 4 Done Sent Done false
19949 6018 Acked Write 9 Done Sent Done false
19950 6019 Acked Write 10 Done Sent Done false
19951 6020 Acked Read 6 Done Sent Done false
19952 6021 Acked Flush 0 Done Sent Done false
19953 6022 Acked Write 3 Done Sent Done false
19954 6023 Acked Write 5 Done Sent Done false
19955 6024 Acked Read 9 Done Sent Done false
19956 6025 Acked Flush 0 Done Sent Done false
19957 6026 Acked Write 7 Done Sent Done false
19958 6027 Acked Read 9 Done Sent Done false
19959 6028 Acked Write 4 Done Sent Done false
19960 6029 Acked Read 9 Done Sent Done false
19961 6030 Acked Read 1 Done Sent Done false
19962 6031 Acked Write 7 Done Sent Done false
19963 6032 Acked Read 5 Done Sent Done false
19964 6033 Acked Read 8 Done Sent Done false
19965 6034 Acked Read 5 Done Sent Done false
19966 6035 Acked Write 4 Done Sent Done false
19967 6036 Acked Read 8 Done Sent Done false
19968 6037 Acked Read 5 Done Sent Done false
19969 6038 Acked Flush 0 Done Sent Done false
19970 6039 Acked Write 7 Done Sent Done false
19971 6040 Acked Flush 0 Done Sent Done false
19972 6041 Acked Write 9 Done Sent Done false
19973 6042 Acked Read 6 Done Sent Done false
19974 6043 Acked Write 6 Done Sent Done false
19975 6044 Acked Write 6 Done Sent Done false
19976 6045 Acked Read 10 Done Sent Done false
19977 6046 Acked Read 4 Done Sent Done false
19978 6047 Acked Write 10 Done Sent Done false
19979 6048 Acked Read 4 Done Sent Done false
19980 6049 Acked Write 1 Done Sent Done false
19981 6050 Acked Read 1 Done Sent Done false
19982 6051 Acked Read 9 Done Sent Done false
19983 6052 Acked Read 5 Done Sent Done false
19984 6053 Acked Write 8 Done Sent Done false
19985 6054 Acked Read 7 Done Sent Done false
19986 6055 Acked Read 3 Done Sent Done false
19987 6056 Acked Write 6 Done Sent Done false
19988 6057 Acked Read 6 Done Sent Done false
19989 6058 Acked Read 4 Done Sent Done false
19990 6059 Acked Write 5 Done Sent Done false
19991 6060 Acked Flush 0 Done Sent Done false
19992 6061 Acked Read 3 Done Sent Done false
19993 6062 Acked Read 4 Done Sent Done false
19994 6063 Acked Read 10 Done Sent Done false
19995 6064 Acked Write 2 Done Sent Done false
19996 6065 Acked Write 4 Done Sent Done false
19997 6066 Acked Write 10 Done Sent Done false
19998 6067 Acked Read 10 Done Sent Done false
19999 6068 Acked Read 7 Done Sent Done false
20000 6069 Acked Read 2 Done Sent Done false
20001 6070 Acked Write 9 Done Sent Done false
20002 6071 Acked Read 1 Done Sent Done false
20003 6072 Acked Write 5 Done Sent Done false
20004 6073 Acked Write 9 Done Sent Done false
20005 6074 Acked Read 9 Done Sent Done false
20006 6075 Acked Read 2 Done Sent Done false
20007 6076 Acked Flush 0 Done Sent Done false
20008 6077 Acked Read 3 Done Sent Done false
20009 6078 Acked Read 5 Done Sent Done false
20010 6079 Acked Write 3 Done Sent Done false
20011 6080 Acked Write 1 Done Sent Done false
20012 6081 Acked Read 8 Done Sent Done false
20013 6082 Acked Read 6 Done Sent Done false
20014 6083 Acked Write 2 Done Sent Done false
20015 6084 Acked Write 9 Done Sent Done false
20016 6085 Acked Read 3 Done Sent Done false
20017 6086 Acked Flush 0 Done Sent Done false
20018 6087 Acked Write 5 Done Sent Done false
20019 6088 Acked Write 6 Done Sent Done false
20020 6089 Acked Read 6 Done Sent Done false
20021 6090 Acked Read 7 Done Sent Done false
20022 6091 Acked Write 3 Done Sent Done false
20023 6092 Acked Read 8 Done Sent Done false
20024 6093 Acked Write 7 Done Sent Done false
20025 6094 Acked Read 8 Done Sent Done false
20026 6095 Acked Flush 0 Done Sent Done false
20027 6096 Acked Write 8 Done Sent Done false
20028 6097 Acked Read 4 Done Sent Done false
20029 6098 Acked Read 2 Done Sent Done false
20030 6099 Acked Flush 0 Done Sent Done false
20031 6100 Acked Flush 0 Done Sent Done false
20032 6101 Acked Read 5 Done Sent Done false
20033 6102 Acked Read 7 Done Sent Done false
20034 6103 Acked Read 10 Done Sent Done false
20035 6104 Acked Read 3 Done Sent Done false
20036 6105 Acked Write 3 Done Sent Done false
20037 6106 Acked Write 7 Done Sent Done false
20038 6107 Acked Write 8 Done Sent Done false
20039 6108 Acked Flush 0 Done Sent Done false
20040 6109 Acked Write 3 Done Sent Done false
20041 6110 Acked Read 1 Done Sent Done false
20042 6111 Acked Write 2 Done Sent Done false
20043 6112 Acked Read 2 Done Sent Done false
20044 6113 Acked Read 3 Done Sent Done false
20045 6114 Acked Write 7 Done Sent Done false
20046 6115 Acked Read 10 Done Sent Done false
20047 6116 Acked Read 3 Done Sent Done false
20048 6117 Acked Read 2 Done Sent Done false
20049 6118 Acked Write 2 Done Sent Done false
20050 6119 Acked Read 3 Done Sent Done false
20051 6120 Acked Read 7 Done Sent Done false
20052 6121 Acked Write 6 Done Sent Done false
20053 6122 Acked Read 9 Done Sent Done false
20054 6123 Acked Write 10 Done Sent Done false
20055 6124 Acked Write 5 Done Sent Done false
20056 6125 Acked Read 10 Done Sent Done false
20057 6126 Acked Write 2 Done Sent Done false
20058 6127 Acked Read 6 Done Sent Done false
20059 6128 Acked Write 3 Done Sent Done false
20060 6129 Acked Read 4 Done Sent Done false
20061 6130 Acked Read 2 Done Sent Done false
20062 6131 Acked Read 7 Done Sent Done false
20063 STATES DS:0 DS:1 DS:2 TOTAL
20064 Sent 0 301 0 301
20065 Done 301 0 301 602
20066 Skipped 0 0 0 0
20067 Error 0 0 0 0
20068 Last Flush: Some(JobId(6108)) Some(JobId(5830)) Some(JobId(6108))
20069 Downstairs last five completed: 5830 5829 5828 5827 5826
20070 Upstairs last five completed: 6131 6130 6129 6128 6127
20071 CLIENT: Up:0 ds:301 act:2
200722025-03-17T22:20:40.042ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
200732025-03-17T22:20:41.016ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
200742025-03-17T22:20:42.017ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
200752025-03-17T22:20:43.020ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
200762025-03-17T22:20:43.281ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
200772025-03-17T22:20:43.281ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:55089) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
200782025-03-17T22:20:43.282ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
200792025-03-17T22:20:43.283ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
200802025-03-17T22:20:43.283ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
200812025-03-17T22:20:43.283ZINFOcrucible: send last flush ID to this DS: Some(JobId(5830)) = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
200822025-03-17T22:20:43.283ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
200832025-03-17T22:20:43.283ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(5830)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
200842025-03-17T22:20:43.283ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(5830)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
20085 ----------------------------------------------------------------
20086 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:23
20087 States: Active Active Active
20088 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20089 6109 Acked Write 3 Done Done Done true
20090 6110 Acked Read 1 Done Done Done true
20091 6111 Acked Write 2 Done Done Done true
20092 6112 Acked Read 2 Done Done Done true
20093 6113 Acked Read 3 Done Done Done true
20094 6114 Acked Write 7 Done Done Done true
20095 6115 Acked Read 10 Done Done Done true
20096 6116 Acked Read 3 Done Done Done true
20097 6117 Acked Read 2 Done Done Done true
20098 6118 Acked Write 2 Done Done Done true
20099 6119 Acked Read 3 Done Done Done true
20100 6120 Acked Read 7 Done Done Done true
20101 6121 Acked Write 6 Done Done Done true
20102 6122 Acked Read 9 Done Done Done true
20103 6123 Acked Write 10 Done Done Done true
20104 6124 Acked Write 5 Done Done Done true
20105 6125 Acked Read 10 Done Done Done true
20106 6126 Acked Write 2 Done Done Done true
20107 6127 Acked Read 6 Done Done Done true
20108 6128 Acked Write 3 Done Done Done true
20109 6129 Acked Read 4 Done Done Done true
20110 6130 Acked Read 2 Done Done Done true
20111 6131 Acked Read 7 Done Done Done true
20112 STATES DS:0 DS:1 DS:2 TOTAL
20113 Sent 0 0 0 0
20114 Done 23 23 23 69
20115 Skipped 0 0 0 0
20116 Error 0 0 0 0
20117 Last Flush: Some(JobId(6108)) Some(JobId(6108)) Some(JobId(6108))
20118 Downstairs last five completed: 6108 6107 6106 6105 6104
20119 Upstairs last five completed: 6131 6130 6129 6128 6127
20120 CLIENT: Up:0 ds:23 act:3
20121 ----------------------------------------------------------------
20122 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
20123 Downstairs last five completed: 6132 6131 6130 6129 6128
20124 Upstairs last five completed: 6132 6131 6130 6129 6128
20125 CLIENT: Up:0 ds:0 act:3
20126 Replay: All jobs finished
20127 Main task has work to do, go find it
20128 got dsc Stop(2)
20129 stop 2
20130 Mar 17 22:20:47.544 INFO request completed, latency_us[8830] Got stop action so:false kr:false
20131 : 364, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 218c2ab9-61f3-48cd-ae98-dd3559bfcef2, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
20132 Mar 17 22:20:47.545 INFO request completed, latency_us: 143, response_code: 200, uri: /state/cid/2, method: GET, req_id: 620dd002-d37c-40a4-b222-d8310fa8236e, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
201332025-03-17T22:20:47.545ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
201342025-03-17T22:20:47.545ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
201352025-03-17T22:20:47.545ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
201362025-03-17T22:20:47.545ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
20137 [8830] Exited with: ExitStatus(unix_wait_status(9))
20138 [8830][2] reports Exit
20139 Mar 17 22:20:51.548 INFO request completed, latency_us: 218, response_code: 200, uri: /state/cid/2, method: GET, req_id: 8db6afd6-9b73-4bc4-a730-63bfd6a8fe21, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
20140 001/300 Write block 1886 len 24576 data: 6 5 5 4 3 2
20141 002/300 Read block 3292 len 8192
20142 003/300 Write block 1390 len 36864 data: 4 4 5 6 5 5 5 5 5
20143 004/300 Read block 2198 len 32768
20144 005/300 Read block 1836 len 20480
20145 006/300 Write block 4860 len 20480 data: 8 8 9 9 9
20146 007/300 Write block 4003 len 8192 data: 5 4
20147 008/300 Write block 25 len 24576 data: 5 6 6 7 5 4
20148 009/300 Write block 534 len 16384 data: 2 3 3 3
20149 010/300 Write block 2115 len 36864 data: 5 5 7 8 8 7 7 5 5
20150 011/300 Write block 4798 len 12288 data: 4 4 4
20151 012/300 Read block 921 len 28672
20152 013/300 Write block 1646 len 12288 data: 5 5 5
20153 014/300 Read block 3164 len 36864
20154 015/300 Write block 4095 len 4096 data: 6
20155 016/300 Read block 1658 len 24576
20156 017/300 Flush
20157 018/300 Read block 633 len 8192
20158 019/300 Read block 2344 len 28672
20159 020/300 Write block 4042 len 8192 data: 4 3
20160 021/300 Write block 1039 len 12288 data: 6 7 8
20161 022/300 Read block 1317 len 40960
20162 023/300 Read block 3065 len 20480
20163 024/300 Flush
20164 025/300 Write block 3887 len 20480 data: 5 4 3 3 2
20165 026/300 Read block 1597 len 4096
20166 027/300 Flush
20167 028/300 Read block 254 len 20480
20168 029/300 Write block 2508 len 8192 data: 4 3
20169 030/300 Read block 4167 len 20480
20170 031/300 Write block 2531 len 32768 data: 5 4 4 5 4 3 2 3
20171 032/300 Read block 3983 len 16384
20172 033/300 Write block 4991 len 28672 data: 3 3 3 3 3 3 3
20173 034/300 Read block 944 len 40960
20174 035/300 Read block 896 len 4096
20175 036/300 Write block 855 len 32768 data: 6 5 5 6 5 4 4 4
20176 037/300 Read block 113 len 12288
20177 038/300 Read block 1229 len 36864
20178 039/300 Read block 4347 len 32768
20179 040/300 Flush
20180 041/300 Read block 1588 len 12288
20181 042/300 Write block 2763 len 28672 data: 3 2 2 2 4 5 5
20182 043/300 Read block 4990 len 16384
20183 044/300 Read block 3823 len 20480
20184 045/300 Write block 2275 len 8192 data: 6 5
20185 046/300 Write block 2162 len 28672 data: 3 3 3 4 4 3 3
20186 047/300 Write block 839 len 4096 data: 5
20187 048/300 Flush
20188 049/300 Write block 415 len 32768 data: 5 5 5 5 4 4 3 3
20189 050/300 Write block 879 len 16384 data: 3 3 3 3
20190 051/300 Flush
20191 052/300 Write block 3005 len 12288 data: 6 5 5
20192 053/300 Read block 1148 len 24576
20193 054/300 Read block 2237 len 20480
20194 055/300 Read block 4290 len 40960
20195 056/300 Read block 3402 len 8192
20196 057/300 Read block 4947 len 8192
20197 058/300 Read block 945 len 36864
20198 059/300 Read block 1030 len 32768
20199 060/300 Read block 1130 len 16384
20200 061/300 Write block 1002 len 36864 data: 2 2 2 2 3 2 2 2 3
20201 062/300 Read block 800 len 36864
20202 063/300 Flush
20203 064/300 Read block 3368 len 8192
20204 065/300 Read block 4815 len 28672
20205 066/300 Flush
20206 067/300 Read block 782 len 20480
20207 068/300 Read block 1339 len 12288
20208 069/300 Write block 1483 len 32768 data: 5 5 6 5 5 5 6 5
20209 070/300 Write block 1829 len 32768 data: 4 4 5 5 5 5 4 3
20210 071/300 Write block 1154 len 4096 data: 4
20211 072/300 Write block 4681 len 40960 data: 3 3 2 3 3 3 3 4 5 5
20212 073/300 Write block 2168 len 24576 data: 4 3 4 4 4 4
20213 074/300 Write block 2714 len 32768 data: 2 2 3 3 3 3 3 3
20214 075/300 Write block 2827 len 36864 data: 9 6 6 7 4 3 5 5 6
20215 076/300 Write block 1607 len 40960 data: 2 3 3 3 4 4 5 4 4 4
20216 077/300 Read block 284 len 4096
20217 078/300 Read block 1600 len 36864
20218 079/300 Write block 4829 len 16384 data: 3 4 4 3
20219 080/300 Read block 578 len 24576
20220 081/300 Write block 4239 len 28672 data: 4 3 4 4 4 3 4
20221 082/300 Read block 237 len 12288
20222 083/300 Read block 4912 len 20480
20223 084/300 Read block 4544 len 12288
20224 085/300 Read block 2174 len 40960
20225 086/300 Flush
20226 087/300 Flush
20227 088/300 Read block 1690 len 28672
20228 089/300 Write block 4568 len 32768 data: 4 4 3 2 2 4 4 6
20229 090/300 Write block 4031 len 32768 data: 4 4 3 3 2 3 3 3
20230 091/300 Read block 688 len 40960
20231 092/300 Read block 3528 len 40960
20232 093/300 Read block 1735 len 16384
20233 094/300 Read block 3611 len 32768
20234 095/300 Read block 94 len 8192
20235 096/300 Write block 463 len 20480 data: 4 6 7 6 5
20236 097/300 Flush
20237 098/300 Read block 3904 len 32768
20238 099/300 Read block 3780 len 36864
20239 100/300 Write block 3390 len 4096 data: 4
20240 101/300 Read block 518 len 12288
20241 102/300 Flush
20242 103/300 Write block 3113 len 20480 data: 3 4 4 3 3
20243 104/300 Read block 935 len 4096
20244 105/300 Read block 73 len 20480
20245 106/300 Write block 1326 len 40960 data: 6 6 6 4 4 4 3 3 3 3
20246 107/300 Flush
20247 108/300 Write block 697 len 4096 data: 4
20248 109/300 Read block 1772 len 16384
20249 110/300 Read block 1908 len 28672
20250 111/300 Write block 3377 len 28672 data: 4 5 5 4 3 2 3
20251 112/300 Write block 417 len 20480 data: 6 6 5 5 4
20252 113/300 Write block 849 len 28672 data: 6 7 7 6 6 7 7
20253 114/300 Read block 3155 len 16384
20254 115/300 Read block 4095 len 16384
20255 116/300 Write block 2105 len 20480 data: 4 6 7 6 5
20256 117/300 Read block 2038 len 36864
20257 118/300 Read block 3143 len 28672
20258 119/300 Write block 3434 len 40960 data: 7 6 5 6 4 4 4 5 4 5
20259 120/300 Read block 448 len 12288
20260 121/300 Read block 3240 len 28672
20261 122/300 Read block 950 len 36864
20262 123/300 Read block 1656 len 8192
20263 124/300 Read block 3000 len 32768
20264 125/300 Write block 3725 len 24576 data: 4 4 4 4 5 8
20265 126/300 Write block 4137 len 36864 data: 4 3 3 4 4 4 4 4 4
20266 127/300 Write block 1614 len 20480 data: 5 5 5 5 3
20267 128/300 Write block 3997 len 20480 data: 4 5 5 6 6
20268 129/300 Write block 1217 len 40960 data: 6 5 7 6 7 6 5 5 4 4
20269 130/300 Write block 3888 len 36864 data: 5 4 4 3 3 4 5 6 7
20270 131/300 Read block 3272 len 8192
20271 132/300 Write block 684 len 4096 data: 6
20272 133/300 Write block 2972 len 4096 data: 4
20273 134/300 Read block 490 len 40960
20274 135/300 Write block 3894 len 4096 data: 6
20275 136/300 Read block 1741 len 20480
20276 137/300 Write block 3829 len 20480 data: 8 8 5 5 5
20277 138/300 Write block 1075 len 12288 data: 7 7 7
20278 139/300 Flush
20279 140/300 Read block 996 len 20480
20280 141/300 Write block 2691 len 36864 data: 5 5 4 4 4 6 7 7 6
20281 142/300 Read block 4755 len 28672
20282 143/300 Write block 2672 len 40960 data: 5 5 6 6 6 5 5 3 3 3
20283 144/300 Write block 166 len 28672 data: 5 5 5 5 6 6 6
20284 145/300 Read block 2848 len 4096
20285 146/300 Write block 2427 len 40960 data: 5 3 4 4 3 4 4 5 6 6
20286 147/300 Write block 3151 len 20480 data: 4 4 3 3 4
20287 148/300 Flush
20288 149/300 Flush
20289 150/300 Write block 4405 len 28672 data: 6 6 5 5 6 4 2
20290 151/300 Write block 617 len 24576 data: 5 3 3 3 3 3
20291 152/300 Flush
20292 153/300 Write block 2692 len 28672 data: 6 5 5 5 7 8 8
20293 154/300 Write block 4826 len 16384 data: 3 3 3 4
20294 155/300 Read block 1989 len 12288
20295 156/300 Write block 2900 len 40960 data: 5 4 3 2 3 4 4 3 3 5
20296 157/300 Write block 4019 len 8192 data: 4 3
20297 158/300 Flush
20298 159/300 Flush
20299 160/300 Write block 292 len 36864 data: 3 3 3 3 2 4 4 4 4
20300 161/300 Write block 4661 len 20480 data: 3 3 4 3 3
20301 162/300 Write block 3794 len 32768 data: 3 4 5 5 5 7 6 6
20302 163/300 Read block 3016 len 8192
20303 164/300 Read block 826 len 12288
20304 165/300 Read block 3199 len 28672
20305 166/300 Read block 3129 len 36864
20306 167/300 Write block 1631 len 12288 data: 2 3 3
20307 168/300 Write block 3618 len 4096 data: 5
20308 169/300 Write block 2254 len 12288 data: 3 3 4
20309 170/300 Read block 2788 len 40960
20310 171/300 Flush
20311 172/300 Write block 2523 len 16384 data: 2 2 2 2
20312 173/300 Write block 2066 len 16384 data: 5 5 5 6
20313 174/300 Write block 4314 len 36864 data: 3 3 3 3 3 3 2 2 2
20314 175/300 Write block 427 len 16384 data: 2 2 3 2
20315 176/300 Write block 4684 len 4096 data: 4
20316 177/300 Read block 4477 len 16384
20317 178/300 Read block 1036 len 12288
20318 179/300 Flush
20319 180/300 Read block 2842 len 16384
20320 181/300 Write block 298 len 20480 data: 5 5 5 5 3
20321 182/300 Write block 2443 len 16384 data: 5 4 4 4
20322 183/300 Read block 1028 len 16384
20323 184/300 Flush
20324 185/300 Write block 914 len 16384 data: 4 4 3 3
20325 186/300 Write block 1411 len 16384 data: 6 5 6 5
20326 187/300 Write block 746 len 4096 data: 3
20327 188/300 Read block 220 len 28672
20328 189/300 Read block 1083 len 24576
20329 190/300 Read block 1274 len 40960
20330 191/300 Read block 1374 len 8192
20331 192/300 Write block 3403 len 16384 data: 6 5 5 5
20332 193/300 Flush
20333 194/300 Read block 185 len 8192
20334 195/300 Write block 4902 len 4096 data: 7
20335 196/300 Write block 2143 len 36864 data: 6 5 4 5 5 5 5 4 3
20336 197/300 Read block 2771 len 12288
20337 198/300 Write block 1357 len 16384 data: 7 8 8 8
20338 199/300 Write block 676 len 16384 data: 3 2 2 3
20339 200/300 Flush
20340 201/300 Write block 4692 len 12288 data: 4 4 3
20341 202/300 Flush
20342 203/300 Read block 3176 len 40960
20343 204/300 Write block 2754 len 40960 data: 2 3 4 6 6 6 6 5 5 4
20344 205/300 Read block 3303 len 12288
20345 206/300 Flush
20346 207/300 Write block 4796 len 40960 data: 4 4 5 5 5 3 4 4 4 5
20347 208/300 Write block 3650 len 40960 data: 4 4 4 5 4 4 4 4 4 4
20348 209/300 Read block 4978 len 40960
20349 210/300 Write block 1684 len 32768 data: 6 6 4 4 4 5 5 5
20350 211/300 Write block 4701 len 8192 data: 3 3
20351 212/300 Read block 2991 len 4096
20352 213/300 Read block 1119 len 12288
20353 214/300 Read block 4477 len 8192
20354 215/300 Read block 2621 len 40960
20355 216/300 Write block 2624 len 20480 data: 3 3 3 3 4
20356 217/300 Read block 311 len 8192
20357 218/300 Write block 400 len 24576 data: 6 6 5 7 7 5
20358 219/300 Write block 4929 len 4096 data: 5
20359 220/300 Read block 1305 len 20480
20360 221/300 Read block 4365 len 4096
20361 222/300 Read block 2805 len 8192
20362 223/300 Read block 2126 len 36864
20363 224/300 Flush
20364 225/300 Read block 3177 len 16384
20365 226/300 Write block 386 len 4096 data: 6
20366 227/300 Read block 4901 len 24576
20367 228/300 Flush
20368 229/300 Write block 594 len 24576 data: 6 6 5 3 4 4
20369 230/300 Write block 3552 len 28672 data: 4 3 3 3 3 4 4
20370 231/300 Read block 2695 len 28672
20371 232/300 Read block 1619 len 20480
20372 233/300 Read block 2917 len 28672
20373 234/300 Flush
20374 235/300 Flush
20375 236/300 Write block 4997 len 8192 data: 4 3
20376 237/300 Read block 4455 len 4096
20377 238/300 Flush
20378 239/300 Read block 3591 len 40960
20379 240/300 Read block 223 len 12288
20380 241/300 Write block 1467 len 8192 data: 5 4
20381 242/300 Read block 4319 len 32768
20382 243/300 Read block 4528 len 24576
20383 244/300 Write block 4541 len 12288 data: 3 4 4
20384 245/300 Read block 3142 len 16384
20385 246/300 Write block 3610 len 32768 data: 2 3 4 6 5 5 5 5
20386 247/300 Read block 4984 len 40960
20387 248/300 Write block 3090 len 12288 data: 4 4 3
20388 249/300 Write block 119 len 32768 data: 6 6 6 6 6 7 5 6
20389 250/300 Write block 4730 len 16384 data: 5 4 5 4
20390 251/300 Write block 4822 len 12288 data: 5 4 4
20391 252/300 Write block 71 len 36864 data: 3 3 3 4 5 6 5 5 5
20392 253/300 Read block 4685 len 4096
20393 254/300 Write block 3310 len 24576 data: 4 3 3 3 3 2
20394 255/300 Read block 2939 len 28672
20395 256/300 Write block 852 len 20480 data: 7 7 8 8 6
20396 257/300 Read block 2469 len 12288
20397 258/300 Read block 441 len 8192
20398 259/300 Flush
20399 260/300 Write block 4001 len 36864 data: 7 7 6 5 2 2 2 2 2
20400 261/300 Read block 388 len 16384
20401 262/300 Flush
20402 263/300 Read block 934 len 32768
20403 264/300 Read block 2221 len 12288
20404 265/300 Write block 4749 len 8192 data: 6 5
20405 266/300 Read block 2354 len 8192
20406 267/300 Read block 4426 len 8192
20407 268/300 Read block 2120 len 28672
20408 269/300 Read block 660 len 8192
20409 270/300 Read block 4984 len 8192
20410 271/300 Write block 4526 len 40960 data: 3 3 3 3 5 7 7 7 6 6
20411 272/300 Write block 2796 len 8192 data: 3 3
20412 273/300 Read block 4593 len 12288
20413 274/300 Write block 2740 len 12288 data: 3 3 3
20414 275/300 Read block 4509 len 28672
20415 276/300 Write block 1340 len 8192 data: 4 6
20416 277/300 Write block 4464 len 40960 data: 6 7 7 6 8 8 9 9 8 6
20417 278/300 Read block 2822 len 36864
20418 279/300 Read block 3888 len 12288
20419 280/300 Write block 1286 len 24576 data: 5 5 6 6 4 5
20420 281/300 Flush
20421 282/300 Write block 138 len 32768 data: 3 3 4 3 3 3 4 4
20422 283/300 Write block 173 len 4096 data: 6
20423 284/300 Read block 1477 len 24576
20424 285/300 Write block 2373 len 40960 data: 2 2 2 3 5 5 5 6 5 5
20425 286/300 Write block 1480 len 28672 data: 6 6 5 6 6 7 6
20426 287/300 Write block 1543 len 32768 data: 3 3 2 2 2 2 3 4
20427 288/300 Read block 2410 len 16384
20428 289/300 Flush
20429 290/300 Write block 239 len 8192 data: 7 7
20430 291/300 Read block 755 len 28672
20431 292/300 Write block 589 len 36864 data: 7 8 7 7 5 7 7 6 4
20432 293/300 Write block 2695 len 16384 data: 6 8 9 9
20433 294/300 Read block 1508 len 28672
20434 295/300 Write block 2666 len 4096 data: 5
20435 296/300 Read block 1417 len 40960
20436 297/300 Write block 46 len 4096 data: 5
20437 298/300 Read block 4068 len 8192
20438 299/300 Write block 3745 len 20480 data: 4 4 6 5 6
20439 300/300 Write block 3167 len 36864 data: 6 6 6 6 5 4 5 5 4
20440 301/300 Flush
20441 Main task has work to do, go find it
20442 got dsc Start(2)
20443 start 2
20444 Mar 17 22:20:51.782 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/2, method: POST, req_id: 04479f10-dcce-496b-b02f-b773731af680, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
20445 Replay: started 2, returned:Ok(())
20446 ----------------------------------------------------------------
20447 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20448 States: Active Active Offline
20449 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20450 6133 Acked Write 6 Done Done Sent false
20451 6134 Acked Read 2 Done Done Sent false
20452 6135 Acked Write 9 Done Done Sent false
20453 6136 Acked Read 8 Done Done Sent false
20454 6137 Acked Read 5 Done Done Sent false
20455 6138 Acked Write 5 Done Done Sent false
20456 6139 Acked Write 2 Done Done Sent false
20457 6140 Acked Write 6 Done Done Sent false
20458 6141 Acked Write 4 Done Done Sent false
20459 6142 Acked Write 9 Done Done Sent false
20460 6143 Acked Write 3 Done Done Sent false
20461 6144 Acked Read 7 Done Done Sent false
20462 6145 Acked Write 3 Done Done Sent false
20463 6146 Acked Read 9 Done Done Sent false
20464 6147 Acked Write 1 Done Done Sent false
20465 6148 Acked Read 6 Done Done Sent false
20466 6149 Acked Flush 0 Done Done Sent false
20467 6150 Acked Read 2 Done Done Sent false
20468 6151 Acked Read 7 Done Done Sent false
20469 6152 Acked Write 2 Done Done Sent false
20470 6153 Acked Write 3 Done Done Sent false
20471 6154 Acked Read 10 Done Done Sent false
20472 6155 Acked Read 5 Done Done Sent false
20473 6156 Acked Flush 0 Done Done Sent false
20474 6157 Acked Write 5 Done Done Sent false
20475 6158 Acked Read 1 Done Done Sent false
20476 6159 Acked Flush 0 Done Done Sent false
20477 6160 Acked Read 5 Done Done Sent false
20478 6161 Acked Write 2 Done Done Sent false
20479 6162 Acked Read 5 Done Done Sent false
20480 6163 Acked Write 8 Done Done Sent false
20481 6164 Acked Read 4 Done Done Sent false
20482 6165 Acked Write 7 Done Done Sent false
20483 6166 Acked Read 10 Done Done Sent false
20484 6167 Acked Read 1 Done Done Sent false
20485 6168 Acked Write 8 Done Done Sent false
20486 6169 Acked Read 3 Done Done Sent false
20487 6170 Acked Read 9 Done Done Sent false
20488 6171 Acked Read 8 Done Done Sent false
20489 6172 Acked Flush 0 Done Done Sent false
20490 6173 Acked Read 3 Done Done Sent false
20491 6174 Acked Write 7 Done Done Sent false
20492 6175 Acked Read 4 Done Done Sent false
20493 6176 Acked Read 5 Done Done Sent false
20494 6177 Acked Write 2 Done Done Sent false
20495 6178 Acked Write 7 Done Done Sent false
20496 6179 Acked Write 1 Done Done Sent false
20497 6180 Acked Flush 0 Done Done Sent false
20498 6181 Acked Write 8 Done Done Sent false
20499 6182 Acked Write 4 Done Done Sent false
20500 6183 Acked Flush 0 Done Done Sent false
20501 6184 Acked Write 3 Done Done Sent false
20502 6185 Acked Read 6 Done Done Sent false
20503 6186 Acked Read 5 Done Done Sent false
20504 6187 Acked Read 10 Done Done Sent false
20505 6188 Acked Read 2 Done Done Sent false
20506 6189 Acked Read 2 Done Done Sent false
20507 6190 Acked Read 9 Done Done Sent false
20508 6191 Acked Read 8 Done Done Sent false
20509 6192 Acked Read 4 Done Done Sent false
20510 6193 Acked Write 9 Done Done Sent false
20511 6194 Acked Read 9 Done Done Sent false
20512 6195 Acked Flush 0 Done Done Sent false
20513 6196 Acked Read 2 Done Done Sent false
20514 6197 Acked Read 7 Done Done Sent false
20515 6198 Acked Flush 0 Done Done Sent false
20516 6199 Acked Read 5 Done Done Sent false
20517 6200 Acked Read 3 Done Done Sent false
20518 6201 Acked Write 8 Done Done Sent false
20519 6202 Acked Write 8 Done Done Sent false
20520 6203 Acked Write 1 Done Done Sent false
20521 6204 Acked Write 10 Done Done Sent false
20522 6205 Acked Write 6 Done Done Sent false
20523 6206 Acked Write 8 Done Done Sent false
20524 6207 Acked Write 9 Done Done Sent false
20525 6208 Acked Write 10 Done Done Sent false
20526 6209 Acked Read 1 Done Done Sent false
20527 6210 Acked Read 9 Done Done Sent false
20528 6211 Acked Write 4 Done Done Sent false
20529 6212 Acked Read 6 Done Done Sent false
20530 6213 Acked Write 7 Done Done Sent false
20531 6214 Acked Read 3 Done Done Sent false
20532 6215 Acked Read 5 Done Done Sent false
20533 6216 Acked Read 3 Done Done Sent false
20534 6217 Acked Read 10 Done Done Sent false
20535 6218 Acked Flush 0 Done Done Sent false
20536 6219 Acked Flush 0 Done Done Sent false
20537 6220 Acked Read 7 Done Done Sent false
20538 6221 Acked Write 8 Done Done Sent false
20539 6222 Acked Write 8 Done Done Sent false
20540 6223 Acked Read 10 Done Done Sent false
20541 6224 Acked Read 10 Done Done Sent false
20542 6225 Acked Read 4 Done Done Sent false
20543 6226 Acked Read 8 Done Done Sent false
20544 6227 Acked Read 2 Done Done Sent false
20545 6228 Acked Write 5 Done Done Sent false
20546 6229 Acked Flush 0 Done Done Sent false
20547 6230 Acked Read 8 Done Done Sent false
20548 6231 Acked Read 9 Done Done Sent false
20549 6232 Acked Write 1 Done Done Sent false
20550 6233 Acked Read 3 Done Done Sent false
20551 6234 Acked Flush 0 Done Done Sent false
20552 6235 Acked Write 5 Done Done Sent false
20553 6236 Acked Read 1 Done Done Sent false
20554 6237 Acked Read 5 Done Done Sent false
20555 6238 Acked Write 10 Done Done Sent false
20556 6239 Acked Flush 0 Done Done Sent false
20557 6240 Acked Write 1 Done Done Sent false
20558 6241 Acked Read 4 Done Done Sent false
20559 6242 Acked Read 7 Done Done Sent false
20560 6243 Acked Write 7 Done Done Sent false
20561 6244 Acked Write 5 Done Done Sent false
20562 6245 Acked Write 7 Done Done Sent false
20563 6246 Acked Read 4 Done Done Sent false
20564 6247 Acked Read 4 Done Done Sent false
20565 6248 Acked Write 5 Done Done Sent false
20566 6249 Acked Read 9 Done Done Sent false
20567 6250 Acked Read 7 Done Done Sent false
20568 6251 Acked Write 10 Done Done Sent false
20569 6252 Acked Read 3 Done Done Sent false
20570 6253 Acked Read 7 Done Done Sent false
20571 6254 Acked Read 9 Done Done Sent false
20572 6255 Acked Read 2 Done Done Sent false
20573 6256 Acked Read 8 Done Done Sent false
20574 6257 Acked Write 6 Done Done Sent false
20575 6258 Acked Write 9 Done Done Sent false
20576 6259 Acked Write 5 Done Done Sent false
20577 6260 Acked Write 5 Done Done Sent false
20578 6261 Acked Write 10 Done Done Sent false
20579 6262 Acked Write 9 Done Done Sent false
20580 6263 Acked Read 2 Done Done Sent false
20581 6264 Acked Write 1 Done Done Sent false
20582 6265 Acked Write 1 Done Done Sent false
20583 6266 Acked Read 10 Done Done Sent false
20584 6267 Acked Write 1 Done Done Sent false
20585 6268 Acked Read 5 Done Done Sent false
20586 6269 Acked Write 5 Done Done Sent false
20587 6270 Acked Write 3 Done Done Sent false
20588 6271 Acked Flush 0 Done Done Sent false
20589 6272 Acked Read 5 Done Done Sent false
20590 6273 Acked Write 9 Done Done Sent false
20591 6274 Acked Read 7 Done Done Sent false
20592 6275 Acked Write 10 Done Done Sent false
20593 6276 Acked Write 7 Done Done Sent false
20594 6277 Acked Read 1 Done Done Sent false
20595 6278 Acked Write 10 Done Done Sent false
20596 6279 Acked Write 5 Done Done Sent false
20597 6280 Acked Flush 0 Done Done Sent false
20598 6281 Acked Flush 0 Done Done Sent false
20599 6282 Acked Write 7 Done Done Sent false
20600 6283 Acked Write 6 Done Done Sent false
20601 6284 Acked Flush 0 Done Done Sent false
20602 6285 Acked Write 7 Done Done Sent false
20603 6286 Acked Write 4 Done Done Sent false
20604 6287 Acked Read 3 Done Done Sent false
20605 6288 Acked Write 10 Done Done Sent false
20606 6289 Acked Write 2 Done Done Sent false
20607 6290 Acked Flush 0 Done Done Sent false
20608 6291 Acked Flush 0 Done Done Sent false
20609 6292 Acked Write 9 Done Done Sent false
20610 6293 Acked Write 5 Done Done Sent false
20611 6294 Acked Write 8 Done Done Sent false
20612 6295 Acked Read 2 Done Done Sent false
20613 6296 Acked Read 3 Done Done Sent false
20614 6297 Acked Read 7 Done Done Sent false
20615 6298 Acked Read 9 Done Done Sent false
20616 6299 Acked Write 3 Done Done Sent false
20617 6300 Acked Write 1 Done Done Sent false
20618 6301 Acked Write 3 Done Done Sent false
20619 6302 Acked Read 10 Done Done Sent false
20620 6303 Acked Flush 0 Done Done Sent false
20621 6304 Acked Write 4 Done Done Sent false
20622 6305 Acked Write 4 Done Done Sent false
20623 6306 Acked Write 9 Done Done Sent false
20624 6307 Acked Write 4 Done Done Sent false
20625 6308 Acked Write 1 Done Done Sent false
20626 6309 Acked Read 4 Done Done Sent false
20627 6310 Acked Read 3 Done Done Sent false
20628 6311 Acked Flush 0 Done Done Sent false
20629 6312 Acked Read 4 Done Done Sent false
20630 6313 Acked Write 5 Done Done Sent false
20631 6314 Acked Write 4 Done Done Sent false
20632 6315 Acked Read 4 Done Done Sent false
20633 6316 Acked Flush 0 Done Done Sent false
20634 6317 Acked Write 4 Done Done Sent false
20635 6318 Acked Write 4 Done Done Sent false
20636 6319 Acked Write 1 Done Done Sent false
20637 6320 Acked Read 7 Done Done Sent false
20638 6321 Acked Read 6 Done Done Sent false
20639 6322 Acked Read 10 Done Done Sent false
20640 6323 Acked Read 2 Done Done Sent false
20641 6324 Acked Write 4 Done Done Sent false
20642 6325 Acked Flush 0 Done Done Sent false
20643 6326 Acked Read 2 Done Done Sent false
20644 6327 Acked Write 1 Done Done Sent false
20645 6328 Acked Write 9 Done Done Sent false
20646 6329 Acked Read 3 Done Done Sent false
20647 6330 Acked Write 4 Done Done Sent false
20648 6331 Acked Write 4 Done Done Sent false
20649 6332 Acked Flush 0 Done Done Sent false
20650 6333 Acked Write 3 Done Done Sent false
20651 6334 Acked Flush 0 Done Done Sent false
20652 6335 Acked Read 10 Done Done Sent false
20653 6336 Acked Write 10 Done Done Sent false
20654 6337 Acked Read 3 Done Done Sent false
20655 6338 Acked Flush 0 Done Done Sent false
20656 6339 Acked Write 10 Done Done Sent false
20657 6340 Acked Write 10 Done Done Sent false
20658 6341 Acked Read 10 Done Done Sent false
20659 6342 Acked Write 8 Done Done Sent false
20660 6343 Acked Write 2 Done Done Sent false
20661 6344 Acked Read 1 Done Done Sent false
20662 6345 Acked Read 3 Done Done Sent false
20663 6346 Acked Read 2 Done Done Sent false
20664 6347 Acked Read 10 Done Done Sent false
20665 6348 Acked Write 5 Done Done Sent false
20666 6349 Acked Read 2 Done Done Sent false
20667 6350 Acked Write 6 Done Done Sent false
20668 6351 Acked Write 1 Done Done Sent false
20669 6352 Acked Read 5 Done Done Sent false
20670 6353 Acked Read 1 Done Done Sent false
20671 6354 Acked Read 2 Done Done Sent false
20672 6355 Acked Read 9 Done Done Sent false
20673 6356 Acked Flush 0 Done Done Sent false
20674 6357 Acked Read 4 Done Done Sent false
20675 6358 Acked Write 1 Done Done Sent false
20676 6359 Acked Read 6 Done Done Sent false
20677 6360 Acked Flush 0 Done Done Sent false
20678 6361 Acked Write 6 Done Done Sent false
20679 6362 Acked Write 7 Done Done Sent false
20680 6363 Acked Read 7 Done Done Sent false
20681 6364 Acked Read 5 Done Done Sent false
20682 6365 Acked Read 7 Done Done Sent false
20683 6366 Acked Flush 0 Done Done Sent false
20684 6367 Acked Flush 0 Done Done Sent false
20685 6368 Acked Write 2 Done Done Sent false
20686 6369 Acked Read 1 Done Done Sent false
20687 6370 Acked Flush 0 Done Done Sent false
20688 6371 Acked Read 10 Done Done Sent false
20689 6372 Acked Read 3 Done Done Sent false
20690 6373 Acked Write 2 Done Done Sent false
20691 6374 Acked Read 8 Done Done Sent false
20692 6375 Acked Read 6 Done Done Sent false
20693 6376 Acked Write 3 Done Done Sent false
20694 6377 Acked Read 4 Done Done Sent false
20695 6378 Acked Write 8 Done Done Sent false
20696 6379 Acked Read 10 Done Done Sent false
20697 6380 Acked Write 3 Done Done Sent false
20698 6381 Acked Write 8 Done Done Sent false
20699 6382 Acked Write 4 Done Done Sent false
20700 6383 Acked Write 3 Done Done Sent false
20701 6384 Acked Write 9 Done Done Sent false
20702 6385 Acked Read 1 Done Done Sent false
20703 6386 Acked Write 6 Done Done Sent false
20704 6387 Acked Read 7 Done Done Sent false
20705 6388 Acked Write 5 Done Done Sent false
20706 6389 Acked Read 3 Done Done Sent false
20707 6390 Acked Read 2 Done Done Sent false
20708 6391 Acked Flush 0 Done Done Sent false
20709 6392 Acked Write 9 Done Done Sent false
20710 6393 Acked Read 4 Done Done Sent false
20711 6394 Acked Flush 0 Done Done Sent false
20712 6395 Acked Read 8 Done Done Sent false
20713 6396 Acked Read 3 Done Done Sent false
20714 6397 Acked Write 2 Done Done Sent false
20715 6398 Acked Read 2 Done Done Sent false
20716 6399 Acked Read 2 Done Done Sent false
20717 6400 Acked Read 7 Done Done Sent false
20718 6401 Acked Read 2 Done Done Sent false
20719 6402 Acked Read 2 Done Done Sent false
20720 6403 Acked Write 10 Done Done Sent false
20721 6404 Acked Write 2 Done Done Sent false
20722 6405 Acked Read 3 Done Done Sent false
20723 6406 Acked Write 3 Done Done Sent false
20724 6407 Acked Read 7 Done Done Sent false
20725 6408 Acked Write 2 Done Done Sent false
20726 6409 Acked Write 10 Done Done Sent false
20727 6410 Acked Read 9 Done Done Sent false
20728 6411 Acked Read 3 Done Done Sent false
20729 6412 Acked Write 6 Done Done Sent false
20730 6413 Acked Flush 0 Done Done Sent false
20731 6414 Acked Write 8 Done Done Sent false
20732 6415 Acked Write 1 Done Done Sent false
20733 6416 Acked Read 6 Done Done Sent false
20734 6417 Acked Write 10 Done Done Sent false
20735 6418 Acked Write 7 Done Done Sent false
20736 6419 Acked Write 8 Done Done Sent false
20737 6420 Acked Read 4 Done Done Sent false
20738 6421 Acked Flush 0 Done Done Sent false
20739 6422 Acked Write 2 Done Done Sent false
20740 6423 Acked Read 7 Done Done Sent false
20741 6424 Acked Write 9 Done Done Sent false
20742 6425 Acked Write 4 Done Done Sent false
20743 6426 Acked Read 7 Done Done Sent false
20744 6427 Acked Write 1 Done Done Sent false
20745 6428 Acked Read 10 Done Done Sent false
20746 6429 Acked Write 1 Done Done Sent false
20747 6430 Acked Read 2 Done Done Sent false
20748 6431 Acked Write 5 Done Done Sent false
20749 6432 Acked Write 9 Done Done Sent false
20750 6433 Acked Flush 0 Done Done Sent false
20751 STATES DS:0 DS:1 DS:2 TOTAL
20752 Sent 0 0 301 301
20753 Done 301 301 0 602
20754 Skipped 0 0 0 0
20755 Error 0 0 0 0
20756 Last Flush: Some(JobId(6433)) Some(JobId(6433)) Some(JobId(6132))
20757 Downstairs last five completed: 6132 6131 6130 6129 6128
20758 Upstairs last five completed: 6433 6432 6431 6430 6429
20759 CLIENT: Up:0 ds:301 act:2
207602025-03-17T22:20:52.285ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
207612025-03-17T22:20:53.036ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
20762 [8830] got start action
20763 [8830] I am going to restart
20764 Starting downstairs at port 8830
20765 Make output file at "/tmp/dsc/downstairs-8830.txt"
20766 [8830][2] reports Starting
20767 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1224)
20768 [8830][2] reports Running
207692025-03-17T22:20:54.036ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
207702025-03-17T22:20:55.038ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
20771 ----------------------------------------------------------------
20772 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20773 States: Active Active Offline
20774 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
20775 6133 Acked Write 6 Done Done Sent false
20776 6134 Acked Read 2 Done Done Sent false
20777 6135 Acked Write 9 Done Done Sent false
20778 6136 Acked Read 8 Done Done Sent false
20779 6137 Acked Read 5 Done Done Sent false
20780 6138 Acked Write 5 Done Done Sent false
20781 6139 Acked Write 2 Done Done Sent false
20782 6140 Acked Write 6 Done Done Sent false
20783 6141 Acked Write 4 Done Done Sent false
20784 6142 Acked Write 9 Done Done Sent false
20785 6143 Acked Write 3 Done Done Sent false
20786 6144 Acked Read 7 Done Done Sent false
20787 6145 Acked Write 3 Done Done Sent false
20788 6146 Acked Read 9 Done Done Sent false
20789 6147 Acked Write 1 Done Done Sent false
20790 6148 Acked Read 6 Done Done Sent false
20791 6149 Acked Flush 0 Done Done Sent false
20792 6150 Acked Read 2 Done Done Sent false
20793 6151 Acked Read 7 Done Done Sent false
20794 6152 Acked Write 2 Done Done Sent false
20795 6153 Acked Write 3 Done Done Sent false
20796 6154 Acked Read 10 Done Done Sent false
20797 6155 Acked Read 5 Done Done Sent false
20798 6156 Acked Flush 0 Done Done Sent false
20799 6157 Acked Write 5 Done Done Sent false
20800 6158 Acked Read 1 Done Done Sent false
20801 6159 Acked Flush 0 Done Done Sent false
20802 6160 Acked Read 5 Done Done Sent false
20803 6161 Acked Write 2 Done Done Sent false
20804 6162 Acked Read 5 Done Done Sent false
20805 6163 Acked Write 8 Done Done Sent false
20806 6164 Acked Read 4 Done Done Sent false
20807 6165 Acked Write 7 Done Done Sent false
20808 6166 Acked Read 10 Done Done Sent false
20809 6167 Acked Read 1 Done Done Sent false
20810 6168 Acked Write 8 Done Done Sent false
20811 6169 Acked Read 3 Done Done Sent false
20812 6170 Acked Read 9 Done Done Sent false
20813 6171 Acked Read 8 Done Done Sent false
20814 6172 Acked Flush 0 Done Done Sent false
20815 6173 Acked Read 3 Done Done Sent false
20816 6174 Acked Write 7 Done Done Sent false
20817 6175 Acked Read 4 Done Done Sent false
20818 6176 Acked Read 5 Done Done Sent false
20819 6177 Acked Write 2 Done Done Sent false
20820 6178 Acked Write 7 Done Done Sent false
20821 6179 Acked Write 1 Done Done Sent false
20822 6180 Acked Flush 0 Done Done Sent false
20823 6181 Acked Write 8 Done Done Sent false
20824 6182 Acked Write 4 Done Done Sent false
20825 6183 Acked Flush 0 Done Done Sent false
20826 6184 Acked Write 3 Done Done Sent false
20827 6185 Acked Read 6 Done Done Sent false
20828 6186 Acked Read 5 Done Done Sent false
20829 6187 Acked Read 10 Done Done Sent false
20830 6188 Acked Read 2 Done Done Sent false
20831 6189 Acked Read 2 Done Done Sent false
20832 6190 Acked Read 9 Done Done Sent false
20833 6191 Acked Read 8 Done Done Sent false
20834 6192 Acked Read 4 Done Done Sent false
20835 6193 Acked Write 9 Done Done Sent false
20836 6194 Acked Read 9 Done Done Sent false
20837 6195 Acked Flush 0 Done Done Sent false
20838 6196 Acked Read 2 Done Done Sent false
20839 6197 Acked Read 7 Done Done Sent false
20840 6198 Acked Flush 0 Done Done Sent false
20841 6199 Acked Read 5 Done Done Sent false
20842 6200 Acked Read 3 Done Done Sent false
20843 6201 Acked Write 8 Done Done Sent false
20844 6202 Acked Write 8 Done Done Sent false
20845 6203 Acked Write 1 Done Done Sent false
20846 6204 Acked Write 10 Done Done Sent false
20847 6205 Acked Write 6 Done Done Sent false
20848 6206 Acked Write 8 Done Done Sent false
20849 6207 Acked Write 9 Done Done Sent false
20850 6208 Acked Write 10 Done Done Sent false
20851 6209 Acked Read 1 Done Done Sent false
20852 6210 Acked Read 9 Done Done Sent false
20853 6211 Acked Write 4 Done Done Sent false
20854 6212 Acked Read 6 Done Done Sent false
20855 6213 Acked Write 7 Done Done Sent false
20856 6214 Acked Read 3 Done Done Sent false
20857 6215 Acked Read 5 Done Done Sent false
20858 6216 Acked Read 3 Done Done Sent false
20859 6217 Acked Read 10 Done Done Sent false
20860 6218 Acked Flush 0 Done Done Sent false
20861 6219 Acked Flush 0 Done Done Sent false
20862 6220 Acked Read 7 Done Done Sent false
20863 6221 Acked Write 8 Done Done Sent false
20864 6222 Acked Write 8 Done Done Sent false
20865 6223 Acked Read 10 Done Done Sent false
20866 6224 Acked Read 10 Done Done Sent false
20867 6225 Acked Read 4 Done Done Sent false
20868 6226 Acked Read 8 Done Done Sent false
20869 6227 Acked Read 2 Done Done Sent false
20870 6228 Acked Write 5 Done Done Sent false
20871 6229 Acked Flush 0 Done Done Sent false
20872 6230 Acked Read 8 Done Done Sent false
20873 6231 Acked Read 9 Done Done Sent false
20874 6232 Acked Write 1 Done Done Sent false
20875 6233 Acked Read 3 Done Done Sent false
20876 6234 Acked Flush 0 Done Done Sent false
20877 6235 Acked Write 5 Done Done Sent false
20878 6236 Acked Read 1 Done Done Sent false
20879 6237 Acked Read 5 Done Done Sent false
20880 6238 Acked Write 10 Done Done Sent false
20881 6239 Acked Flush 0 Done Done Sent false
20882 6240 Acked Write 1 Done Done Sent false
20883 6241 Acked Read 4 Done Done Sent false
20884 6242 Acked Read 7 Done Done Sent false
20885 6243 Acked Write 7 Done Done Sent false
20886 6244 Acked Write 5 Done Done Sent false
20887 6245 Acked Write 7 Done Done Sent false
20888 6246 Acked Read 4 Done Done Sent false
20889 6247 Acked Read 4 Done Done Sent false
20890 6248 Acked Write 5 Done Done Sent false
20891 6249 Acked Read 9 Done Done Sent false
20892 6250 Acked Read 7 Done Done Sent false
20893 6251 Acked Write 10 Done Done Sent false
20894 6252 Acked Read 3 Done Done Sent false
20895 6253 Acked Read 7 Done Done Sent false
20896 6254 Acked Read 9 Done Done Sent false
20897 6255 Acked Read 2 Done Done Sent false
20898 6256 Acked Read 8 Done Done Sent false
20899 6257 Acked Write 6 Done Done Sent false
20900 6258 Acked Write 9 Done Done Sent false
20901 6259 Acked Write 5 Done Done Sent false
20902 6260 Acked Write 5 Done Done Sent false
20903 6261 Acked Write 10 Done Done Sent false
20904 6262 Acked Write 9 Done Done Sent false
20905 6263 Acked Read 2 Done Done Sent false
20906 6264 Acked Write 1 Done Done Sent false
20907 6265 Acked Write 1 Done Done Sent false
20908 6266 Acked Read 10 Done Done Sent false
20909 6267 Acked Write 1 Done Done Sent false
20910 6268 Acked Read 5 Done Done Sent false
20911 6269 Acked Write 5 Done Done Sent false
20912 6270 Acked Write 3 Done Done Sent false
20913 6271 Acked Flush 0 Done Done Sent false
20914 6272 Acked Read 5 Done Done Sent false
20915 6273 Acked Write 9 Done Done Sent false
20916 6274 Acked Read 7 Done Done Sent false
20917 6275 Acked Write 10 Done Done Sent false
20918 6276 Acked Write 7 Done Done Sent false
20919 6277 Acked Read 1 Done Done Sent false
20920 6278 Acked Write 10 Done Done Sent false
20921 6279 Acked Write 5 Done Done Sent false
20922 6280 Acked Flush 0 Done Done Sent false
20923 6281 Acked Flush 0 Done Done Sent false
20924 6282 Acked Write 7 Done Done Sent false
20925 6283 Acked Write 6 Done Done Sent false
20926 6284 Acked Flush 0 Done Done Sent false
20927 6285 Acked Write 7 Done Done Sent false
20928 6286 Acked Write 4 Done Done Sent false
20929 6287 Acked Read 3 Done Done Sent false
20930 6288 Acked Write 10 Done Done Sent false
20931 6289 Acked Write 2 Done Done Sent false
20932 6290 Acked Flush 0 Done Done Sent false
20933 6291 Acked Flush 0 Done Done Sent false
20934 6292 Acked Write 9 Done Done Sent false
20935 6293 Acked Write 5 Done Done Sent false
20936 6294 Acked Write 8 Done Done Sent false
20937 6295 Acked Read 2 Done Done Sent false
20938 6296 Acked Read 3 Done Done Sent false
20939 6297 Acked Read 7 Done Done Sent false
20940 6298 Acked Read 9 Done Done Sent false
20941 6299 Acked Write 3 Done Done Sent false
20942 6300 Acked Write 1 Done Done Sent false
20943 6301 Acked Write 3 Done Done Sent false
20944 6302 Acked Read 10 Done Done Sent false
20945 6303 Acked Flush 0 Done Done Sent false
20946 6304 Acked Write 4 Done Done Sent false
20947 6305 Acked Write 4 Done Done Sent false
20948 6306 Acked Write 9 Done Done Sent false
20949 6307 Acked Write 4 Done Done Sent false
20950 6308 Acked Write 1 Done Done Sent false
20951 6309 Acked Read 4 Done Done Sent false
20952 6310 Acked Read 3 Done Done Sent false
20953 6311 Acked Flush 0 Done Done Sent false
20954 6312 Acked Read 4 Done Done Sent false
20955 6313 Acked Write 5 Done Done Sent false
20956 6314 Acked Write 4 Done Done Sent false
20957 6315 Acked Read 4 Done Done Sent false
20958 6316 Acked Flush 0 Done Done Sent false
20959 6317 Acked Write 4 Done Done Sent false
20960 6318 Acked Write 4 Done Done Sent false
20961 6319 Acked Write 1 Done Done Sent false
20962 6320 Acked Read 7 Done Done Sent false
20963 6321 Acked Read 6 Done Done Sent false
20964 6322 Acked Read 10 Done Done Sent false
20965 6323 Acked Read 2 Done Done Sent false
20966 6324 Acked Write 4 Done Done Sent false
20967 6325 Acked Flush 0 Done Done Sent false
20968 6326 Acked Read 2 Done Done Sent false
20969 6327 Acked Write 1 Done Done Sent false
20970 6328 Acked Write 9 Done Done Sent false
20971 6329 Acked Read 3 Done Done Sent false
20972 6330 Acked Write 4 Done Done Sent false
20973 6331 Acked Write 4 Done Done Sent false
20974 6332 Acked Flush 0 Done Done Sent false
20975 6333 Acked Write 3 Done Done Sent false
20976 6334 Acked Flush 0 Done Done Sent false
20977 6335 Acked Read 10 Done Done Sent false
20978 6336 Acked Write 10 Done Done Sent false
20979 6337 Acked Read 3 Done Done Sent false
20980 6338 Acked Flush 0 Done Done Sent false
20981 6339 Acked Write 10 Done Done Sent false
20982 6340 Acked Write 10 Done Done Sent false
20983 6341 Acked Read 10 Done Done Sent false
20984 6342 Acked Write 8 Done Done Sent false
20985 6343 Acked Write 2 Done Done Sent false
20986 6344 Acked Read 1 Done Done Sent false
20987 6345 Acked Read 3 Done Done Sent false
20988 6346 Acked Read 2 Done Done Sent false
20989 6347 Acked Read 10 Done Done Sent false
20990 6348 Acked Write 5 Done Done Sent false
20991 6349 Acked Read 2 Done Done Sent false
20992 6350 Acked Write 6 Done Done Sent false
20993 6351 Acked Write 1 Done Done Sent false
20994 6352 Acked Read 5 Done Done Sent false
20995 6353 Acked Read 1 Done Done Sent false
20996 6354 Acked Read 2 Done Done Sent false
20997 6355 Acked Read 9 Done Done Sent false
20998 6356 Acked Flush 0 Done Done Sent false
20999 6357 Acked Read 4 Done Done Sent false
21000 6358 Acked Write 1 Done Done Sent false
21001 6359 Acked Read 6 Done Done Sent false
21002 6360 Acked Flush 0 Done Done Sent false
21003 6361 Acked Write 6 Done Done Sent false
21004 6362 Acked Write 7 Done Done Sent false
21005 6363 Acked Read 7 Done Done Sent false
21006 6364 Acked Read 5 Done Done Sent false
21007 6365 Acked Read 7 Done Done Sent false
21008 6366 Acked Flush 0 Done Done Sent false
21009 6367 Acked Flush 0 Done Done Sent false
21010 6368 Acked Write 2 Done Done Sent false
21011 6369 Acked Read 1 Done Done Sent false
21012 6370 Acked Flush 0 Done Done Sent false
21013 6371 Acked Read 10 Done Done Sent false
21014 6372 Acked Read 3 Done Done Sent false
21015 6373 Acked Write 2 Done Done Sent false
21016 6374 Acked Read 8 Done Done Sent false
21017 6375 Acked Read 6 Done Done Sent false
21018 6376 Acked Write 3 Done Done Sent false
21019 6377 Acked Read 4 Done Done Sent false
21020 6378 Acked Write 8 Done Done Sent false
21021 6379 Acked Read 10 Done Done Sent false
21022 6380 Acked Write 3 Done Done Sent false
21023 6381 Acked Write 8 Done Done Sent false
21024 6382 Acked Write 4 Done Done Sent false
21025 6383 Acked Write 3 Done Done Sent false
21026 6384 Acked Write 9 Done Done Sent false
21027 6385 Acked Read 1 Done Done Sent false
21028 6386 Acked Write 6 Done Done Sent false
21029 6387 Acked Read 7 Done Done Sent false
21030 6388 Acked Write 5 Done Done Sent false
21031 6389 Acked Read 3 Done Done Sent false
21032 6390 Acked Read 2 Done Done Sent false
21033 6391 Acked Flush 0 Done Done Sent false
21034 6392 Acked Write 9 Done Done Sent false
21035 6393 Acked Read 4 Done Done Sent false
21036 6394 Acked Flush 0 Done Done Sent false
21037 6395 Acked Read 8 Done Done Sent false
21038 6396 Acked Read 3 Done Done Sent false
21039 6397 Acked Write 2 Done Done Sent false
21040 6398 Acked Read 2 Done Done Sent false
21041 6399 Acked Read 2 Done Done Sent false
21042 6400 Acked Read 7 Done Done Sent false
21043 6401 Acked Read 2 Done Done Sent false
21044 6402 Acked Read 2 Done Done Sent false
21045 6403 Acked Write 10 Done Done Sent false
21046 6404 Acked Write 2 Done Done Sent false
21047 6405 Acked Read 3 Done Done Sent false
21048 6406 Acked Write 3 Done Done Sent false
21049 6407 Acked Read 7 Done Done Sent false
21050 6408 Acked Write 2 Done Done Sent false
21051 6409 Acked Write 10 Done Done Sent false
21052 6410 Acked Read 9 Done Done Sent false
21053 6411 Acked Read 3 Done Done Sent false
21054 6412 Acked Write 6 Done Done Sent false
21055 6413 Acked Flush 0 Done Done Sent false
21056 6414 Acked Write 8 Done Done Sent false
21057 6415 Acked Write 1 Done Done Sent false
21058 6416 Acked Read 6 Done Done Sent false
21059 6417 Acked Write 10 Done Done Sent false
21060 6418 Acked Write 7 Done Done Sent false
21061 6419 Acked Write 8 Done Done Sent false
21062 6420 Acked Read 4 Done Done Sent false
21063 6421 Acked Flush 0 Done Done Sent false
21064 6422 Acked Write 2 Done Done Sent false
21065 6423 Acked Read 7 Done Done Sent false
21066 6424 Acked Write 9 Done Done Sent false
21067 6425 Acked Write 4 Done Done Sent false
21068 6426 Acked Read 7 Done Done Sent false
21069 6427 Acked Write 1 Done Done Sent false
21070 6428 Acked Read 10 Done Done Sent false
21071 6429 Acked Write 1 Done Done Sent false
21072 6430 Acked Read 2 Done Done Sent false
21073 6431 Acked Write 5 Done Done Sent false
21074 6432 Acked Write 9 Done Done Sent false
21075 6433 Acked Flush 0 Done Done Sent false
21076 STATES DS:0 DS:1 DS:2 TOTAL
21077 Sent 0 0 301 301
21078 Done 301 301 0 602
21079 Skipped 0 0 0 0
21080 Error 0 0 0 0
21081 Last Flush: Some(JobId(6433)) Some(JobId(6433)) Some(JobId(6132))
21082 Downstairs last five completed: 6132 6131 6130 6129 6128
21083 Upstairs last five completed: 6433 6432 6431 6430 6429
21084 CLIENT: Up:0 ds:301 act:2
210852025-03-17T22:20:56.287ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
210862025-03-17T22:20:57.042ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
210872025-03-17T22:20:57.546ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
210882025-03-17T22:20:57.547ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:61954) = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
210892025-03-17T22:20:57.548ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
210902025-03-17T22:20:57.548ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
210912025-03-17T22:20:57.548ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
210922025-03-17T22:20:57.549ZINFOcrucible: send last flush ID to this DS: Some(JobId(6132)) = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
210932025-03-17T22:20:57.549ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
210942025-03-17T22:20:57.549ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(6132)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
210952025-03-17T22:20:57.549ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(6132)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
21096 ----------------------------------------------------------------
21097 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
21098 Downstairs last five completed: 6433 6432 6431 6430 6429
21099 Upstairs last five completed: 6433 6432 6431 6430 6429
21100 CLIENT: Up:0 ds:0 act:3
21101 Replay: All jobs finished
21102 Main task has work to do, go find it
21103 got dsc Stop(2)
21104 stop 2
21105 [8830] Got stop action so:false kr:false
21106 Mar 17 22:20:59.788 INFO request completed, latency_us: 343, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 7b5ad622-058e-43b5-a11d-5af643b4427d, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
211072025-03-17T22:20:59.788ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
211082025-03-17T22:20:59.789ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
211092025-03-17T22:20:59.789ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
21110 Mar 17 22:20:59.789 INFO request completed, latency_us: 150, response_code: 200, uri: /state/cid/2, method: GET, req_id: a0d6d10a-1b8e-4c2f-8f57-fcfdf3de98b8, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
211112025-03-17T22:20:59.789ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
21112 [8830] Exited with: ExitStatus(unix_wait_status(9))
21113 [8830][2] reports Exit
21114 Mar 17 22:21:03.791 INFO request completed, latency_us: 368, response_code: 200, uri: /state/cid/2, method: GET, req_id: a43e8ccc-2237-4636-9a02-262d2eed42da, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
21115 001/300 Read block 2832 len 36864
21116 002/300 Write block 4981 len 4096 data: 3
21117 003/300 Read block 1849 len 40960
21118 004/300 Write block 4453 len 36864 data: 5 6 5 4 2 2 2 2 4
21119 005/300 Read block 3424 len 28672
21120 006/300 Write block 2903 len 8192 data: 3 4
21121 007/300 Read block 899 len 8192
21122 008/300 Write block 1983 len 20480 data: 2 3 5 7 6
21123 009/300 Write block 4405 len 36864 data: 7 7 6 6 7 5 3 2 2
21124 010/300 Flush
21125 011/300 Write block 3544 len 32768 data: 4 3 3 3 5 4 4 4
21126 012/300 Write block 2784 len 36864 data: 5 5 5 4 4 4 4 5 5
21127 013/300 Write block 2824 len 20480 data: 8 8 9 10 7
21128 014/300 Write block 2107 len 32768 data: 8 7 6 6 5 6 5 5
21129 015/300 Read block 227 len 4096
21130 016/300 Read block 861 len 16384
21131 017/300 Write block 2603 len 32768 data: 4 6 5 7 8 7 8 8
21132 018/300 Write block 1293 len 8192 data: 3 2
21133 019/300 Read block 2131 len 16384
21134 020/300 Flush
21135 021/300 Write block 3507 len 16384 data: 3 3 3 4
21136 022/300 Write block 2203 len 24576 data: 7 5 4 6 7 6
21137 023/300 Read block 2618 len 16384
21138 024/300 Read block 1722 len 40960
21139 025/300 Write block 4811 len 20480 data: 8 6 5 4 5
21140 026/300 Read block 4144 len 4096
21141 027/300 Write block 791 len 16384 data: 3 3 3 3
21142 028/300 Flush
21143 029/300 Read block 4421 len 4096
21144 030/300 Read block 2923 len 8192
21145 031/300 Write block 3375 len 16384 data: 3 4 5 6
21146 032/300 Read block 499 len 40960
21147 033/300 Write block 1537 len 8192 data: 3 3
21148 034/300 Flush
21149 035/300 Write block 3981 len 12288 data: 3 3 3
21150 036/300 Read block 2017 len 4096
21151 037/300 Read block 1596 len 32768
21152 038/300 Read block 4518 len 28672
21153 039/300 Read block 2191 len 20480
21154 040/300 Read block 4455 len 28672
21155 041/300 Flush
21156 042/300 Write block 3914 len 24576 data: 4 4 5 6 6 5
21157 043/300 Read block 1446 len 24576
21158 044/300 Read block 1874 len 24576
21159 045/300 Read block 2228 len 8192
21160 046/300 Read block 3363 len 12288
21161 047/300 Write block 4842 len 40960 data: 2 2 2 2 2 2 4 4 4 5
21162 048/300 Write block 3920 len 12288 data: 4 4 5
21163 049/300 Write block 3927 len 8192 data: 3 4
21164 050/300 Read block 1331 len 24576
21165 051/300 Flush
21166 052/300 Write block 2335 len 20480 data: 5 5 4 4 4
21167 053/300 Write block 1111 len 16384 data: 2 3 4 4
21168 054/300 Read block 3458 len 12288
21169 055/300 Write block 3630 len 16384 data: 2 3 3 3
21170 056/300 Read block 1055 len 12288
21171 057/300 Write block 1870 len 40960 data: 5 6 4 4 5 4 4 4 3 3
21172 058/300 Write block 1527 len 24576 data: 11 10 9 9 9 7
21173 059/300 Flush
21174 060/300 Read block 3017 len 16384
21175 061/300 Write block 3363 len 32768 data: 2 3 3 3 3 3 3 3
21176 062/300 Read block 3655 len 24576
21177 063/300 Read block 752 len 32768
21178 064/300 Flush
21179 065/300 Read block 1533 len 20480
21180 066/300 Flush
21181 067/300 Read block 2893 len 32768
21182 068/300 Read block 707 len 16384
21183 069/300 Read block 4411 len 28672
21184 070/300 Read block 1494 len 28672
21185 071/300 Write block 3941 len 40960 data: 4 3 3 3 5 4 4 4 4 4
21186 072/300 Read block 461 len 40960
21187 073/300 Write block 3039 len 24576 data: 6 4 3 3 3 4
21188 074/300 Write block 1252 len 28672 data: 4 5 6 6 6 6 5
21189 075/300 Read block 4386 len 4096
21190 076/300 Read block 758 len 40960
21191 077/300 Write block 1556 len 12288 data: 3 3 5
21192 078/300 Write block 1989 len 4096 data: 8
21193 079/300 Read block 1813 len 32768
21194 080/300 Read block 3303 len 12288
21195 081/300 Flush
21196 082/300 Write block 1703 len 32768 data: 3 4 4 4 4 2 3 3
21197 083/300 Read block 344 len 20480
21198 084/300 Read block 2892 len 20480
21199 085/300 Write block 2724 len 16384 data: 2 2 2 2
21200 086/300 Read block 4992 len 28672
21201 087/300 Flush
21202 088/300 Read block 4121 len 36864
21203 089/300 Read block 1679 len 32768
21204 090/300 Write block 4328 len 28672 data: 4 3 5 5 5 4 3
21205 091/300 Read block 4983 len 8192
21206 092/300 Write block 1679 len 12288 data: 4 6 6
21207 093/300 Read block 872 len 4096
21208 094/300 Read block 4366 len 32768
21209 095/300 Read block 4614 len 20480
21210 096/300 Read block 3439 len 16384
21211 097/300 Read block 3052 len 12288
21212 098/300 Flush
21213 099/300 Write block 4454 len 24576 data: 7 6 5 3 3 3
21214 100/300 Read block 4937 len 28672
21215 101/300 Write block 1925 len 28672 data: 6 7 6 7 4 4 5
21216 102/300 Write block 2708 len 8192 data: 3 2
21217 103/300 Write block 4971 len 12288 data: 2 3 3
21218 104/300 Read block 1021 len 4096
21219 105/300 Read block 425 len 12288
21220 106/300 Read block 1191 len 8192
21221 107/300 Write block 3638 len 20480 data: 5 4 4 6 7
21222 108/300 Write block 1504 len 4096 data: 5
21223 109/300 Read block 3503 len 36864
21224 110/300 Read block 1526 len 36864
21225 111/300 Read block 1202 len 20480
21226 112/300 Write block 3758 len 20480 data: 2 2 3 3 3
21227 113/300 Write block 4003 len 8192 data: 7 6
21228 114/300 Write block 2896 len 12288 data: 4 5 5
21229 115/300 Write block 4337 len 12288 data: 6 6 5
21230 116/300 Write block 4937 len 32768 data: 2 4 4 4 3 3 4 3
21231 117/300 Write block 2816 len 12288 data: 3 3 4
21232 118/300 Read block 1606 len 24576
21233 119/300 Read block 1718 len 24576
21234 120/300 Write block 1744 len 16384 data: 3 4 4 6
21235 121/300 Read block 2020 len 8192
21236 122/300 Read block 4146 len 4096
21237 123/300 Read block 2053 len 20480
21238 124/300 Write block 4822 len 28672 data: 6 5 5 4 4 4 4
21239 125/300 Read block 3048 len 32768
21240 126/300 Flush
21241 127/300 Write block 39 len 20480 data: 3 3 3 3 4
21242 128/300 Read block 1481 len 32768
21243 129/300 Write block 4822 len 20480 data: 7 6 6 5 5
21244 130/300 Write block 1092 len 32768 data: 4 4 5 5 5 5 6 6
21245 131/300 Read block 1391 len 4096
21246 132/300 Read block 1662 len 12288
21247 133/300 Read block 3460 len 40960
21248 134/300 Write block 4282 len 36864 data: 4 4 3 4 4 2 3 3 3
21249 135/300 Write block 2977 len 32768 data: 4 4 3 3 3 3 3 3
21250 136/300 Read block 1625 len 40960
21251 137/300 Read block 1129 len 12288
21252 138/300 Write block 2577 len 40960 data: 3 2 2 2 2 2 2 2 2 3
21253 139/300 Write block 4724 len 20480 data: 8 6 5 4 4
21254 140/300 Write block 4909 len 20480 data: 3 3 3 3 3
21255 141/300 Read block 1861 len 24576
21256 142/300 Read block 2413 len 8192
21257 143/300 Write block 2915 len 36864 data: 5 5 5 5 5 5 5 5 3
21258 144/300 Write block 3741 len 24576 data: 5 5 3 3 5 5
21259 145/300 Read block 17 len 20480
21260 146/300 Read block 3202 len 24576
21261 147/300 Read block 2934 len 32768
21262 148/300 Read block 37 len 32768
21263 149/300 Read block 319 len 32768
21264 150/300 Write block 3966 len 32768 data: 2 2 3 2 3 3 3 3
21265 151/300 Read block 3581 len 28672
21266 152/300 Write block 3123 len 12288 data: 4 4 4
21267 153/300 Read block 527 len 32768
21268 154/300 Read block 2992 len 16384
21269 155/300 Flush
21270 156/300 Read block 4048 len 16384
21271 157/300 Write block 1222 len 28672 data: 7 6 6 5 5 4 5
21272 158/300 Write block 1385 len 32768 data: 3 3 3 3 4 5 5 6
21273 159/300 Write block 3248 len 4096 data: 6
21274 160/300 Read block 4199 len 20480
21275 161/300 Write block 1262 len 32768 data: 3 3 3 3 3 2 2 2
21276 162/300 Read block 1111 len 24576
21277 163/300 Read block 2923 len 32768
21278 164/300 Read block 4649 len 40960
21279 165/300 Write block 888 len 24576 data: 5 6 6 5 5 5
21280 166/300 Flush
21281 167/300 Read block 2496 len 40960
21282 168/300 Read block 3958 len 20480
21283 169/300 Write block 4843 len 32768 data: 3 3 3 3 3 5 5 5
21284 170/300 Write block 2764 len 36864 data: 3 3 3 5 6 6 6 6 3
21285 171/300 Write block 3063 len 4096 data: 4
21286 172/300 Read block 1056 len 32768
21287 173/300 Read block 1192 len 28672
21288 174/300 Read block 2732 len 20480
21289 175/300 Write block 669 len 8192 data: 4 4
21290 176/300 Write block 4679 len 20480 data: 3 3 4 4 3
21291 177/300 Write block 4887 len 8192 data: 7 5
21292 178/300 Read block 3040 len 12288
21293 179/300 Flush
21294 180/300 Read block 78 len 28672
21295 181/300 Write block 2829 len 4096 data: 7
21296 182/300 Flush
21297 183/300 Write block 1642 len 24576 data: 7 5 5 6 6 6
21298 184/300 Read block 420 len 28672
21299 185/300 Read block 2023 len 24576
21300 186/300 Flush
21301 187/300 Flush
21302 188/300 Write block 2098 len 28672 data: 3 3 2 2 2 3 3
21303 189/300 Read block 4194 len 40960
21304 190/300 Read block 4243 len 12288
21305 191/300 Flush
21306 192/300 Read block 3683 len 20480
21307 193/300 Read block 4862 len 20480
21308 194/300 Write block 3438 len 12288 data: 5 5 5
21309 195/300 Read block 2467 len 4096
21310 196/300 Read block 3892 len 40960
21311 197/300 Write block 2063 len 36864 data: 6 6 5 6 6 6 7 5 6
21312 198/300 Read block 1220 len 12288
21313 199/300 Read block 562 len 32768
21314 200/300 Write block 368 len 32768 data: 3 3 3 3 5 6 6 5
21315 201/300 Read block 190 len 24576
21316 202/300 Read block 2066 len 32768
21317 203/300 Write block 4625 len 40960 data: 4 4 3 4 4 4 4 3 5 5
21318 204/300 Read block 3744 len 8192
21319 205/300 Read block 1560 len 12288
21320 206/300 Write block 3642 len 36864 data: 8 7 7 5 3 3 3 4 5
21321 207/300 Write block 4866 len 28672 data: 7 7 5 4 5 4 4
21322 208/300 Write block 1774 len 4096 data: 3
21323 209/300 Write block 4045 len 4096 data: 3
21324 210/300 Write block 1274 len 12288 data: 5 6 6
21325 211/300 Write block 1940 len 36864 data: 4 4 4 4 4 4 3 3 3
21326 212/300 Write block 4265 len 28672 data: 2 4 3 3 4 4 5
21327 213/300 Write block 3804 len 4096 data: 6
21328 214/300 Flush
21329 215/300 Flush
21330 216/300 Flush
21331 217/300 Read block 1881 len 4096
21332 218/300 Read block 3167 len 16384
21333 219/300 Read block 913 len 32768
21334 220/300 Write block 2462 len 24576 data: 2 3 3 3 2 2
21335 221/300 Write block 1792 len 20480 data: 3 3 4 5 6
21336 222/300 Read block 3894 len 28672
21337 223/300 Read block 4534 len 4096
21338 224/300 Write block 4339 len 32768 data: 6 5 3 5 4 4 3 3
21339 225/300 Read block 3905 len 4096
21340 226/300 Write block 869 len 36864 data: 2 2 3 3 4 4 4 3 3
21341 227/300 Read block 4215 len 24576
21342 228/300 Read block 472 len 32768
21343 229/300 Read block 4520 len 12288
21344 230/300 Read block 4574 len 28672
21345 231/300 Read block 3274 len 32768
21346 232/300 Flush
21347 233/300 Write block 2179 len 40960 data: 7 6 7 6 6 5 5 5 4 2
21348 234/300 Write block 2155 len 4096 data: 3
21349 235/300 Read block 1873 len 4096
21350 236/300 Write block 1573 len 28672 data: 5 4 4 4 4 4 4
21351 237/300 Read block 1115 len 36864
21352 238/300 Write block 1545 len 12288 data: 3 3 3
21353 239/300 Read block 4726 len 28672
21354 240/300 Write block 745 len 4096 data: 3
21355 241/300 Flush
21356 242/300 Read block 892 len 8192
21357 243/300 Read block 1903 len 32768
21358 244/300 Read block 4239 len 16384
21359 245/300 Read block 4932 len 40960
21360 246/300 Read block 465 len 24576
21361 247/300 Write block 3268 len 12288 data: 4 6 6
21362 248/300 Read block 127 len 8192
21363 249/300 Read block 4351 len 20480
21364 250/300 Write block 1569 len 28672 data: 5 6 5 5 6 5 5
21365 251/300 Write block 4482 len 32768 data: 2 2 3 4 5 4 4 4
21366 252/300 Read block 4530 len 12288
21367 253/300 Read block 2576 len 36864
21368 254/300 Flush
21369 255/300 Flush
21370 256/300 Write block 2262 len 20480 data: 2 2 2 3 3
21371 257/300 Read block 1327 len 28672
21372 258/300 Flush
21373 259/300 Read block 906 len 36864
21374 260/300 Write block 1470 len 16384 data: 4 3 4 5
21375 261/300 Read block 4192 len 24576
21376 262/300 Write block 3530 len 40960 data: 4 4 4 5 5 6 6 6 4 3
21377 263/300 Read block 4906 len 4096
21378 264/300 Read block 4744 len 16384
21379 265/300 Read block 1888 len 16384
21380 266/300 Write block 4915 len 4096 data: 4
21381 267/300 Read block 2958 len 32768
21382 268/300 Flush
21383 269/300 Flush
21384 270/300 Write block 2665 len 16384 data: 5 6 6 5
21385 271/300 Read block 4516 len 24576
21386 272/300 Write block 1624 len 16384 data: 3 2 2 2
21387 273/300 Read block 1131 len 12288
21388 274/300 Write block 2039 len 32768 data: 6 6 7 7 6 6 5 6
21389 275/300 Write block 1775 len 4096 data: 3
21390 276/300 Write block 2491 len 40960 data: 4 4 3 4 6 5 5 5 5 4
21391 277/300 Write block 3882 len 24576 data: 5 5 4 4 4 6
21392 278/300 Write block 2791 len 24576 data: 6 6 5 3 2 4
21393 279/300 Read block 3709 len 24576
21394 280/300 Read block 4178 len 8192
21395 281/300 Write block 892 len 24576 data: 6 6 4 3 4 3
21396 282/300 Flush
21397 283/300 Read block 4822 len 24576
21398 284/300 Flush
21399 285/300 Read block 4519 len 8192
21400 286/300 Write block 3345 len 12288 data: 2 3 4
21401 287/300 Write block 2834 len 28672 data: 6 7 6 6 7 6 6
21402 288/300 Read block 1736 len 32768
21403 289/300 Read block 1585 len 16384
21404 290/300 Write block 2877 len 4096 data: 4
21405 291/300 Read block 2939 len 20480
21406 292/300 Write block 4148 len 20480 data: 7 8 6 5 4
21407 293/300 Write block 4058 len 40960 data: 7 6 6 6 6 6 6 4 4 5
21408 294/300 Write block 4329 len 28672 data: 4 6 6 6 5 4 4
21409 295/300 Write block 1011 len 32768 data: 4 4 3 4 4 4 4 4
21410 296/300 Read block 2165 len 32768
21411 297/300 Read block 3173 len 24576
21412 298/300 Write block 1213 len 20480 data: 5 5 6 6 7
21413 299/300 Write block 4615 len 36864 data: 6 5 5 3 3 2 3 3 3
21414 300/300 Read block 3268 len 40960
21415 301/300 Read block 306 len 12288
21416 Main task has work to do, go find it
21417 got dsc Start(2)
21418 start 2
21419 Mar 17 22:21:04.040 INFO request completed, latency_us: 225, response_code: 204, uri: /start/cid/2, method: POST, req_id: 5212774b-6f20-40db-918f-b6254f25f894, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
21420 Replay: started 2, returned:Ok(())
21421 ----------------------------------------------------------------
21422 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21423 States: Active Active Offline
21424 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21425 6434 Acked Read 9 Done Done Sent false
21426 6435 Acked Write 1 Done Done Sent false
21427 6436 Acked Read 10 Done Done Sent false
21428 6437 Acked Write 9 Done Done Sent false
21429 6438 Acked Read 7 Done Done Sent false
21430 6439 Acked Write 2 Done Done Sent false
21431 6440 Acked Read 2 Done Done Sent false
21432 6441 Acked Write 5 Done Done Sent false
21433 6442 Acked Write 9 Done Done Sent false
21434 6443 Acked Flush 0 Done Done Sent false
21435 6444 Acked Write 8 Done Done Sent false
21436 6445 Acked Write 9 Done Done Sent false
21437 6446 Acked Write 5 Done Done Sent false
21438 6447 Acked Write 8 Done Done Sent false
21439 6448 Acked Read 1 Done Done Sent false
21440 6449 Acked Read 4 Done Done Sent false
21441 6450 Acked Write 8 Done Done Sent false
21442 6451 Acked Write 2 Done Done Sent false
21443 6452 Acked Read 4 Done Done Sent false
21444 6453 Acked Flush 0 Done Done Sent false
21445 6454 Acked Write 4 Done Done Sent false
21446 6455 Acked Write 6 Done Done Sent false
21447 6456 Acked Read 4 Done Done Sent false
21448 6457 Acked Read 10 Done Done Sent false
21449 6458 Acked Write 5 Done Done Sent false
21450 6459 Acked Read 1 Done Done Sent false
21451 6460 Acked Write 4 Done Done Sent false
21452 6461 Acked Flush 0 Done Done Sent false
21453 6462 Acked Read 1 Done Done Sent false
21454 6463 Acked Read 2 Done Done Sent false
21455 6464 Acked Write 4 Done Done Sent false
21456 6465 Acked Read 10 Done Done Sent false
21457 6466 Acked Write 2 Done Done Sent false
21458 6467 Acked Flush 0 Done Done Sent false
21459 6468 Acked Write 3 Done Done Sent false
21460 6469 Acked Read 1 Done Done Sent false
21461 6470 Acked Read 8 Done Done Sent false
21462 6471 Acked Read 7 Done Done Sent false
21463 6472 Acked Read 5 Done Done Sent false
21464 6473 Acked Read 7 Done Done Sent false
21465 6474 Acked Flush 0 Done Done Sent false
21466 6475 Acked Write 6 Done Done Sent false
21467 6476 Acked Read 6 Done Done Sent false
21468 6477 Acked Read 6 Done Done Sent false
21469 6478 Acked Read 2 Done Done Sent false
21470 6479 Acked Read 3 Done Done Sent false
21471 6480 Acked Write 10 Done Done Sent false
21472 6481 Acked Write 3 Done Done Sent false
21473 6482 Acked Write 2 Done Done Sent false
21474 6483 Acked Read 6 Done Done Sent false
21475 6484 Acked Flush 0 Done Done Sent false
21476 6485 Acked Write 5 Done Done Sent false
21477 6486 Acked Write 4 Done Done Sent false
21478 6487 Acked Read 3 Done Done Sent false
21479 6488 Acked Write 4 Done Done Sent false
21480 6489 Acked Read 3 Done Done Sent false
21481 6490 Acked Write 10 Done Done Sent false
21482 6491 Acked Write 6 Done Done Sent false
21483 6492 Acked Flush 0 Done Done Sent false
21484 6493 Acked Read 4 Done Done Sent false
21485 6494 Acked Write 8 Done Done Sent false
21486 6495 Acked Read 6 Done Done Sent false
21487 6496 Acked Read 8 Done Done Sent false
21488 6497 Acked Flush 0 Done Done Sent false
21489 6498 Acked Read 5 Done Done Sent false
21490 6499 Acked Flush 0 Done Done Sent false
21491 6500 Acked Read 8 Done Done Sent false
21492 6501 Acked Read 4 Done Done Sent false
21493 6502 Acked Read 7 Done Done Sent false
21494 6503 Acked Read 7 Done Done Sent false
21495 6504 Acked Write 10 Done Done Sent false
21496 6505 Acked Read 10 Done Done Sent false
21497 6506 Acked Write 6 Done Done Sent false
21498 6507 Acked Write 7 Done Done Sent false
21499 6508 Acked Read 1 Done Done Sent false
21500 6509 Acked Read 10 Done Done Sent false
21501 6510 Acked Write 3 Done Done Sent false
21502 6511 Acked Write 1 Done Done Sent false
21503 6512 Acked Read 8 Done Done Sent false
21504 6513 Acked Read 3 Done Done Sent false
21505 6514 Acked Flush 0 Done Done Sent false
21506 6515 Acked Write 8 Done Done Sent false
21507 6516 Acked Read 5 Done Done Sent false
21508 6517 Acked Read 5 Done Done Sent false
21509 6518 Acked Write 4 Done Done Sent false
21510 6519 Acked Read 7 Done Done Sent false
21511 6520 Acked Flush 0 Done Done Sent false
21512 6521 Acked Read 9 Done Done Sent false
21513 6522 Acked Read 8 Done Done Sent false
21514 6523 Acked Write 7 Done Done Sent false
21515 6524 Acked Read 2 Done Done Sent false
21516 6525 Acked Write 3 Done Done Sent false
21517 6526 Acked Read 1 Done Done Sent false
21518 6527 Acked Read 8 Done Done Sent false
21519 6528 Acked Read 5 Done Done Sent false
21520 6529 Acked Read 4 Done Done Sent false
21521 6530 Acked Read 3 Done Done Sent false
21522 6531 Acked Flush 0 Done Done Sent false
21523 6532 Acked Write 6 Done Done Sent false
21524 6533 Acked Read 7 Done Done Sent false
21525 6534 Acked Write 7 Done Done Sent false
21526 6535 Acked Write 2 Done Done Sent false
21527 6536 Acked Write 3 Done Done Sent false
21528 6537 Acked Read 1 Done Done Sent false
21529 6538 Acked Read 3 Done Done Sent false
21530 6539 Acked Read 2 Done Done Sent false
21531 6540 Acked Write 5 Done Done Sent false
21532 6541 Acked Write 1 Done Done Sent false
21533 6542 Acked Read 9 Done Done Sent false
21534 6543 Acked Read 9 Done Done Sent false
21535 6544 Acked Read 5 Done Done Sent false
21536 6545 Acked Write 5 Done Done Sent false
21537 6546 Acked Write 2 Done Done Sent false
21538 6547 Acked Write 3 Done Done Sent false
21539 6548 Acked Write 3 Done Done Sent false
21540 6549 Acked Write 8 Done Done Sent false
21541 6550 Acked Write 3 Done Done Sent false
21542 6551 Acked Read 6 Done Done Sent false
21543 6552 Acked Read 6 Done Done Sent false
21544 6553 Acked Write 4 Done Done Sent false
21545 6554 Acked Read 2 Done Done Sent false
21546 6555 Acked Read 1 Done Done Sent false
21547 6556 Acked Read 5 Done Done Sent false
21548 6557 Acked Write 7 Done Done Sent false
21549 6558 Acked Read 8 Done Done Sent false
21550 6559 Acked Flush 0 Done Done Sent false
21551 6560 Acked Write 5 Done Done Sent false
21552 6561 Acked Read 8 Done Done Sent false
21553 6562 Acked Write 5 Done Done Sent false
21554 6563 Acked Write 8 Done Done Sent false
21555 6564 Acked Read 1 Done Done Sent false
21556 6565 Acked Read 3 Done Done Sent false
21557 6566 Acked Read 10 Done Done Sent false
21558 6567 Acked Write 9 Done Done Sent false
21559 6568 Acked Write 8 Done Done Sent false
21560 6569 Acked Read 10 Done Done Sent false
21561 6570 Acked Read 3 Done Done Sent false
21562 6571 Acked Write 10 Done Done Sent false
21563 6572 Acked Write 5 Done Done Sent false
21564 6573 Acked Write 5 Done Done Sent false
21565 6574 Acked Read 6 Done Done Sent false
21566 6575 Acked Read 2 Done Done Sent false
21567 6576 Acked Write 9 Done Done Sent false
21568 6577 Acked Write 6 Done Done Sent false
21569 6578 Acked Read 5 Done Done Sent false
21570 6579 Acked Read 6 Done Done Sent false
21571 6580 Acked Read 8 Done Done Sent false
21572 6581 Acked Read 8 Done Done Sent false
21573 6582 Acked Read 8 Done Done Sent false
21574 6583 Acked Write 8 Done Done Sent false
21575 6584 Acked Read 7 Done Done Sent false
21576 6585 Acked Write 3 Done Done Sent false
21577 6586 Acked Read 8 Done Done Sent false
21578 6587 Acked Read 4 Done Done Sent false
21579 6588 Acked Flush 0 Done Done Sent false
21580 6589 Acked Read 4 Done Done Sent false
21581 6590 Acked Write 7 Done Done Sent false
21582 6591 Acked Write 8 Done Done Sent false
21583 6592 Acked Write 1 Done Done Sent false
21584 6593 Acked Read 5 Done Done Sent false
21585 6594 Acked Write 8 Done Done Sent false
21586 6595 Acked Read 6 Done Done Sent false
21587 6596 Acked Read 8 Done Done Sent false
21588 6597 Acked Read 10 Done Done Sent false
21589 6598 Acked Write 6 Done Done Sent false
21590 6599 Acked Flush 0 Done Done Sent false
21591 6600 Acked Read 10 Done Done Sent false
21592 6601 Acked Read 5 Done Done Sent false
21593 6602 Acked Write 8 Done Done Sent false
21594 6603 Acked Write 9 Done Done Sent false
21595 6604 Acked Write 1 Done Done Sent false
21596 6605 Acked Read 8 Done Done Sent false
21597 6606 Acked Read 7 Done Done Sent false
21598 6607 Acked Read 5 Done Done Sent false
21599 6608 Acked Write 2 Done Done Sent false
21600 6609 Acked Write 5 Done Done Sent false
21601 6610 Acked Write 2 Done Done Sent false
21602 6611 Acked Read 3 Done Done Sent false
21603 6612 Acked Flush 0 Done Done Sent false
21604 6613 Acked Read 7 Done Done Sent false
21605 6614 Acked Write 1 Done Done Sent false
21606 6615 Acked Flush 0 Done Done Sent false
21607 6616 Acked Write 6 Done Done Sent false
21608 6617 Acked Read 7 Done Done Sent false
21609 6618 Acked Read 6 Done Done Sent false
21610 6619 Acked Flush 0 Done Done Sent false
21611 6620 Acked Flush 0 Done Done Sent false
21612 6621 Acked Write 7 Done Done Sent false
21613 6622 Acked Read 10 Done Done Sent false
21614 6623 Acked Read 3 Done Done Sent false
21615 6624 Acked Flush 0 Done Done Sent false
21616 6625 Acked Read 5 Done Done Sent false
21617 6626 Acked Read 5 Done Done Sent false
21618 6627 Acked Write 3 Done Done Sent false
21619 6628 Acked Read 1 Done Done Sent false
21620 6629 Acked Read 10 Done Done Sent false
21621 6630 Acked Write 9 Done Done Sent false
21622 6631 Acked Read 3 Done Done Sent false
21623 6632 Acked Read 8 Done Done Sent false
21624 6633 Acked Write 8 Done Done Sent false
21625 6634 Acked Read 6 Done Done Sent false
21626 6635 Acked Read 8 Done Done Sent false
21627 6636 Acked Write 10 Done Done Sent false
21628 6637 Acked Read 2 Done Done Sent false
21629 6638 Acked Read 3 Done Done Sent false
21630 6639 Acked Write 9 Done Done Sent false
21631 6640 Acked Write 7 Done Done Sent false
21632 6641 Acked Write 1 Done Done Sent false
21633 6642 Acked Write 1 Done Done Sent false
21634 6643 Acked Write 3 Done Done Sent false
21635 6644 Acked Write 9 Done Done Sent false
21636 6645 Acked Write 7 Done Done Sent false
21637 6646 Acked Write 1 Done Done Sent false
21638 6647 Acked Flush 0 Done Done Sent false
21639 6648 Acked Flush 0 Done Done Sent false
21640 6649 Acked Flush 0 Done Done Sent false
21641 6650 Acked Read 1 Done Done Sent false
21642 6651 Acked Read 4 Done Done Sent false
21643 6652 Acked Read 8 Done Done Sent false
21644 6653 Acked Write 6 Done Done Sent false
21645 6654 Acked Write 5 Done Done Sent false
21646 6655 Acked Read 7 Done Done Sent false
21647 6656 Acked Read 1 Done Done Sent false
21648 6657 Acked Write 8 Done Done Sent false
21649 6658 Acked Read 1 Done Done Sent false
21650 6659 Acked Write 9 Done Done Sent false
21651 6660 Acked Read 6 Done Done Sent false
21652 6661 Acked Read 8 Done Done Sent false
21653 6662 Acked Read 3 Done Done Sent false
21654 6663 Acked Read 7 Done Done Sent false
21655 6664 Acked Read 8 Done Done Sent false
21656 6665 Acked Flush 0 Done Done Sent false
21657 6666 Acked Write 10 Done Done Sent false
21658 6667 Acked Write 1 Done Done Sent false
21659 6668 Acked Read 1 Done Done Sent false
21660 6669 Acked Write 7 Done Done Sent false
21661 6670 Acked Read 9 Done Done Sent false
21662 6671 Acked Write 3 Done Done Sent false
21663 6672 Acked Read 7 Done Done Sent false
21664 6673 Acked Write 1 Done Done Sent false
21665 6674 Acked Flush 0 Done Done Sent false
21666 6675 Acked Read 2 Done Done Sent false
21667 6676 Acked Read 8 Done Done Sent false
21668 6677 Acked Read 4 Done Done Sent false
21669 6678 Acked Read 10 Done Done Sent false
21670 6679 Acked Read 6 Done Done Sent false
21671 6680 Acked Write 3 Done Done Sent false
21672 6681 Acked Read 2 Done Done Sent false
21673 6682 Acked Read 5 Done Done Sent false
21674 6683 Acked Write 7 Done Done Sent false
21675 6684 Acked Write 8 Done Done Sent false
21676 6685 Acked Read 3 Done Done Sent false
21677 6686 Acked Read 9 Done Done Sent false
21678 6687 Acked Flush 0 Done Done Sent false
21679 6688 Acked Flush 0 Done Done Sent false
21680 6689 Acked Write 5 Done Done Sent false
21681 6690 Acked Read 7 Done Done Sent false
21682 6691 Acked Flush 0 Done Done Sent false
21683 6692 Acked Read 9 Done Done Sent false
21684 6693 Acked Write 4 Done Done Sent false
21685 6694 Acked Read 6 Done Done Sent false
21686 6695 Acked Write 10 Done Done Sent false
21687 6696 Acked Read 1 Done Done Sent false
21688 6697 Acked Read 4 Done Done Sent false
21689 6698 Acked Read 4 Done Done Sent false
21690 6699 Acked Write 1 Done Done Sent false
21691 6700 Acked Read 8 Done Done Sent false
21692 6701 Acked Flush 0 Done Done Sent false
21693 6702 Acked Flush 0 Done Done Sent false
21694 6703 Acked Write 4 Done Done Sent false
21695 6704 Acked Read 6 Done Done Sent false
21696 6705 Acked Write 4 Done Done Sent false
21697 6706 Acked Read 3 Done Done Sent false
21698 6707 Acked Write 8 Done Done Sent false
21699 6708 Acked Write 1 Done Done Sent false
21700 6709 Acked Write 10 Done Done Sent false
21701 6710 Acked Write 6 Done Done Sent false
21702 6711 Acked Write 6 Done Done Sent false
21703 6712 Acked Read 6 Done Done Sent false
21704 6713 Acked Read 2 Done Done Sent false
21705 6714 Acked Write 6 Done Done Sent false
21706 6715 Acked Flush 0 Done Done Sent false
21707 6716 Acked Read 6 Done Done Sent false
21708 6717 Acked Flush 0 Done Done Sent false
21709 6718 Acked Read 2 Done Done Sent false
21710 6719 Acked Write 3 Done Done Sent false
21711 6720 Acked Write 7 Done Done Sent false
21712 6721 Acked Read 8 Done Done Sent false
21713 6722 Acked Read 4 Done Done Sent false
21714 6723 Acked Write 1 Done Done Sent false
21715 6724 Acked Read 5 Done Done Sent false
21716 6725 Acked Write 5 Done Done Sent false
21717 6726 Acked Write 10 Done Done Sent false
21718 6727 Acked Write 7 Done Done Sent false
21719 6728 Acked Write 8 Done Done Sent false
21720 6729 Acked Read 8 Done Done Sent false
21721 6730 Acked Read 6 Done Done Sent false
21722 6731 Acked Write 5 Done Done Sent false
21723 6732 Acked Write 9 Done Done Sent false
21724 6733 Acked Read 10 Done Done Sent false
21725 6734 Acked Read 3 Done Done Sent false
21726 STATES DS:0 DS:1 DS:2 TOTAL
21727 Sent 0 0 301 301
21728 Done 301 301 0 602
21729 Skipped 0 0 0 0
21730 Error 0 0 0 0
21731 Last Flush: Some(JobId(6717)) Some(JobId(6717)) Some(JobId(6433))
21732 Downstairs last five completed: 6433 6432 6431 6430 6429
21733 Upstairs last five completed: 6734 6733 6732 6731 6730
21734 CLIENT: Up:0 ds:301 act:2
217352025-03-17T22:21:05.054ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
21736 [8830] got start action
21737 [8830] I am going to restart
21738 Starting downstairs at port 8830
21739 Make output file at "/tmp/dsc/downstairs-8830.txt"
21740 [8830][2] reports Starting
21741 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1225)
21742 [8830][2] reports Running
217432025-03-17T22:21:06.054ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
217442025-03-17T22:21:07.056ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
21745 ----------------------------------------------------------------
21746 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21747 States: Active Active Offline
21748 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
21749 6434 Acked Read 9 Done Done Sent false
21750 6435 Acked Write 1 Done Done Sent false
21751 6436 Acked Read 10 Done Done Sent false
21752 6437 Acked Write 9 Done Done Sent false
21753 6438 Acked Read 7 Done Done Sent false
21754 6439 Acked Write 2 Done Done Sent false
21755 6440 Acked Read 2 Done Done Sent false
21756 6441 Acked Write 5 Done Done Sent false
21757 6442 Acked Write 9 Done Done Sent false
21758 6443 Acked Flush 0 Done Done Sent false
21759 6444 Acked Write 8 Done Done Sent false
21760 6445 Acked Write 9 Done Done Sent false
21761 6446 Acked Write 5 Done Done Sent false
21762 6447 Acked Write 8 Done Done Sent false
21763 6448 Acked Read 1 Done Done Sent false
21764 6449 Acked Read 4 Done Done Sent false
21765 6450 Acked Write 8 Done Done Sent false
21766 6451 Acked Write 2 Done Done Sent false
21767 6452 Acked Read 4 Done Done Sent false
21768 6453 Acked Flush 0 Done Done Sent false
21769 6454 Acked Write 4 Done Done Sent false
21770 6455 Acked Write 6 Done Done Sent false
21771 6456 Acked Read 4 Done Done Sent false
21772 6457 Acked Read 10 Done Done Sent false
21773 6458 Acked Write 5 Done Done Sent false
21774 6459 Acked Read 1 Done Done Sent false
21775 6460 Acked Write 4 Done Done Sent false
21776 6461 Acked Flush 0 Done Done Sent false
21777 6462 Acked Read 1 Done Done Sent false
21778 6463 Acked Read 2 Done Done Sent false
21779 6464 Acked Write 4 Done Done Sent false
21780 6465 Acked Read 10 Done Done Sent false
21781 6466 Acked Write 2 Done Done Sent false
21782 6467 Acked Flush 0 Done Done Sent false
21783 6468 Acked Write 3 Done Done Sent false
21784 6469 Acked Read 1 Done Done Sent false
21785 6470 Acked Read 8 Done Done Sent false
21786 6471 Acked Read 7 Done Done Sent false
21787 6472 Acked Read 5 Done Done Sent false
21788 6473 Acked Read 7 Done Done Sent false
21789 6474 Acked Flush 0 Done Done Sent false
21790 6475 Acked Write 6 Done Done Sent false
21791 6476 Acked Read 6 Done Done Sent false
21792 6477 Acked Read 6 Done Done Sent false
21793 6478 Acked Read 2 Done Done Sent false
21794 6479 Acked Read 3 Done Done Sent false
21795 6480 Acked Write 10 Done Done Sent false
21796 6481 Acked Write 3 Done Done Sent false
21797 6482 Acked Write 2 Done Done Sent false
21798 6483 Acked Read 6 Done Done Sent false
21799 6484 Acked Flush 0 Done Done Sent false
21800 6485 Acked Write 5 Done Done Sent false
21801 6486 Acked Write 4 Done Done Sent false
21802 6487 Acked Read 3 Done Done Sent false
21803 6488 Acked Write 4 Done Done Sent false
21804 6489 Acked Read 3 Done Done Sent false
21805 6490 Acked Write 10 Done Done Sent false
21806 6491 Acked Write 6 Done Done Sent false
21807 6492 Acked Flush 0 Done Done Sent false
21808 6493 Acked Read 4 Done Done Sent false
21809 6494 Acked Write 8 Done Done Sent false
21810 6495 Acked Read 6 Done Done Sent false
21811 6496 Acked Read 8 Done Done Sent false
21812 6497 Acked Flush 0 Done Done Sent false
21813 6498 Acked Read 5 Done Done Sent false
21814 6499 Acked Flush 0 Done Done Sent false
21815 6500 Acked Read 8 Done Done Sent false
21816 6501 Acked Read 4 Done Done Sent false
21817 6502 Acked Read 7 Done Done Sent false
21818 6503 Acked Read 7 Done Done Sent false
21819 6504 Acked Write 10 Done Done Sent false
21820 6505 Acked Read 10 Done Done Sent false
21821 6506 Acked Write 6 Done Done Sent false
21822 6507 Acked Write 7 Done Done Sent false
21823 6508 Acked Read 1 Done Done Sent false
21824 6509 Acked Read 10 Done Done Sent false
21825 6510 Acked Write 3 Done Done Sent false
21826 6511 Acked Write 1 Done Done Sent false
21827 6512 Acked Read 8 Done Done Sent false
21828 6513 Acked Read 3 Done Done Sent false
21829 6514 Acked Flush 0 Done Done Sent false
21830 6515 Acked Write 8 Done Done Sent false
21831 6516 Acked Read 5 Done Done Sent false
21832 6517 Acked Read 5 Done Done Sent false
21833 6518 Acked Write 4 Done Done Sent false
21834 6519 Acked Read 7 Done Done Sent false
21835 6520 Acked Flush 0 Done Done Sent false
21836 6521 Acked Read 9 Done Done Sent false
21837 6522 Acked Read 8 Done Done Sent false
21838 6523 Acked Write 7 Done Done Sent false
21839 6524 Acked Read 2 Done Done Sent false
21840 6525 Acked Write 3 Done Done Sent false
21841 6526 Acked Read 1 Done Done Sent false
21842 6527 Acked Read 8 Done Done Sent false
21843 6528 Acked Read 5 Done Done Sent false
21844 6529 Acked Read 4 Done Done Sent false
21845 6530 Acked Read 3 Done Done Sent false
21846 6531 Acked Flush 0 Done Done Sent false
21847 6532 Acked Write 6 Done Done Sent false
21848 6533 Acked Read 7 Done Done Sent false
21849 6534 Acked Write 7 Done Done Sent false
21850 6535 Acked Write 2 Done Done Sent false
21851 6536 Acked Write 3 Done Done Sent false
21852 6537 Acked Read 1 Done Done Sent false
21853 6538 Acked Read 3 Done Done Sent false
21854 6539 Acked Read 2 Done Done Sent false
21855 6540 Acked Write 5 Done Done Sent false
21856 6541 Acked Write 1 Done Done Sent false
21857 6542 Acked Read 9 Done Done Sent false
21858 6543 Acked Read 9 Done Done Sent false
21859 6544 Acked Read 5 Done Done Sent false
21860 6545 Acked Write 5 Done Done Sent false
21861 6546 Acked Write 2 Done Done Sent false
21862 6547 Acked Write 3 Done Done Sent false
21863 6548 Acked Write 3 Done Done Sent false
21864 6549 Acked Write 8 Done Done Sent false
21865 6550 Acked Write 3 Done Done Sent false
21866 6551 Acked Read 6 Done Done Sent false
21867 6552 Acked Read 6 Done Done Sent false
21868 6553 Acked Write 4 Done Done Sent false
21869 6554 Acked Read 2 Done Done Sent false
21870 6555 Acked Read 1 Done Done Sent false
21871 6556 Acked Read 5 Done Done Sent false
21872 6557 Acked Write 7 Done Done Sent false
21873 6558 Acked Read 8 Done Done Sent false
21874 6559 Acked Flush 0 Done Done Sent false
21875 6560 Acked Write 5 Done Done Sent false
21876 6561 Acked Read 8 Done Done Sent false
21877 6562 Acked Write 5 Done Done Sent false
21878 6563 Acked Write 8 Done Done Sent false
21879 6564 Acked Read 1 Done Done Sent false
21880 6565 Acked Read 3 Done Done Sent false
21881 6566 Acked Read 10 Done Done Sent false
21882 6567 Acked Write 9 Done Done Sent false
21883 6568 Acked Write 8 Done Done Sent false
21884 6569 Acked Read 10 Done Done Sent false
21885 6570 Acked Read 3 Done Done Sent false
21886 6571 Acked Write 10 Done Done Sent false
21887 6572 Acked Write 5 Done Done Sent false
21888 6573 Acked Write 5 Done Done Sent false
21889 6574 Acked Read 6 Done Done Sent false
21890 6575 Acked Read 2 Done Done Sent false
21891 6576 Acked Write 9 Done Done Sent false
21892 6577 Acked Write 6 Done Done Sent false
21893 6578 Acked Read 5 Done Done Sent false
21894 6579 Acked Read 6 Done Done Sent false
21895 6580 Acked Read 8 Done Done Sent false
21896 6581 Acked Read 8 Done Done Sent false
21897 6582 Acked Read 8 Done Done Sent false
21898 6583 Acked Write 8 Done Done Sent false
21899 6584 Acked Read 7 Done Done Sent false
21900 6585 Acked Write 3 Done Done Sent false
21901 6586 Acked Read 8 Done Done Sent false
21902 6587 Acked Read 4 Done Done Sent false
21903 6588 Acked Flush 0 Done Done Sent false
21904 6589 Acked Read 4 Done Done Sent false
21905 6590 Acked Write 7 Done Done Sent false
21906 6591 Acked Write 8 Done Done Sent false
21907 6592 Acked Write 1 Done Done Sent false
21908 6593 Acked Read 5 Done Done Sent false
21909 6594 Acked Write 8 Done Done Sent false
21910 6595 Acked Read 6 Done Done Sent false
21911 6596 Acked Read 8 Done Done Sent false
21912 6597 Acked Read 10 Done Done Sent false
21913 6598 Acked Write 6 Done Done Sent false
21914 6599 Acked Flush 0 Done Done Sent false
21915 6600 Acked Read 10 Done Done Sent false
21916 6601 Acked Read 5 Done Done Sent false
21917 6602 Acked Write 8 Done Done Sent false
21918 6603 Acked Write 9 Done Done Sent false
21919 6604 Acked Write 1 Done Done Sent false
21920 6605 Acked Read 8 Done Done Sent false
21921 6606 Acked Read 7 Done Done Sent false
21922 6607 Acked Read 5 Done Done Sent false
21923 6608 Acked Write 2 Done Done Sent false
21924 6609 Acked Write 5 Done Done Sent false
21925 6610 Acked Write 2 Done Done Sent false
21926 6611 Acked Read 3 Done Done Sent false
21927 6612 Acked Flush 0 Done Done Sent false
21928 6613 Acked Read 7 Done Done Sent false
21929 6614 Acked Write 1 Done Done Sent false
21930 6615 Acked Flush 0 Done Done Sent false
21931 6616 Acked Write 6 Done Done Sent false
21932 6617 Acked Read 7 Done Done Sent false
21933 6618 Acked Read 6 Done Done Sent false
21934 6619 Acked Flush 0 Done Done Sent false
21935 6620 Acked Flush 0 Done Done Sent false
21936 6621 Acked Write 7 Done Done Sent false
21937 6622 Acked Read 10 Done Done Sent false
21938 6623 Acked Read 3 Done Done Sent false
21939 6624 Acked Flush 0 Done Done Sent false
21940 6625 Acked Read 5 Done Done Sent false
21941 6626 Acked Read 5 Done Done Sent false
21942 6627 Acked Write 3 Done Done Sent false
21943 6628 Acked Read 1 Done Done Sent false
21944 6629 Acked Read 10 Done Done Sent false
21945 6630 Acked Write 9 Done Done Sent false
21946 6631 Acked Read 3 Done Done Sent false
21947 6632 Acked Read 8 Done Done Sent false
21948 6633 Acked Write 8 Done Done Sent false
21949 6634 Acked Read 6 Done Done Sent false
21950 6635 Acked Read 8 Done Done Sent false
21951 6636 Acked Write 10 Done Done Sent false
21952 6637 Acked Read 2 Done Done Sent false
21953 6638 Acked Read 3 Done Done Sent false
21954 6639 Acked Write 9 Done Done Sent false
21955 6640 Acked Write 7 Done Done Sent false
21956 6641 Acked Write 1 Done Done Sent false
21957 6642 Acked Write 1 Done Done Sent false
21958 6643 Acked Write 3 Done Done Sent false
21959 6644 Acked Write 9 Done Done Sent false
21960 6645 Acked Write 7 Done Done Sent false
21961 6646 Acked Write 1 Done Done Sent false
21962 6647 Acked Flush 0 Done Done Sent false
21963 6648 Acked Flush 0 Done Done Sent false
21964 6649 Acked Flush 0 Done Done Sent false
21965 6650 Acked Read 1 Done Done Sent false
21966 6651 Acked Read 4 Done Done Sent false
21967 6652 Acked Read 8 Done Done Sent false
21968 6653 Acked Write 6 Done Done Sent false
21969 6654 Acked Write 5 Done Done Sent false
21970 6655 Acked Read 7 Done Done Sent false
21971 6656 Acked Read 1 Done Done Sent false
21972 6657 Acked Write 8 Done Done Sent false
21973 6658 Acked Read 1 Done Done Sent false
21974 6659 Acked Write 9 Done Done Sent false
21975 6660 Acked Read 6 Done Done Sent false
21976 6661 Acked Read 8 Done Done Sent false
21977 6662 Acked Read 3 Done Done Sent false
21978 6663 Acked Read 7 Done Done Sent false
21979 6664 Acked Read 8 Done Done Sent false
21980 6665 Acked Flush 0 Done Done Sent false
21981 6666 Acked Write 10 Done Done Sent false
21982 6667 Acked Write 1 Done Done Sent false
21983 6668 Acked Read 1 Done Done Sent false
21984 6669 Acked Write 7 Done Done Sent false
21985 6670 Acked Read 9 Done Done Sent false
21986 6671 Acked Write 3 Done Done Sent false
21987 6672 Acked Read 7 Done Done Sent false
21988 6673 Acked Write 1 Done Done Sent false
21989 6674 Acked Flush 0 Done Done Sent false
21990 6675 Acked Read 2 Done Done Sent false
21991 6676 Acked Read 8 Done Done Sent false
21992 6677 Acked Read 4 Done Done Sent false
21993 6678 Acked Read 10 Done Done Sent false
21994 6679 Acked Read 6 Done Done Sent false
21995 6680 Acked Write 3 Done Done Sent false
21996 6681 Acked Read 2 Done Done Sent false
21997 6682 Acked Read 5 Done Done Sent false
21998 6683 Acked Write 7 Done Done Sent false
21999 6684 Acked Write 8 Done Done Sent false
22000 6685 Acked Read 3 Done Done Sent false
22001 6686 Acked Read 9 Done Done Sent false
22002 6687 Acked Flush 0 Done Done Sent false
22003 6688 Acked Flush 0 Done Done Sent false
22004 6689 Acked Write 5 Done Done Sent false
22005 6690 Acked Read 7 Done Done Sent false
22006 6691 Acked Flush 0 Done Done Sent false
22007 6692 Acked Read 9 Done Done Sent false
22008 6693 Acked Write 4 Done Done Sent false
22009 6694 Acked Read 6 Done Done Sent false
22010 6695 Acked Write 10 Done Done Sent false
22011 6696 Acked Read 1 Done Done Sent false
22012 6697 Acked Read 4 Done Done Sent false
22013 6698 Acked Read 4 Done Done Sent false
22014 6699 Acked Write 1 Done Done Sent false
22015 6700 Acked Read 8 Done Done Sent false
22016 6701 Acked Flush 0 Done Done Sent false
22017 6702 Acked Flush 0 Done Done Sent false
22018 6703 Acked Write 4 Done Done Sent false
22019 6704 Acked Read 6 Done Done Sent false
22020 6705 Acked Write 4 Done Done Sent false
22021 6706 Acked Read 3 Done Done Sent false
22022 6707 Acked Write 8 Done Done Sent false
22023 6708 Acked Write 1 Done Done Sent false
22024 6709 Acked Write 10 Done Done Sent false
22025 6710 Acked Write 6 Done Done Sent false
22026 6711 Acked Write 6 Done Done Sent false
22027 6712 Acked Read 6 Done Done Sent false
22028 6713 Acked Read 2 Done Done Sent false
22029 6714 Acked Write 6 Done Done Sent false
22030 6715 Acked Flush 0 Done Done Sent false
22031 6716 Acked Read 6 Done Done Sent false
22032 6717 Acked Flush 0 Done Done Sent false
22033 6718 Acked Read 2 Done Done Sent false
22034 6719 Acked Write 3 Done Done Sent false
22035 6720 Acked Write 7 Done Done Sent false
22036 6721 Acked Read 8 Done Done Sent false
22037 6722 Acked Read 4 Done Done Sent false
22038 6723 Acked Write 1 Done Done Sent false
22039 6724 Acked Read 5 Done Done Sent false
22040 6725 Acked Write 5 Done Done Sent false
22041 6726 Acked Write 10 Done Done Sent false
22042 6727 Acked Write 7 Done Done Sent false
22043 6728 Acked Write 8 Done Done Sent false
22044 6729 Acked Read 8 Done Done Sent false
22045 6730 Acked Read 6 Done Done Sent false
22046 6731 Acked Write 5 Done Done Sent false
22047 6732 Acked Write 9 Done Done Sent false
22048 6733 Acked Read 10 Done Done Sent false
22049 6734 Acked Read 3 Done Done Sent false
22050 STATES DS:0 DS:1 DS:2 TOTAL
22051 Sent 0 0 301 301
22052 Done 301 301 0 602
22053 Skipped 0 0 0 0
22054 Error 0 0 0 0
22055 Last Flush: Some(JobId(6717)) Some(JobId(6717)) Some(JobId(6433))
22056 Downstairs last five completed: 6433 6432 6431 6430 6429
22057 Upstairs last five completed: 6734 6733 6732 6731 6730
22058 CLIENT: Up:0 ds:301 act:2
220592025-03-17T22:21:09.058ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
220602025-03-17T22:21:09.789ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
220612025-03-17T22:21:09.790ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:55535) = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
220622025-03-17T22:21:09.791ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
220632025-03-17T22:21:09.792ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
220642025-03-17T22:21:09.792ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
220652025-03-17T22:21:09.792ZINFOcrucible: send last flush ID to this DS: Some(JobId(6433)) = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
220662025-03-17T22:21:09.792ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
220672025-03-17T22:21:09.792ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(6433)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
220682025-03-17T22:21:09.792ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(6433)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
22069 ----------------------------------------------------------------
22070 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
22071 Downstairs last five completed: 6735 6734 6733 6732 6731
22072 Upstairs last five completed: 6735 6734 6733 6732 6731
22073 CLIENT: Up:0 ds:0 act:3
22074 Replay: All jobs finished
22075 Main task has work to do, go find it
22076 got dsc Stop(1)
22077 stop 1
22078 Mar 17 22:21:12.046 INFO request completed, latency_us: 364, response_code: 204, uri: /stop/cid/1, method: [8820] Got stop action so:false kr:false
22079 POST, req_id: df30ff84-87d3-4873-b88a-f2f965fccc87, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
220802025-03-17T22:21:12.047ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
220812025-03-17T22:21:12.047ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
220822025-03-17T22:21:12.047ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
22083 Mar 17 22:21:12.047 INFO request completed, latency_us: 132, response_code: 200, uri: /state/cid/1, method: GET, req_id: 83da5b8d-9086-4c7d-aaa3-b1283fb8cf7e, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
220842025-03-17T22:21:12.047ZINFOcrucible: client task is exiting = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
22085 [8820] Exited with: ExitStatus(unix_wait_status(9))
22086 [8820][1] reports Exit
22087 Mar 17 22:21:16.049 INFO request completed, latency_us: 260, response_code: 200, uri: /state/cid/1, method: GET, req_id: 924ba279-a5ca-47c3-bfc6-330a0b0847ad, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
22088 001/300 Read block 3583 len 32768
22089 002/300 Read block 3613 len 12288
22090 003/300 Write block 2886 len 40960 data: 7 7 5 5 4 3 4 4 4 4
22091 004/300 Write block 4728 len 28672 data: 5 5 6 5 6 5 5
22092 005/300 Read block 1531 len 4096
22093 006/300 Write block 4569 len 12288 data: 5 4 3
22094 007/300 Flush
22095 008/300 Read block 2152 len 40960
22096 009/300 Write block 4492 len 36864 data: 4 5 3 2 2 3 3 3 3
22097 010/300 Write block 1216 len 40960 data: 7 8 6 8 7 8 8 7 7 6
22098 011/300 Flush
22099 012/300 Read block 3149 len 12288
22100 013/300 Read block 144 len 28672
22101 014/300 Read block 4351 len 16384
22102 015/300 Read block 4149 len 16384
22103 016/300 Read block 4932 len 32768
22104 017/300 Write block 4683 len 24576 data: 4 5 4 4 4 5
22105 018/300 Read block 2234 len 12288
22106 019/300 Write block 1488 len 4096 data: 6
22107 020/300 Write block 1952 len 8192 data: 5 5
22108 021/300 Write block 4486 len 8192 data: 6 5
22109 022/300 Flush
22110 023/300 Read block 1326 len 40960
22111 024/300 Read block 3122 len 16384
22112 025/300 Write block 1575 len 8192 data: 6 5
22113 026/300 Flush
22114 027/300 Write block 4244 len 4096 data: 4
22115 028/300 Flush
22116 029/300 Read block 554 len 20480
22117 030/300 Write block 3009 len 16384 data: 3 4 4 4
22118 031/300 Read block 1564 len 36864
22119 032/300 Flush
22120 033/300 Read block 2443 len 32768
22121 034/300 Write block 3470 len 16384 data: 2 2 2 3
22122 035/300 Write block 1235 len 12288 data: 5 5 5
22123 036/300 Read block 2966 len 36864
22124 037/300 Read block 3513 len 4096
22125 038/300 Write block 1921 len 12288 data: 5 4 4
22126 039/300 Flush
22127 040/300 Read block 702 len 16384
22128 041/300 Read block 3543 len 8192
22129 042/300 Read block 3027 len 12288
22130 043/300 Write block 498 len 40960 data: 5 4 4 4 6 5 7 7 7 6
22131 044/300 Read block 4536 len 40960
22132 045/300 Read block 3908 len 20480
22133 046/300 Write block 115 len 12288 data: 5 5 6
22134 047/300 Write block 4251 len 40960 data: 5 5 5 5 5 4 5 6 6 5
22135 048/300 Read block 2499 len 4096
22136 049/300 Flush
22137 050/300 Write block 219 len 28672 data: 3 3 3 4 4 4 4
22138 051/300 Read block 4438 len 8192
22139 052/300 Write block 2622 len 40960 data: 3 4 4 4 4 4 5 4 4 3
22140 053/300 Read block 3395 len 32768
22141 054/300 Write block 3375 len 8192 data: 4 5
22142 055/300 Write block 620 len 40960 data: 4 4 4 2 4 5 5 5 6 6
22143 056/300 Write block 2704 len 4096 data: 7
22144 057/300 Write block 980 len 24576 data: 3 3 3 3 3 3
22145 058/300 Read block 1382 len 24576
22146 059/300 Write block 554 len 24576 data: 4 5 6 8 8 8
22147 060/300 Flush
22148 061/300 Read block 3723 len 28672
22149 062/300 Read block 4990 len 16384
22150 063/300 Write block 1451 len 16384 data: 3 3 3 3
22151 064/300 Write block 4239 len 36864 data: 5 4 5 5 5 5 5 4 5
22152 065/300 Read block 3951 len 24576
22153 066/300 Write block 2855 len 20480 data: 4 3 3 3 2
22154 067/300 Write block 211 len 24576 data: 3 3 3 3 3 3
22155 068/300 Write block 1370 len 4096 data: 4
22156 069/300 Read block 3325 len 8192
22157 070/300 Write block 2663 len 4096 data: 4
22158 071/300 Read block 4593 len 40960
22159 072/300 Write block 4865 len 12288 data: 9 8 8
22160 073/300 Write block 1162 len 16384 data: 7 6 5 7
22161 074/300 Flush
22162 075/300 Read block 1411 len 36864
22163 076/300 Read block 4743 len 24576
22164 077/300 Read block 2230 len 12288
22165 078/300 Read block 176 len 8192
22166 079/300 Read block 1597 len 28672
22167 080/300 Read block 3349 len 4096
22168 081/300 Read block 2494 len 12288
22169 082/300 Read block 1192 len 24576
22170 083/300 Write block 3269 len 40960 data: 7 7 5 5 5 5 6 6 3 4
22171 084/300 Write block 4186 len 32768 data: 4 4 3 3 3 3 3 2
22172 085/300 Write block 4145 len 28672 data: 5 6 7 8 9 7 6
22173 086/300 Read block 3250 len 28672
22174 087/300 Read block 4872 len 12288
22175 088/300 Read block 1159 len 32768
22176 089/300 Read block 1316 len 4096
22177 090/300 Write block 4073 len 8192 data: 3 3
22178 091/300 Read block 3336 len 28672
22179 092/300 Read block 142 len 8192
22180 093/300 Write block 2414 len 36864 data: 3 4 4 6 4 5 4 5 4
22181 094/300 Write block 3418 len 4096 data: 6
22182 095/300 Write block 2498 len 16384 data: 6 6 5 3
22183 096/300 Write block 314 len 16384 data: 5 5 7 7
22184 097/300 Read block 4431 len 8192
22185 098/300 Read block 2793 len 28672
22186 099/300 Read block 2859 len 28672
22187 100/300 Flush
22188 101/300 Write block 4155 len 12288 data: 3 4 4
22189 102/300 Read block 3581 len 20480
22190 103/300 Read block 4212 len 12288
22191 104/300 Flush
22192 105/300 Write block 3207 len 32768 data: 5 3 3 2 2 2 4 5
22193 106/300 Read block 2627 len 4096
22194 107/300 Write block 2028 len 36864 data: 2 2 2 2 3 3 4 4 5
22195 108/300 Read block 4636 len 36864
22196 109/300 Write block 44 len 4096 data: 5
22197 110/300 Flush
22198 111/300 Read block 2761 len 8192
22199 112/300 Write block 2775 len 4096 data: 4
22200 113/300 Flush
22201 114/300 Read block 2583 len 4096
22202 115/300 Write block 4337 len 16384 data: 7 7 7 6
22203 116/300 Read block 3090 len 28672
22204 117/300 Read block 1919 len 12288
22205 118/300 Write block 4367 len 32768 data: 7 7 6 6 7 7 5 4
22206 119/300 Write block 1481 len 40960 data: 7 6 7 7 8 7 6 7 7 6
22207 120/300 Write block 3118 len 40960 data: 4 5 4 4 4 5 5 5 3 5
22208 121/300 Read block 552 len 4096
22209 122/300 Write block 1037 len 24576 data: 4 4 7 8 9 5
22210 123/300 Write block 1890 len 36864 data: 4 3 2 3 3 4 4 4 4
22211 124/300 Read block 4449 len 8192
22212 125/300 Write block 1619 len 28672 data: 3 3 4 4 4 4 3
22213 126/300 Write block 2940 len 28672 data: 6 7 6 7 6 7 9
22214 127/300 Read block 3959 len 12288
22215 128/300 Flush
22216 129/300 Read block 1451 len 36864
22217 130/300 Write block 396 len 12288 data: 4 4 6
22218 131/300 Read block 399 len 32768
22219 132/300 Write block 3071 len 20480 data: 5 4 5 4 3
22220 133/300 Write block 2234 len 12288 data: 2 3 3
22221 134/300 Read block 259 len 36864
22222 135/300 Read block 1695 len 32768
22223 136/300 Write block 2897 len 16384 data: 6 6 4 6
22224 137/300 Read block 3682 len 32768
22225 138/300 Read block 2191 len 8192
22226 139/300 Flush
22227 140/300 Read block 2659 len 40960
22228 141/300 Read block 4374 len 40960
22229 142/300 Read block 1120 len 24576
22230 143/300 Write block 3061 len 32768 data: 5 5 5 4 5 5 5 5
22231 144/300 Read block 4100 len 36864
22232 145/300 Write block 4685 len 4096 data: 5
22233 146/300 Flush
22234 147/300 Flush
22235 148/300 Read block 3831 len 16384
22236 149/300 Flush
22237 150/300 Read block 445 len 4096
22238 151/300 Read block 1545 len 16384
22239 152/300 Read block 1046 len 8192
22240 153/300 Read block 3876 len 32768
22241 154/300 Read block 1012 len 8192
22242 155/300 Flush
22243 156/300 Read block 2937 len 28672
22244 157/300 Read block 2426 len 40960
22245 158/300 Write block 1818 len 40960 data: 3 3 4 3 3 3 4 2 4 4
22246 159/300 Read block 4001 len 20480
22247 160/300 Flush
22248 161/300 Flush
22249 162/300 Read block 3491 len 4096
22250 163/300 Write block 4073 len 32768 data: 4 4 3 3 4 2 4 4
22251 164/300 Read block 1142 len 40960
22252 165/300 Flush
22253 166/300 Write block 4878 len 28672 data: 6 5 4 4 4 7 8
22254 167/300 Read block 1058 len 32768
22255 168/300 Read block 3834 len 40960
22256 169/300 Write block 496 len 12288 data: 5 6 6
22257 170/300 Write block 1304 len 20480 data: 2 4 4 5 7
22258 171/300 Read block 967 len 36864
22259 172/300 Write block 1748 len 8192 data: 5 5
22260 173/300 Read block 84 len 16384
22261 174/300 Read block 2980 len 40960
22262 175/300 Write block 1074 len 32768 data: 6 8 8 8 6 7 6 5
22263 176/300 Write block 4702 len 8192 data: 4 4
22264 177/300 Write block 1651 len 20480 data: 2 2 3 3 3
22265 178/300 Write block 3402 len 28672 data: 5 7 6 6 6 4 4
22266 179/300 Write block 1929 len 4096 data: 5
22267 180/300 Read block 2445 len 20480
22268 181/300 Write block 3766 len 4096 data: 3
22269 182/300 Write block 1245 len 8192 data: 4 4
22270 183/300 Write block 1972 len 40960 data: 2 2 3 3 3 3 4 2 2 2
22271 184/300 Write block 1893 len 24576 data: 4 4 5 5 5 5
22272 185/300 Write block 4676 len 4096 data: 4
22273 186/300 Write block 4684 len 28672 data: 6 6 5 5 6 6 6
22274 187/300 Read block 1848 len 4096
22275 188/300 Read block 583 len 16384
22276 189/300 Flush
22277 190/300 Read block 3713 len 4096
22278 191/300 Read block 4562 len 16384
22279 192/300 Flush
22280 193/300 Write block 2017 len 28672 data: 5 3 3 4 4 4 4
22281 194/300 Write block 1160 len 40960 data: 4 5 8 7 6 8 8 8 8 6
22282 195/300 Write block 4214 len 20480 data: 4 3 3 4 4
22283 196/300 Write block 1497 len 32768 data: 3 3 2 3 3 4 5 6
22284 197/300 Read block 1438 len 16384
22285 198/300 Write block 3331 len 16384 data: 5 4 4 4
22286 199/300 Write block 4117 len 24576 data: 3 3 2 2 2 3
22287 200/300 Write block 2109 len 12288 data: 7 7 6
22288 201/300 Read block 146 len 20480
22289 202/300 Flush
22290 203/300 Read block 225 len 20480
22291 204/300 Write block 4027 len 12288 data: 2 2 2
22292 205/300 Write block 2485 len 28672 data: 5 5 5 5 5 5 5
22293 206/300 Read block 1493 len 4096
22294 207/300 Write block 2812 len 28672 data: 2 2 2 2 4 4 5
22295 208/300 Read block 653 len 20480
22296 209/300 Write block 1887 len 4096 data: 6
22297 210/300 Write block 4208 len 4096 data: 4
22298 211/300 Read block 3933 len 36864
22299 212/300 Write block 4446 len 24576 data: 4 4 5 6 6 6
22300 213/300 Flush
22301 214/300 Write block 2006 len 16384 data: 2 3 4 3
22302 215/300 Write block 1692 len 8192 data: 4 5
22303 216/300 Read block 2073 len 8192
22304 217/300 Flush
22305 218/300 Write block 508 len 36864 data: 5 5 4 2 2 2 2 2 2
22306 219/300 Write block 3953 len 20480 data: 5 4 3 3 4
22307 220/300 Write block 4810 len 28672 data: 8 9 7 6 5 6 5
22308 221/300 Flush
22309 222/300 Flush
22310 223/300 Read block 1012 len 32768
22311 224/300 Read block 2831 len 16384
22312 225/300 Flush
22313 226/300 Read block 4491 len 16384
22314 227/300 Read block 1211 len 28672
22315 228/300 Read block 4357 len 40960
22316 229/300 Read block 3564 len 8192
22317 230/300 Flush
22318 231/300 Read block 812 len 24576
22319 232/300 Read block 1753 len 4096
22320 233/300 Read block 3259 len 40960
22321 234/300 Read block 3903 len 20480
22322 235/300 Read block 3687 len 8192
22323 236/300 Read block 2520 len 16384
22324 237/300 Write block 3892 len 36864 data: 4 5 7 7 8 8 8 9 8
22325 238/300 Read block 2315 len 12288
22326 239/300 Write block 352 len 24576 data: 4 4 4 5 6 5
22327 240/300 Write block 3608 len 20480 data: 2 2 3 4 5
22328 241/300 Read block 4794 len 28672
22329 242/300 Flush
22330 243/300 Read block 351 len 8192
22331 244/300 Write block 4499 len 20480 data: 4 4 3 4 4
22332 245/300 Read block 4008 len 4096
22333 246/300 Write block 1642 len 16384 data: 8 6 6 7
22334 247/300 Read block 1278 len 28672
22335 248/300 Read block 1988 len 4096
22336 249/300 Read block 2273 len 4096
22337 250/300 Read block 1053 len 32768
22338 251/300 Write block 1747 len 4096 data: 7
22339 252/300 Write block 273 len 12288 data: 5 4 4
22340 253/300 Write block 817 len 12288 data: 2 2 2
22341 254/300 Read block 231 len 32768
22342 255/300 Write block 135 len 8192 data: 3 3
22343 256/300 Read block 248 len 12288
22344 257/300 Read block 4193 len 28672
22345 258/300 Write block 592 len 28672 data: 8 6 8 8 7 5 5
22346 259/300 Write block 3507 len 32768 data: 4 4 4 5 3 5 5 6
22347 260/300 Read block 1335 len 36864
22348 261/300 Write block 1977 len 24576 data: 4 5 3 3 3 2
22349 262/300 Write block 3697 len 32768 data: 2 2 4 3 5 5 5 5
22350 263/300 Read block 1947 len 24576
22351 264/300 Read block 2075 len 12288
22352 265/300 Read block 4215 len 16384
22353 266/300 Read block 3204 len 4096
22354 267/300 Read block 0 len 40960
22355 268/300 Flush
22356 269/300 Read block 685 len 28672
22357 270/300 Read block 3599 len 28672
22358 271/300 Read block 3786 len 28672
22359 272/300 Write block 4122 len 28672 data: 4 3 3 3 3 4 5
22360 273/300 Write block 4870 len 4096 data: 6
22361 274/300 Write block 2986 len 40960 data: 3 4 4 4 4 4 4 5 4 4
22362 275/300 Flush
22363 276/300 Read block 1518 len 4096
22364 277/300 Write block 2668 len 24576 data: 6 2 2 4 6 6
22365 278/300 Read block 704 len 16384
22366 279/300 Flush
22367 280/300 Read block 1519 len 20480
22368 281/300 Read block 1432 len 16384
22369 282/300 Read block 3197 len 8192
22370 283/300 Read block 3166 len 28672
22371 284/300 Write block 4425 len 32768 data: 6 5 5 4 4 5 4 4
22372 285/300 Write block 1940 len 24576 data: 5 5 5 5 5 5
22373 286/300 Flush
22374 287/300 Read block 2161 len 12288
22375 288/300 Read block 1630 len 32768
22376 289/300 Read block 237 len 12288
22377 290/300 Read block 2863 len 36864
22378 291/300 Flush
22379 292/300 Write block 3781 len 8192 data: 3 3
22380 293/300 Read block 4209 len 24576
22381 294/300 Read block 3137 len 28672
22382 295/300 Read block 3546 len 28672
22383 296/300 Write block 805 len 4096 data: 4
22384 297/300 Read block 1339 len 20480
22385 298/300 Write block 3653 len 4096 data: 6
22386 299/300 Read block 2441 len 4096
22387 300/300 Write block 4602 len 20480 data: 4 4 6 6 9
22388 301/300 Read block 1134 len 12288
22389 Main task has work to do, go find it
22390 got dsc Start(1)
22391 start 1
22392 Mar 17 22:21:16.288 INFO request completed, latency_us: 235, response_code: 204, uri: /start/cid/1, method: POST, req_id: e2a0fb5e-ebdf-466d-a194-cc146f9e018c, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
22393 Replay: started 1, returned:Ok(())
22394 ----------------------------------------------------------------
22395 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22396 States: Active Offline Active
22397 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22398 6736 Acked Read 8 Done Sent Done false
22399 6737 Acked Read 3 Done Sent Done false
22400 6738 Acked Write 10 Done Sent Done false
22401 6739 Acked Write 7 Done Sent Done false
22402 6740 Acked Read 1 Done Sent Done false
22403 6741 Acked Write 3 Done Sent Done false
22404 6742 Acked Flush 0 Done Sent Done false
22405 6743 Acked Read 10 Done Sent Done false
22406 6744 Acked Write 9 Done Sent Done false
22407 6745 Acked Write 10 Done Sent Done false
22408 6746 Acked Flush 0 Done Sent Done false
22409 6747 Acked Read 3 Done Sent Done false
22410 6748 Acked Read 7 Done Sent Done false
22411 6749 Acked Read 4 Done Sent Done false
22412 6750 Acked Read 4 Done Sent Done false
22413 6751 Acked Read 8 Done Sent Done false
22414 6752 Acked Write 6 Done Sent Done false
22415 6753 Acked Read 3 Done Sent Done false
22416 6754 Acked Write 1 Done Sent Done false
22417 6755 Acked Write 2 Done Sent Done false
22418 6756 Acked Write 2 Done Sent Done false
22419 6757 Acked Flush 0 Done Sent Done false
22420 6758 Acked Read 10 Done Sent Done false
22421 6759 Acked Read 4 Done Sent Done false
22422 6760 Acked Write 2 Done Sent Done false
22423 6761 Acked Flush 0 Done Sent Done false
22424 6762 Acked Write 1 Done Sent Done false
22425 6763 Acked Flush 0 Done Sent Done false
22426 6764 Acked Read 5 Done Sent Done false
22427 6765 Acked Write 4 Done Sent Done false
22428 6766 Acked Read 9 Done Sent Done false
22429 6767 Acked Flush 0 Done Sent Done false
22430 6768 Acked Read 8 Done Sent Done false
22431 6769 Acked Write 4 Done Sent Done false
22432 6770 Acked Write 3 Done Sent Done false
22433 6771 Acked Read 9 Done Sent Done false
22434 6772 Acked Read 1 Done Sent Done false
22435 6773 Acked Write 3 Done Sent Done false
22436 6774 Acked Flush 0 Done Sent Done false
22437 6775 Acked Read 4 Done Sent Done false
22438 6776 Acked Read 2 Done Sent Done false
22439 6777 Acked Read 3 Done Sent Done false
22440 6778 Acked Write 10 Done Sent Done false
22441 6779 Acked Read 10 Done Sent Done false
22442 6780 Acked Read 5 Done Sent Done false
22443 6781 Acked Write 3 Done Sent Done false
22444 6782 Acked Write 10 Done Sent Done false
22445 6783 Acked Read 1 Done Sent Done false
22446 6784 Acked Flush 0 Done Sent Done false
22447 6785 Acked Write 7 Done Sent Done false
22448 6786 Acked Read 2 Done Sent Done false
22449 6787 Acked Write 10 Done Sent Done false
22450 6788 Acked Read 8 Done Sent Done false
22451 6789 Acked Write 2 Done Sent Done false
22452 6790 Acked Write 10 Done Sent Done false
22453 6791 Acked Write 1 Done Sent Done false
22454 6792 Acked Write 6 Done Sent Done false
22455 6793 Acked Read 6 Done Sent Done false
22456 6794 Acked Write 6 Done Sent Done false
22457 6795 Acked Flush 0 Done Sent Done false
22458 6796 Acked Read 7 Done Sent Done false
22459 6797 Acked Read 4 Done Sent Done false
22460 6798 Acked Write 4 Done Sent Done false
22461 6799 Acked Write 9 Done Sent Done false
22462 6800 Acked Read 6 Done Sent Done false
22463 6801 Acked Write 5 Done Sent Done false
22464 6802 Acked Write 6 Done Sent Done false
22465 6803 Acked Write 1 Done Sent Done false
22466 6804 Acked Read 2 Done Sent Done false
22467 6805 Acked Write 1 Done Sent Done false
22468 6806 Acked Read 10 Done Sent Done false
22469 6807 Acked Write 3 Done Sent Done false
22470 6808 Acked Write 4 Done Sent Done false
22471 6809 Acked Flush 0 Done Sent Done false
22472 6810 Acked Read 9 Done Sent Done false
22473 6811 Acked Read 6 Done Sent Done false
22474 6812 Acked Read 3 Done Sent Done false
22475 6813 Acked Read 2 Done Sent Done false
22476 6814 Acked Read 7 Done Sent Done false
22477 6815 Acked Read 1 Done Sent Done false
22478 6816 Acked Read 3 Done Sent Done false
22479 6817 Acked Read 6 Done Sent Done false
22480 6818 Acked Write 10 Done Sent Done false
22481 6819 Acked Write 8 Done Sent Done false
22482 6820 Acked Write 7 Done Sent Done false
22483 6821 Acked Read 7 Done Sent Done false
22484 6822 Acked Read 3 Done Sent Done false
22485 6823 Acked Read 8 Done Sent Done false
22486 6824 Acked Read 1 Done Sent Done false
22487 6825 Acked Write 2 Done Sent Done false
22488 6826 Acked Read 7 Done Sent Done false
22489 6827 Acked Read 2 Done Sent Done false
22490 6828 Acked Write 9 Done Sent Done false
22491 6829 Acked Write 1 Done Sent Done false
22492 6830 Acked Write 4 Done Sent Done false
22493 6831 Acked Write 4 Done Sent Done false
22494 6832 Acked Read 2 Done Sent Done false
22495 6833 Acked Read 7 Done Sent Done false
22496 6834 Acked Read 7 Done Sent Done false
22497 6835 Acked Flush 0 Done Sent Done false
22498 6836 Acked Write 3 Done Sent Done false
22499 6837 Acked Read 5 Done Sent Done false
22500 6838 Acked Read 3 Done Sent Done false
22501 6839 Acked Flush 0 Done Sent Done false
22502 6840 Acked Write 8 Done Sent Done false
22503 6841 Acked Read 1 Done Sent Done false
22504 6842 Acked Write 9 Done Sent Done false
22505 6843 Acked Read 9 Done Sent Done false
22506 6844 Acked Write 1 Done Sent Done false
22507 6845 Acked Flush 0 Done Sent Done false
22508 6846 Acked Read 2 Done Sent Done false
22509 6847 Acked Write 1 Done Sent Done false
22510 6848 Acked Flush 0 Done Sent Done false
22511 6849 Acked Read 1 Done Sent Done false
22512 6850 Acked Write 4 Done Sent Done false
22513 6851 Acked Read 7 Done Sent Done false
22514 6852 Acked Read 3 Done Sent Done false
22515 6853 Acked Write 8 Done Sent Done false
22516 6854 Acked Write 10 Done Sent Done false
22517 6855 Acked Write 10 Done Sent Done false
22518 6856 Acked Read 1 Done Sent Done false
22519 6857 Acked Write 6 Done Sent Done false
22520 6858 Acked Write 9 Done Sent Done false
22521 6859 Acked Read 2 Done Sent Done false
22522 6860 Acked Write 7 Done Sent Done false
22523 6861 Acked Write 7 Done Sent Done false
22524 6862 Acked Read 3 Done Sent Done false
22525 6863 Acked Flush 0 Done Sent Done false
22526 6864 Acked Read 9 Done Sent Done false
22527 6865 Acked Write 3 Done Sent Done false
22528 6866 Acked Read 8 Done Sent Done false
22529 6867 Acked Write 5 Done Sent Done false
22530 6868 Acked Write 3 Done Sent Done false
22531 6869 Acked Read 9 Done Sent Done false
22532 6870 Acked Read 8 Done Sent Done false
22533 6871 Acked Write 4 Done Sent Done false
22534 6872 Acked Read 8 Done Sent Done false
22535 6873 Acked Read 2 Done Sent Done false
22536 6874 Acked Flush 0 Done Sent Done false
22537 6875 Acked Read 10 Done Sent Done false
22538 6876 Acked Read 10 Done Sent Done false
22539 6877 Acked Read 6 Done Sent Done false
22540 6878 Acked Write 8 Done Sent Done false
22541 6879 Acked Read 9 Done Sent Done false
22542 6880 Acked Write 1 Done Sent Done false
22543 6881 Acked Flush 0 Done Sent Done false
22544 6882 Acked Flush 0 Done Sent Done false
22545 6883 Acked Read 4 Done Sent Done false
22546 6884 Acked Flush 0 Done Sent Done false
22547 6885 Acked Read 1 Done Sent Done false
22548 6886 Acked Read 4 Done Sent Done false
22549 6887 Acked Read 2 Done Sent Done false
22550 6888 Acked Read 8 Done Sent Done false
22551 6889 Acked Read 2 Done Sent Done false
22552 6890 Acked Flush 0 Done Sent Done false
22553 6891 Acked Read 7 Done Sent Done false
22554 6892 Acked Read 10 Done Sent Done false
22555 6893 Acked Write 10 Done Sent Done false
22556 6894 Acked Read 5 Done Sent Done false
22557 6895 Acked Flush 0 Done Sent Done false
22558 6896 Acked Flush 0 Done Sent Done false
22559 6897 Acked Read 1 Done Sent Done false
22560 6898 Acked Write 8 Done Sent Done false
22561 6899 Acked Read 10 Done Sent Done false
22562 6900 Acked Flush 0 Done Sent Done false
22563 6901 Acked Write 7 Done Sent Done false
22564 6902 Acked Read 8 Done Sent Done false
22565 6903 Acked Read 10 Done Sent Done false
22566 6904 Acked Write 3 Done Sent Done false
22567 6905 Acked Write 5 Done Sent Done false
22568 6906 Acked Read 9 Done Sent Done false
22569 6907 Acked Write 2 Done Sent Done false
22570 6908 Acked Read 4 Done Sent Done false
22571 6909 Acked Read 10 Done Sent Done false
22572 6910 Acked Write 8 Done Sent Done false
22573 6911 Acked Write 2 Done Sent Done false
22574 6912 Acked Write 5 Done Sent Done false
22575 6913 Acked Write 7 Done Sent Done false
22576 6914 Acked Write 1 Done Sent Done false
22577 6915 Acked Read 5 Done Sent Done false
22578 6916 Acked Write 1 Done Sent Done false
22579 6917 Acked Write 2 Done Sent Done false
22580 6918 Acked Write 10 Done Sent Done false
22581 6919 Acked Write 6 Done Sent Done false
22582 6920 Acked Write 1 Done Sent Done false
22583 6921 Acked Write 7 Done Sent Done false
22584 6922 Acked Read 1 Done Sent Done false
22585 6923 Acked Read 4 Done Sent Done false
22586 6924 Acked Flush 0 Done Sent Done false
22587 6925 Acked Read 1 Done Sent Done false
22588 6926 Acked Read 4 Done Sent Done false
22589 6927 Acked Flush 0 Done Sent Done false
22590 6928 Acked Write 7 Done Sent Done false
22591 6929 Acked Write 10 Done Sent Done false
22592 6930 Acked Write 5 Done Sent Done false
22593 6931 Acked Write 8 Done Sent Done false
22594 6932 Acked Read 4 Done Sent Done false
22595 6933 Acked Write 4 Done Sent Done false
22596 6934 Acked Write 6 Done Sent Done false
22597 6935 Acked Write 3 Done Sent Done false
22598 6936 Acked Read 5 Done Sent Done false
22599 6937 Acked Flush 0 Done Sent Done false
22600 6938 Acked Read 5 Done Sent Done false
22601 6939 Acked Write 3 Done Sent Done false
22602 6940 Acked Write 7 Done Sent Done false
22603 6941 Acked Read 1 Done Sent Done false
22604 6942 Acked Write 7 Done Sent Done false
22605 6943 Acked Read 5 Done Sent Done false
22606 6944 Acked Write 1 Done Sent Done false
22607 6945 Acked Write 1 Done Sent Done false
22608 6946 Acked Read 9 Done Sent Done false
22609 6947 Acked Write 6 Done Sent Done false
22610 6948 Acked Flush 0 Done Sent Done false
22611 6949 Acked Write 4 Done Sent Done false
22612 6950 Acked Write 2 Done Sent Done false
22613 6951 Acked Read 2 Done Sent Done false
22614 6952 Acked Flush 0 Done Sent Done false
22615 6953 Acked Write 9 Done Sent Done false
22616 6954 Acked Write 5 Done Sent Done false
22617 6955 Acked Write 7 Done Sent Done false
22618 6956 Acked Flush 0 Done Sent Done false
22619 6957 Acked Flush 0 Done Sent Done false
22620 6958 Acked Read 8 Done Sent Done false
22621 6959 Acked Read 4 Done Sent Done false
22622 6960 Acked Flush 0 Done Sent Done false
22623 6961 Acked Read 4 Done Sent Done false
22624 6962 Acked Read 7 Done Sent Done false
22625 6963 Acked Read 10 Done Sent Done false
22626 6964 Acked Read 2 Done Sent Done false
22627 6965 Acked Flush 0 Done Sent Done false
22628 6966 Acked Read 6 Done Sent Done false
22629 6967 Acked Read 1 Done Sent Done false
22630 6968 Acked Read 10 Done Sent Done false
22631 6969 Acked Read 5 Done Sent Done false
22632 6970 Acked Read 2 Done Sent Done false
22633 6971 Acked Read 4 Done Sent Done false
22634 6972 Acked Write 9 Done Sent Done false
22635 6973 Acked Read 3 Done Sent Done false
22636 6974 Acked Write 6 Done Sent Done false
22637 6975 Acked Write 5 Done Sent Done false
22638 6976 Acked Read 7 Done Sent Done false
22639 6977 Acked Flush 0 Done Sent Done false
22640 6978 Acked Read 2 Done Sent Done false
22641 6979 Acked Write 5 Done Sent Done false
22642 6980 Acked Read 1 Done Sent Done false
22643 6981 Acked Write 4 Done Sent Done false
22644 6982 Acked Read 7 Done Sent Done false
22645 6983 Acked Read 1 Done Sent Done false
22646 6984 Acked Read 1 Done Sent Done false
22647 6985 Acked Read 8 Done Sent Done false
22648 6986 Acked Write 1 Done Sent Done false
22649 6987 Acked Write 3 Done Sent Done false
22650 6988 Acked Write 3 Done Sent Done false
22651 6989 Acked Read 8 Done Sent Done false
22652 6990 Acked Write 2 Done Sent Done false
22653 6991 Acked Read 3 Done Sent Done false
22654 6992 Acked Read 7 Done Sent Done false
22655 6993 Acked Write 7 Done Sent Done false
22656 6994 Acked Write 8 Done Sent Done false
22657 6995 Acked Read 9 Done Sent Done false
22658 6996 Acked Write 6 Done Sent Done false
22659 6997 Acked Write 8 Done Sent Done false
22660 6998 Acked Read 6 Done Sent Done false
22661 6999 Acked Read 3 Done Sent Done false
22662 7000 Acked Read 4 Done Sent Done false
22663 7001 Acked Read 1 Done Sent Done false
22664 7002 Acked Read 10 Done Sent Done false
22665 7003 Acked Flush 0 Done Sent Done false
22666 7004 Acked Read 7 Done Sent Done false
22667 7005 Acked Read 7 Done Sent Done false
22668 7006 Acked Read 7 Done Sent Done false
22669 7007 Acked Write 7 Done Sent Done false
22670 7008 Acked Write 1 Done Sent Done false
22671 7009 Acked Write 10 Done Sent Done false
22672 7010 Acked Flush 0 Done Sent Done false
22673 7011 Acked Read 1 Done Sent Done false
22674 7012 Acked Write 6 Done Sent Done false
22675 7013 Acked Read 4 Done Sent Done false
22676 7014 Acked Flush 0 Done Sent Done false
22677 7015 Acked Read 5 Done Sent Done false
22678 7016 Acked Read 4 Done Sent Done false
22679 7017 Acked Read 2 Done Sent Done false
22680 7018 Acked Read 7 Done Sent Done false
22681 7019 Acked Write 8 Done Sent Done false
22682 7020 Acked Write 6 Done Sent Done false
22683 7021 Acked Flush 0 Done Sent Done false
22684 7022 Acked Read 3 Done Sent Done false
22685 7023 Acked Read 8 Done Sent Done false
22686 7024 Acked Read 3 Done Sent Done false
22687 7025 Acked Read 9 Done Sent Done false
22688 7026 Acked Flush 0 Done Sent Done false
22689 7027 Acked Write 2 Done Sent Done false
22690 7028 Acked Read 6 Done Sent Done false
22691 7029 Acked Read 7 Done Sent Done false
22692 7030 Acked Read 7 Done Sent Done false
22693 7031 Acked Write 1 Done Sent Done false
22694 7032 Acked Read 5 Done Sent Done false
22695 7033 Acked Write 1 Done Sent Done false
22696 7034 Acked Read 1 Done Sent Done false
22697 7035 Acked Write 5 Done Sent Done false
22698 7036 Acked Read 3 Done Sent Done false
22699 STATES DS:0 DS:1 DS:2 TOTAL
22700 Sent 0 301 0 301
22701 Done 301 0 301 602
22702 Skipped 0 0 0 0
22703 Error 0 0 0 0
22704 Last Flush: Some(JobId(7026)) Some(JobId(6735)) Some(JobId(7026))
22705 Downstairs last five completed: 6735 6734 6733 6732 6731
22706 Upstairs last five completed: 7036 7035 7034 7033 7032
22707 CLIENT: Up:0 ds:301 act:2
227082025-03-17T22:21:17.072ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
227092025-03-17T22:21:18.072ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
227102025-03-17T22:21:19.074ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
22711 ----------------------------------------------------------------
22712 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22713 States: Active Offline Active
22714 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
22715 6736 Acked Read 8 Done Sent Done false
22716 6737 Acked Read 3 Done Sent Done false
22717 6738 Acked Write 10 Done Sent Done false
22718 6739 Acked Write 7 Done Sent Done false
22719 6740 Acked Read 1 Done Sent Done false
22720 6741 Acked Write 3 Done Sent Done false
22721 6742 Acked Flush 0 Done Sent Done false
22722 6743 Acked Read 10 Done Sent Done false
22723 6744 Acked Write 9 Done Sent Done false
22724 6745 Acked Write 10 Done Sent Done false
22725 6746 Acked Flush 0 Done Sent Done false
22726 6747 Acked Read 3 Done Sent Done false
22727 6748 Acked Read 7 Done Sent Done false
22728 6749 Acked Read 4 Done Sent Done false
22729 6750 Acked Read 4 Done Sent Done false
22730 6751 Acked Read 8 Done Sent Done false
22731 6752 Acked Write 6 Done Sent Done false
22732 6753 Acked Read 3 Done Sent Done false
22733 6754 Acked Write 1 Done Sent Done false
22734 6755 Acked Write 2 Done Sent Done false
22735 6756 Acked Write 2 Done Sent Done false
22736 6757 Acked Flush 0 Done Sent Done false
22737 6758 Acked Read 10 Done Sent Done false
22738 6759 Acked Read 4 Done Sent Done false
22739 6760 Acked Write 2 Done Sent Done false
22740 6761 Acked Flush 0 Done Sent Done false
22741 6762 Acked Write 1 Done Sent Done false
22742 6763 Acked Flush 0 Done Sent Done false
22743 6764 Acked Read 5 Done Sent Done false
22744 6765 Acked Write 4 Done Sent Done false
22745 6766 Acked Read 9 Done Sent Done false
22746 6767 Acked Flush 0 Done Sent Done false
22747 6768 Acked Read 8 Done Sent Done false
22748 6769 Acked Write 4 Done Sent Done false
22749 6770 Acked Write 3 Done Sent Done false
22750 6771 Acked Read 9 Done Sent Done false
22751 6772 Acked Read 1 Done Sent Done false
22752 6773 Acked Write 3 Done Sent Done false
22753 6774 Acked Flush 0 Done Sent Done false
22754 6775 Acked Read 4 Done Sent Done false
22755 6776 Acked Read 2 Done Sent Done false
22756 6777 Acked Read 3 Done Sent Done false
22757 6778 Acked Write 10 Done Sent Done false
22758 6779 Acked Read 10 Done Sent Done false
22759 6780 Acked Read 5 Done Sent Done false
22760 6781 Acked Write 3 Done Sent Done false
22761 6782 Acked Write 10 Done Sent Done false
22762 6783 Acked Read 1 Done Sent Done false
22763 6784 Acked Flush 0 Done Sent Done false
22764 6785 Acked Write 7 Done Sent Done false
22765 6786 Acked Read 2 Done Sent Done false
22766 6787 Acked Write 10 Done Sent Done false
22767 6788 Acked Read 8 Done Sent Done false
22768 6789 Acked Write 2 Done Sent Done false
22769 6790 Acked Write 10 Done Sent Done false
22770 6791 Acked Write 1 Done Sent Done false
22771 6792 Acked Write 6 Done Sent Done false
22772 6793 Acked Read 6 Done Sent Done false
22773 6794 Acked Write 6 Done Sent Done false
22774 6795 Acked Flush 0 Done Sent Done false
22775 6796 Acked Read 7 Done Sent Done false
22776 6797 Acked Read 4 Done Sent Done false
22777 6798 Acked Write 4 Done Sent Done false
22778 6799 Acked Write 9 Done Sent Done false
22779 6800 Acked Read 6 Done Sent Done false
22780 6801 Acked Write 5 Done Sent Done false
22781 6802 Acked Write 6 Done Sent Done false
22782 6803 Acked Write 1 Done Sent Done false
22783 6804 Acked Read 2 Done Sent Done false
22784 6805 Acked Write 1 Done Sent Done false
22785 6806 Acked Read 10 Done Sent Done false
22786 6807 Acked Write 3 Done Sent Done false
22787 6808 Acked Write 4 Done Sent Done false
22788 6809 Acked Flush 0 Done Sent Done false
22789 6810 Acked Read 9 Done Sent Done false
22790 6811 Acked Read 6 Done Sent Done false
22791 6812 Acked Read 3 Done Sent Done false
22792 6813 Acked Read 2 Done Sent Done false
22793 6814 Acked Read 7 Done Sent Done false
22794 6815 Acked Read 1 Done Sent Done false
22795 6816 Acked Read 3 Done Sent Done false
22796 6817 Acked Read 6 Done Sent Done false
22797 6818 Acked Write 10 Done Sent Done false
22798 6819 Acked Write 8 Done Sent Done false
22799 6820 Acked Write 7 Done Sent Done false
22800 6821 Acked Read 7 Done Sent Done false
22801 6822 Acked Read 3 Done Sent Done false
22802 6823 Acked Read 8 Done Sent Done false
22803 6824 Acked Read 1 Done Sent Done false
22804 6825 Acked Write 2 Done Sent Done false
22805 6826 Acked Read 7 Done Sent Done false
22806 6827 Acked Read 2 Done Sent Done false
22807 6828 Acked Write 9 Done Sent Done false
22808 6829 Acked Write 1 Done Sent Done false
22809 6830 Acked Write 4 Done Sent Done false
22810 6831 Acked Write 4 Done Sent Done false
22811 6832 Acked Read 2 Done Sent Done false
22812 6833 Acked Read 7 Done Sent Done false
22813 6834 Acked Read 7 Done Sent Done false
22814 6835 Acked Flush 0 Done Sent Done false
22815 6836 Acked Write 3 Done Sent Done false
22816 6837 Acked Read 5 Done Sent Done false
22817 6838 Acked Read 3 Done Sent Done false
22818 6839 Acked Flush 0 Done Sent Done false
22819 6840 Acked Write 8 Done Sent Done false
22820 6841 Acked Read 1 Done Sent Done false
22821 6842 Acked Write 9 Done Sent Done false
22822 6843 Acked Read 9 Done Sent Done false
22823 6844 Acked Write 1 Done Sent Done false
22824 6845 Acked Flush 0 Done Sent Done false
22825 6846 Acked Read 2 Done Sent Done false
22826 6847 Acked Write 1 Done Sent Done false
22827 6848 Acked Flush 0 Done Sent Done false
22828 6849 Acked Read 1 Done Sent Done false
22829 6850 Acked Write 4 Done Sent Done false
22830 6851 Acked Read 7 Done Sent Done false
22831 6852 Acked Read 3 Done Sent Done false
22832 6853 Acked Write 8 Done Sent Done false
22833 6854 Acked Write 10 Done Sent Done false
22834 6855 Acked Write 10 Done Sent Done false
22835 6856 Acked Read 1 Done Sent Done false
22836 6857 Acked Write 6 Done Sent Done false
22837 6858 Acked Write 9 Done Sent Done false
22838 6859 Acked Read 2 Done Sent Done false
22839 6860 Acked Write 7 Done Sent Done false
22840 6861 Acked Write 7 Done Sent Done false
22841 6862 Acked Read 3 Done Sent Done false
22842 6863 Acked Flush 0 Done Sent Done false
22843 6864 Acked Read 9 Done Sent Done false
22844 6865 Acked Write 3 Done Sent Done false
22845 6866 Acked Read 8 Done Sent Done false
22846 6867 Acked Write 5 Done Sent Done false
22847 6868 Acked Write 3 Done Sent Done false
22848 6869 Acked Read 9 Done Sent Done false
22849 6870 Acked Read 8 Done Sent Done false
22850 6871 Acked Write 4 Done Sent Done false
22851 6872 Acked Read 8 Done Sent Done false
22852 6873 Acked Read 2 Done Sent Done false
22853 6874 Acked Flush 0 Done Sent Done false
22854 6875 Acked Read 10 Done Sent Done false
22855 6876 Acked Read 10 Done Sent Done false
22856 6877 Acked Read 6 Done Sent Done false
22857 6878 Acked Write 8 Done Sent Done false
22858 6879 Acked Read 9 Done Sent Done false
22859 6880 Acked Write 1 Done Sent Done false
22860 6881 Acked Flush 0 Done Sent Done false
22861 6882 Acked Flush 0 Done Sent Done false
22862 6883 Acked Read 4 Done Sent Done false
22863 6884 Acked Flush 0 Done Sent Done false
22864 6885 Acked Read 1 Done Sent Done false
22865 6886 Acked Read 4 Done Sent Done false
22866 6887 Acked Read 2 Done Sent Done false
22867 6888 Acked Read 8 Done Sent Done false
22868 6889 Acked Read 2 Done Sent Done false
22869 6890 Acked Flush 0 Done Sent Done false
22870 6891 Acked Read 7 Done Sent Done false
22871 6892 Acked Read 10 Done Sent Done false
22872 6893 Acked Write 10 Done Sent Done false
22873 6894 Acked Read 5 Done Sent Done false
22874 6895 Acked Flush 0 Done Sent Done false
22875 6896 Acked Flush 0 Done Sent Done false
22876 6897 Acked Read 1 Done Sent Done false
22877 6898 Acked Write 8 Done Sent Done false
22878 6899 Acked Read 10 Done Sent Done false
22879 6900 Acked Flush 0 Done Sent Done false
22880 6901 Acked Write 7 Done Sent Done false
22881 6902 Acked Read 8 Done Sent Done false
22882 6903 Acked Read 10 Done Sent Done false
22883 6904 Acked Write 3 Done Sent Done false
22884 6905 Acked Write 5 Done Sent Done false
22885 6906 Acked Read 9 Done Sent Done false
22886 6907 Acked Write 2 Done Sent Done false
22887 6908 Acked Read 4 Done Sent Done false
22888 6909 Acked Read 10 Done Sent Done false
22889 6910 Acked Write 8 Done Sent Done false
22890 6911 Acked Write 2 Done Sent Done false
22891 6912 Acked Write 5 Done Sent Done false
22892 6913 Acked Write 7 Done Sent Done false
22893 6914 Acked Write 1 Done Sent Done false
22894 6915 Acked Read 5 Done Sent Done false
22895 6916 Acked Write 1 Done Sent Done false
22896 6917 Acked Write 2 Done Sent Done false
22897 6918 Acked Write 10 Done Sent Done false
22898 6919 Acked Write 6 Done Sent Done false
22899 6920 Acked Write 1 Done Sent Done false
22900 6921 Acked Write 7 Done Sent Done false
22901 6922 Acked Read 1 Done Sent Done false
22902 6923 Acked Read 4 Done Sent Done false
22903 6924 Acked Flush 0 Done Sent Done false
22904 6925 Acked Read 1 Done Sent Done false
22905 6926 Acked Read 4 Done Sent Done false
22906 6927 Acked Flush 0 Done Sent Done false
22907 6928 Acked Write 7 Done Sent Done false
22908 6929 Acked Write 10 Done Sent Done false
22909 6930 Acked Write 5 Done Sent Done false
22910 6931 Acked Write 8 Done Sent Done false
22911 6932 Acked Read 4 Done Sent Done false
22912 6933 Acked Write 4 Done Sent Done false
22913 6934 Acked Write 6 Done Sent Done false
22914 6935 Acked Write 3 Done Sent Done false
22915 6936 Acked Read 5 Done Sent Done false
22916 6937 Acked Flush 0 Done Sent Done false
22917 6938 Acked Read 5 Done Sent Done false
22918 6939 Acked Write 3 Done Sent Done false
22919 6940 Acked Write 7 Done Sent Done false
22920 6941 Acked Read 1 Done Sent Done false
22921 6942 Acked Write 7 Done Sent Done false
22922 6943 Acked Read 5 Done Sent Done false
22923 6944 Acked Write 1 Done Sent Done false
22924 6945 Acked Write 1 Done Sent Done false
22925 6946 Acked Read 9 Done Sent Done false
22926 6947 Acked Write 6 Done Sent Done false
22927 6948 Acked Flush 0 Done Sent Done false
22928 6949 Acked Write 4 Done Sent Done false
22929 6950 Acked Write 2 Done Sent Done false
22930 6951 Acked Read 2 Done Sent Done false
22931 6952 Acked Flush 0 Done Sent Done false
22932 6953 Acked Write 9 Done Sent Done false
22933 6954 Acked Write 5 Done Sent Done false
22934 6955 Acked Write 7 Done Sent Done false
22935 6956 Acked Flush 0 Done Sent Done false
22936 6957 Acked Flush 0 Done Sent Done false
22937 6958 Acked Read 8 Done Sent Done false
22938 6959 Acked Read 4 Done Sent Done false
22939 6960 Acked Flush 0 Done Sent Done false
22940 6961 Acked Read 4 Done Sent Done false
22941 6962 Acked Read 7 Done Sent Done false
22942 6963 Acked Read 10 Done Sent Done false
22943 6964 Acked Read 2 Done Sent Done false
22944 6965 Acked Flush 0 Done Sent Done false
22945 6966 Acked Read 6 Done Sent Done false
22946 6967 Acked Read 1 Done Sent Done false
22947 6968 Acked Read 10 Done Sent Done false
22948 6969 Acked Read 5 Done Sent Done false
22949 6970 Acked Read 2 Done Sent Done false
22950 6971 Acked Read 4 Done Sent Done false
22951 6972 Acked Write 9 Done Sent Done false
22952 6973 Acked Read 3 Done Sent Done false
22953 6974 Acked Write 6 Done Sent Done false
22954 6975 Acked Write 5 Done Sent Done false
22955 6976 Acked Read 7 Done Sent Done false
22956 6977 Acked Flush 0 Done Sent Done false
22957 6978 Acked Read 2 Done Sent Done false
22958 6979 Acked Write 5 Done Sent Done false
22959 6980 Acked Read 1 Done Sent Done false
22960 6981 Acked Write 4 Done Sent Done false
22961 6982 Acked Read 7 Done Sent Done false
22962 6983 Acked Read 1 Done Sent Done false
22963 6984 Acked Read 1 Done Sent Done false
22964 6985 Acked Read 8 Done Sent Done false
22965 6986 Acked Write 1 Done Sent Done false
22966 6987 Acked Write 3 Done Sent Done false
22967 6988 Acked Write 3 Done Sent Done false
22968 6989 Acked Read 8 Done Sent Done false
22969 6990 Acked Write 2 Done Sent Done false
22970 6991 Acked Read 3 Done Sent Done false
22971 6992 Acked Read 7 Done Sent Done false
22972 6993 Acked Write 7 Done Sent Done false
22973 6994 Acked Write 8 Done Sent Done false
22974 6995 Acked Read 9 Done Sent Done false
22975 6996 Acked Write 6 Done Sent Done false
22976 6997 Acked Write 8 Done Sent Done false
22977 6998 Acked Read 6 Done Sent Done false
22978 6999 Acked Read 3 Done Sent Done false
22979 7000 Acked Read 4 Done Sent Done false
22980 7001 Acked Read 1 Done Sent Done false
22981 7002 Acked Read 10 Done Sent Done false
22982 7003 Acked Flush 0 Done Sent Done false
22983 7004 Acked Read 7 Done Sent Done false
22984 7005 Acked Read 7 Done Sent Done false
22985 7006 Acked Read 7 Done Sent Done false
22986 7007 Acked Write 7 Done Sent Done false
22987 7008 Acked Write 1 Done Sent Done false
22988 7009 Acked Write 10 Done Sent Done false
22989 7010 Acked Flush 0 Done Sent Done false
22990 7011 Acked Read 1 Done Sent Done false
22991 7012 Acked Write 6 Done Sent Done false
22992 7013 Acked Read 4 Done Sent Done false
22993 7014 Acked Flush 0 Done Sent Done false
22994 7015 Acked Read 5 Done Sent Done false
22995 7016 Acked Read 4 Done Sent Done false
22996 7017 Acked Read 2 Done Sent Done false
22997 7018 Acked Read 7 Done Sent Done false
22998 7019 Acked Write 8 Done Sent Done false
22999 7020 Acked Write 6 Done Sent Done false
23000 7021 Acked Flush 0 Done Sent Done false
23001 7022 Acked Read 3 Done Sent Done false
23002 7023 Acked Read 8 Done Sent Done false
23003 7024 Acked Read 3 Done Sent Done false
23004 7025 Acked Read 9 Done Sent Done false
23005 7026 Acked Flush 0 Done Sent Done false
23006 7027 Acked Write 2 Done Sent Done false
23007 7028 Acked Read 6 Done Sent Done false
23008 7029 Acked Read 7 Done Sent Done false
23009 7030 Acked Read 7 Done Sent Done false
23010 7031 Acked Write 1 Done Sent Done false
23011 7032 Acked Read 5 Done Sent Done false
23012 7033 Acked Write 1 Done Sent Done false
23013 7034 Acked Read 1 Done Sent Done false
23014 7035 Acked Write 5 Done Sent Done false
23015 7036 Acked Read 3 Done Sent Done false
23016 STATES DS:0 DS:1 DS:2 TOTAL
23017 Sent 0 301 0 301
23018 Done 301 0 301 602
23019 Skipped 0 0 0 0
23020 Error 0 0 0 0
23021 Last Flush: Some(JobId(7026)) Some(JobId(6735)) Some(JobId(7026))
23022 Downstairs last five completed: 6735 6734 6733 6732 6731
23023 Upstairs last five completed: 7036 7035 7034 7033 7032
23024 CLIENT: Up:0 ds:301 act:2
230252025-03-17T22:21:21.077ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
230262025-03-17T22:21:22.047ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
230272025-03-17T22:21:22.047ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
230282025-03-17T22:21:22.047ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
230292025-03-17T22:21:22.047ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
230302025-03-17T22:21:22.048ZINFOcrucible: client task is exiting = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
230312025-03-17T22:21:22.549ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
230322025-03-17T22:21:23.081ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
230332025-03-17T22:21:24.083ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
23034 ----------------------------------------------------------------
23035 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23036 States: Active Offline Active
23037 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23038 6736 Acked Read 8 Done Sent Done false
23039 6737 Acked Read 3 Done Sent Done false
23040 6738 Acked Write 10 Done Sent Done false
23041 6739 Acked Write 7 Done Sent Done false
23042 6740 Acked Read 1 Done Sent Done false
23043 6741 Acked Write 3 Done Sent Done false
23044 6742 Acked Flush 0 Done Sent Done false
23045 6743 Acked Read 10 Done Sent Done false
23046 6744 Acked Write 9 Done Sent Done false
23047 6745 Acked Write 10 Done Sent Done false
23048 6746 Acked Flush 0 Done Sent Done false
23049 6747 Acked Read 3 Done Sent Done false
23050 6748 Acked Read 7 Done Sent Done false
23051 6749 Acked Read 4 Done Sent Done false
23052 6750 Acked Read 4 Done Sent Done false
23053 6751 Acked Read 8 Done Sent Done false
23054 6752 Acked Write 6 Done Sent Done false
23055 6753 Acked Read 3 Done Sent Done false
23056 6754 Acked Write 1 Done Sent Done false
23057 6755 Acked Write 2 Done Sent Done false
23058 6756 Acked Write 2 Done Sent Done false
23059 6757 Acked Flush 0 Done Sent Done false
23060 6758 Acked Read 10 Done Sent Done false
23061 6759 Acked Read 4 Done Sent Done false
23062 6760 Acked Write 2 Done Sent Done false
23063 6761 Acked Flush 0 Done Sent Done false
23064 6762 Acked Write 1 Done Sent Done false
23065 6763 Acked Flush 0 Done Sent Done false
23066 6764 Acked Read 5 Done Sent Done false
23067 6765 Acked Write 4 Done Sent Done false
23068 6766 Acked Read 9 Done Sent Done false
23069 6767 Acked Flush 0 Done Sent Done false
23070 6768 Acked Read 8 Done Sent Done false
23071 6769 Acked Write 4 Done Sent Done false
23072 6770 Acked Write 3 Done Sent Done false
23073 6771 Acked Read 9 Done Sent Done false
23074 6772 Acked Read 1 Done Sent Done false
23075 6773 Acked Write 3 Done Sent Done false
23076 6774 Acked Flush 0 Done Sent Done false
23077 6775 Acked Read 4 Done Sent Done false
23078 6776 Acked Read 2 Done Sent Done false
23079 6777 Acked Read 3 Done Sent Done false
23080 6778 Acked Write 10 Done Sent Done false
23081 6779 Acked Read 10 Done Sent Done false
23082 6780 Acked Read 5 Done Sent Done false
23083 6781 Acked Write 3 Done Sent Done false
23084 6782 Acked Write 10 Done Sent Done false
23085 6783 Acked Read 1 Done Sent Done false
23086 6784 Acked Flush 0 Done Sent Done false
23087 6785 Acked Write 7 Done Sent Done false
23088 6786 Acked Read 2 Done Sent Done false
23089 6787 Acked Write 10 Done Sent Done false
23090 6788 Acked Read 8 Done Sent Done false
23091 6789 Acked Write 2 Done Sent Done false
23092 6790 Acked Write 10 Done Sent Done false
23093 6791 Acked Write 1 Done Sent Done false
23094 6792 Acked Write 6 Done Sent Done false
23095 6793 Acked Read 6 Done Sent Done false
23096 6794 Acked Write 6 Done Sent Done false
23097 6795 Acked Flush 0 Done Sent Done false
23098 6796 Acked Read 7 Done Sent Done false
23099 6797 Acked Read 4 Done Sent Done false
23100 6798 Acked Write 4 Done Sent Done false
23101 6799 Acked Write 9 Done Sent Done false
23102 6800 Acked Read 6 Done Sent Done false
23103 6801 Acked Write 5 Done Sent Done false
23104 6802 Acked Write 6 Done Sent Done false
23105 6803 Acked Write 1 Done Sent Done false
23106 6804 Acked Read 2 Done Sent Done false
23107 6805 Acked Write 1 Done Sent Done false
23108 6806 Acked Read 10 Done Sent Done false
23109 6807 Acked Write 3 Done Sent Done false
23110 6808 Acked Write 4 Done Sent Done false
23111 6809 Acked Flush 0 Done Sent Done false
23112 6810 Acked Read 9 Done Sent Done false
23113 6811 Acked Read 6 Done Sent Done false
23114 6812 Acked Read 3 Done Sent Done false
23115 6813 Acked Read 2 Done Sent Done false
23116 6814 Acked Read 7 Done Sent Done false
23117 6815 Acked Read 1 Done Sent Done false
23118 6816 Acked Read 3 Done Sent Done false
23119 6817 Acked Read 6 Done Sent Done false
23120 6818 Acked Write 10 Done Sent Done false
23121 6819 Acked Write 8 Done Sent Done false
23122 6820 Acked Write 7 Done Sent Done false
23123 6821 Acked Read 7 Done Sent Done false
23124 6822 Acked Read 3 Done Sent Done false
23125 6823 Acked Read 8 Done Sent Done false
23126 6824 Acked Read 1 Done Sent Done false
23127 6825 Acked Write 2 Done Sent Done false
23128 6826 Acked Read 7 Done Sent Done false
23129 6827 Acked Read 2 Done Sent Done false
23130 6828 Acked Write 9 Done Sent Done false
23131 6829 Acked Write 1 Done Sent Done false
23132 6830 Acked Write 4 Done Sent Done false
23133 6831 Acked Write 4 Done Sent Done false
23134 6832 Acked Read 2 Done Sent Done false
23135 6833 Acked Read 7 Done Sent Done false
23136 6834 Acked Read 7 Done Sent Done false
23137 6835 Acked Flush 0 Done Sent Done false
23138 6836 Acked Write 3 Done Sent Done false
23139 6837 Acked Read 5 Done Sent Done false
23140 6838 Acked Read 3 Done Sent Done false
23141 6839 Acked Flush 0 Done Sent Done false
23142 6840 Acked Write 8 Done Sent Done false
23143 6841 Acked Read 1 Done Sent Done false
23144 6842 Acked Write 9 Done Sent Done false
23145 6843 Acked Read 9 Done Sent Done false
23146 6844 Acked Write 1 Done Sent Done false
23147 6845 Acked Flush 0 Done Sent Done false
23148 6846 Acked Read 2 Done Sent Done false
23149 6847 Acked Write 1 Done Sent Done false
23150 6848 Acked Flush 0 Done Sent Done false
23151 6849 Acked Read 1 Done Sent Done false
23152 6850 Acked Write 4 Done Sent Done false
23153 6851 Acked Read 7 Done Sent Done false
23154 6852 Acked Read 3 Done Sent Done false
23155 6853 Acked Write 8 Done Sent Done false
23156 6854 Acked Write 10 Done Sent Done false
23157 6855 Acked Write 10 Done Sent Done false
23158 6856 Acked Read 1 Done Sent Done false
23159 6857 Acked Write 6 Done Sent Done false
23160 6858 Acked Write 9 Done Sent Done false
23161 6859 Acked Read 2 Done Sent Done false
23162 6860 Acked Write 7 Done Sent Done false
23163 6861 Acked Write 7 Done Sent Done false
23164 6862 Acked Read 3 Done Sent Done false
23165 6863 Acked Flush 0 Done Sent Done false
23166 6864 Acked Read 9 Done Sent Done false
23167 6865 Acked Write 3 Done Sent Done false
23168 6866 Acked Read 8 Done Sent Done false
23169 6867 Acked Write 5 Done Sent Done false
23170 6868 Acked Write 3 Done Sent Done false
23171 6869 Acked Read 9 Done Sent Done false
23172 6870 Acked Read 8 Done Sent Done false
23173 6871 Acked Write 4 Done Sent Done false
23174 6872 Acked Read 8 Done Sent Done false
23175 6873 Acked Read 2 Done Sent Done false
23176 6874 Acked Flush 0 Done Sent Done false
23177 6875 Acked Read 10 Done Sent Done false
23178 6876 Acked Read 10 Done Sent Done false
23179 6877 Acked Read 6 Done Sent Done false
23180 6878 Acked Write 8 Done Sent Done false
23181 6879 Acked Read 9 Done Sent Done false
23182 6880 Acked Write 1 Done Sent Done false
23183 6881 Acked Flush 0 Done Sent Done false
23184 6882 Acked Flush 0 Done Sent Done false
23185 6883 Acked Read 4 Done Sent Done false
23186 6884 Acked Flush 0 Done Sent Done false
23187 6885 Acked Read 1 Done Sent Done false
23188 6886 Acked Read 4 Done Sent Done false
23189 6887 Acked Read 2 Done Sent Done false
23190 6888 Acked Read 8 Done Sent Done false
23191 6889 Acked Read 2 Done Sent Done false
23192 6890 Acked Flush 0 Done Sent Done false
23193 6891 Acked Read 7 Done Sent Done false
23194 6892 Acked Read 10 Done Sent Done false
23195 6893 Acked Write 10 Done Sent Done false
23196 6894 Acked Read 5 Done Sent Done false
23197 6895 Acked Flush 0 Done Sent Done false
23198 6896 Acked Flush 0 Done Sent Done false
23199 6897 Acked Read 1 Done Sent Done false
23200 6898 Acked Write 8 Done Sent Done false
23201 6899 Acked Read 10 Done Sent Done false
23202 6900 Acked Flush 0 Done Sent Done false
23203 6901 Acked Write 7 Done Sent Done false
23204 6902 Acked Read 8 Done Sent Done false
23205 6903 Acked Read 10 Done Sent Done false
23206 6904 Acked Write 3 Done Sent Done false
23207 6905 Acked Write 5 Done Sent Done false
23208 6906 Acked Read 9 Done Sent Done false
23209 6907 Acked Write 2 Done Sent Done false
23210 6908 Acked Read 4 Done Sent Done false
23211 6909 Acked Read 10 Done Sent Done false
23212 6910 Acked Write 8 Done Sent Done false
23213 6911 Acked Write 2 Done Sent Done false
23214 6912 Acked Write 5 Done Sent Done false
23215 6913 Acked Write 7 Done Sent Done false
23216 6914 Acked Write 1 Done Sent Done false
23217 6915 Acked Read 5 Done Sent Done false
23218 6916 Acked Write 1 Done Sent Done false
23219 6917 Acked Write 2 Done Sent Done false
23220 6918 Acked Write 10 Done Sent Done false
23221 6919 Acked Write 6 Done Sent Done false
23222 6920 Acked Write 1 Done Sent Done false
23223 6921 Acked Write 7 Done Sent Done false
23224 6922 Acked Read 1 Done Sent Done false
23225 6923 Acked Read 4 Done Sent Done false
23226 6924 Acked Flush 0 Done Sent Done false
23227 6925 Acked Read 1 Done Sent Done false
23228 6926 Acked Read 4 Done Sent Done false
23229 6927 Acked Flush 0 Done Sent Done false
23230 6928 Acked Write 7 Done Sent Done false
23231 6929 Acked Write 10 Done Sent Done false
23232 6930 Acked Write 5 Done Sent Done false
23233 6931 Acked Write 8 Done Sent Done false
23234 6932 Acked Read 4 Done Sent Done false
23235 6933 Acked Write 4 Done Sent Done false
23236 6934 Acked Write 6 Done Sent Done false
23237 6935 Acked Write 3 Done Sent Done false
23238 6936 Acked Read 5 Done Sent Done false
23239 6937 Acked Flush 0 Done Sent Done false
23240 6938 Acked Read 5 Done Sent Done false
23241 6939 Acked Write 3 Done Sent Done false
23242 6940 Acked Write 7 Done Sent Done false
23243 6941 Acked Read 1 Done Sent Done false
23244 6942 Acked Write 7 Done Sent Done false
23245 6943 Acked Read 5 Done Sent Done false
23246 6944 Acked Write 1 Done Sent Done false
23247 6945 Acked Write 1 Done Sent Done false
23248 6946 Acked Read 9 Done Sent Done false
23249 6947 Acked Write 6 Done Sent Done false
23250 6948 Acked Flush 0 Done Sent Done false
23251 6949 Acked Write 4 Done Sent Done false
23252 6950 Acked Write 2 Done Sent Done false
23253 6951 Acked Read 2 Done Sent Done false
23254 6952 Acked Flush 0 Done Sent Done false
23255 6953 Acked Write 9 Done Sent Done false
23256 6954 Acked Write 5 Done Sent Done false
23257 6955 Acked Write 7 Done Sent Done false
23258 6956 Acked Flush 0 Done Sent Done false
23259 6957 Acked Flush 0 Done Sent Done false
23260 6958 Acked Read 8 Done Sent Done false
23261 6959 Acked Read 4 Done Sent Done false
23262 6960 Acked Flush 0 Done Sent Done false
23263 6961 Acked Read 4 Done Sent Done false
23264 6962 Acked Read 7 Done Sent Done false
23265 6963 Acked Read 10 Done Sent Done false
23266 6964 Acked Read 2 Done Sent Done false
23267 6965 Acked Flush 0 Done Sent Done false
23268 6966 Acked Read 6 Done Sent Done false
23269 6967 Acked Read 1 Done Sent Done false
23270 6968 Acked Read 10 Done Sent Done false
23271 6969 Acked Read 5 Done Sent Done false
23272 6970 Acked Read 2 Done Sent Done false
23273 6971 Acked Read 4 Done Sent Done false
23274 6972 Acked Write 9 Done Sent Done false
23275 6973 Acked Read 3 Done Sent Done false
23276 6974 Acked Write 6 Done Sent Done false
23277 6975 Acked Write 5 Done Sent Done false
23278 6976 Acked Read 7 Done Sent Done false
23279 6977 Acked Flush 0 Done Sent Done false
23280 6978 Acked Read 2 Done Sent Done false
23281 6979 Acked Write 5 Done Sent Done false
23282 6980 Acked Read 1 Done Sent Done false
23283 6981 Acked Write 4 Done Sent Done false
23284 6982 Acked Read 7 Done Sent Done false
23285 6983 Acked Read 1 Done Sent Done false
23286 6984 Acked Read 1 Done Sent Done false
23287 6985 Acked Read 8 Done Sent Done false
23288 6986 Acked Write 1 Done Sent Done false
23289 6987 Acked Write 3 Done Sent Done false
23290 6988 Acked Write 3 Done Sent Done false
23291 6989 Acked Read 8 Done Sent Done false
23292 6990 Acked Write 2 Done Sent Done false
23293 6991 Acked Read 3 Done Sent Done false
23294 6992 Acked Read 7 Done Sent Done false
23295 6993 Acked Write 7 Done Sent Done false
23296 6994 Acked Write 8 Done Sent Done false
23297 6995 Acked Read 9 Done Sent Done false
23298 6996 Acked Write 6 Done Sent Done false
23299 6997 Acked Write 8 Done Sent Done false
23300 6998 Acked Read 6 Done Sent Done false
23301 6999 Acked Read 3 Done Sent Done false
23302 7000 Acked Read 4 Done Sent Done false
23303 7001 Acked Read 1 Done Sent Done false
23304 7002 Acked Read 10 Done Sent Done false
23305 7003 Acked Flush 0 Done Sent Done false
23306 7004 Acked Read 7 Done Sent Done false
23307 7005 Acked Read 7 Done Sent Done false
23308 7006 Acked Read 7 Done Sent Done false
23309 7007 Acked Write 7 Done Sent Done false
23310 7008 Acked Write 1 Done Sent Done false
23311 7009 Acked Write 10 Done Sent Done false
23312 7010 Acked Flush 0 Done Sent Done false
23313 7011 Acked Read 1 Done Sent Done false
23314 7012 Acked Write 6 Done Sent Done false
23315 7013 Acked Read 4 Done Sent Done false
23316 7014 Acked Flush 0 Done Sent Done false
23317 7015 Acked Read 5 Done Sent Done false
23318 7016 Acked Read 4 Done Sent Done false
23319 7017 Acked Read 2 Done Sent Done false
23320 7018 Acked Read 7 Done Sent Done false
23321 7019 Acked Write 8 Done Sent Done false
23322 7020 Acked Write 6 Done Sent Done false
23323 7021 Acked Flush 0 Done Sent Done false
23324 7022 Acked Read 3 Done Sent Done false
23325 7023 Acked Read 8 Done Sent Done false
23326 7024 Acked Read 3 Done Sent Done false
23327 7025 Acked Read 9 Done Sent Done false
23328 7026 Acked Flush 0 Done Sent Done false
23329 7027 Acked Write 2 Done Sent Done false
23330 7028 Acked Read 6 Done Sent Done false
23331 7029 Acked Read 7 Done Sent Done false
23332 7030 Acked Read 7 Done Sent Done false
23333 7031 Acked Write 1 Done Sent Done false
23334 7032 Acked Read 5 Done Sent Done false
23335 7033 Acked Write 1 Done Sent Done false
23336 7034 Acked Read 1 Done Sent Done false
23337 7035 Acked Write 5 Done Sent Done false
23338 7036 Acked Read 3 Done Sent Done false
23339 STATES DS:0 DS:1 DS:2 TOTAL
23340 Sent 0 301 0 301
23341 Done 301 0 301 602
23342 Skipped 0 0 0 0
23343 Error 0 0 0 0
23344 Last Flush: Some(JobId(7026)) Some(JobId(6735)) Some(JobId(7026))
23345 Downstairs last five completed: 6735 6734 6733 6732 6731
23346 Upstairs last five completed: 7036 7035 7034 7033 7032
23347 CLIENT: Up:0 ds:301 act:2
233482025-03-17T22:21:25.294ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
23349 [8820] got start action
23350 [8820] I am going to restart
23351 Starting downstairs at port 8820
23352 Make output file at "/tmp/dsc/downstairs-8820.txt"
23353 [8820][1] reports Starting
23354 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1226)
23355 [8820][1] reports Running
233562025-03-17T22:21:26.086ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
233572025-03-17T22:21:27.087ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
233582025-03-17T22:21:28.089ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
23359 ----------------------------------------------------------------
23360 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23361 States: Active Offline Active
23362 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23363 6736 Acked Read 8 Done Sent Done false
23364 6737 Acked Read 3 Done Sent Done false
23365 6738 Acked Write 10 Done Sent Done false
23366 6739 Acked Write 7 Done Sent Done false
23367 6740 Acked Read 1 Done Sent Done false
23368 6741 Acked Write 3 Done Sent Done false
23369 6742 Acked Flush 0 Done Sent Done false
23370 6743 Acked Read 10 Done Sent Done false
23371 6744 Acked Write 9 Done Sent Done false
23372 6745 Acked Write 10 Done Sent Done false
23373 6746 Acked Flush 0 Done Sent Done false
23374 6747 Acked Read 3 Done Sent Done false
23375 6748 Acked Read 7 Done Sent Done false
23376 6749 Acked Read 4 Done Sent Done false
23377 6750 Acked Read 4 Done Sent Done false
23378 6751 Acked Read 8 Done Sent Done false
23379 6752 Acked Write 6 Done Sent Done false
23380 6753 Acked Read 3 Done Sent Done false
23381 6754 Acked Write 1 Done Sent Done false
23382 6755 Acked Write 2 Done Sent Done false
23383 6756 Acked Write 2 Done Sent Done false
23384 6757 Acked Flush 0 Done Sent Done false
23385 6758 Acked Read 10 Done Sent Done false
23386 6759 Acked Read 4 Done Sent Done false
23387 6760 Acked Write 2 Done Sent Done false
23388 6761 Acked Flush 0 Done Sent Done false
23389 6762 Acked Write 1 Done Sent Done false
23390 6763 Acked Flush 0 Done Sent Done false
23391 6764 Acked Read 5 Done Sent Done false
23392 6765 Acked Write 4 Done Sent Done false
23393 6766 Acked Read 9 Done Sent Done false
23394 6767 Acked Flush 0 Done Sent Done false
23395 6768 Acked Read 8 Done Sent Done false
23396 6769 Acked Write 4 Done Sent Done false
23397 6770 Acked Write 3 Done Sent Done false
23398 6771 Acked Read 9 Done Sent Done false
23399 6772 Acked Read 1 Done Sent Done false
23400 6773 Acked Write 3 Done Sent Done false
23401 6774 Acked Flush 0 Done Sent Done false
23402 6775 Acked Read 4 Done Sent Done false
23403 6776 Acked Read 2 Done Sent Done false
23404 6777 Acked Read 3 Done Sent Done false
23405 6778 Acked Write 10 Done Sent Done false
23406 6779 Acked Read 10 Done Sent Done false
23407 6780 Acked Read 5 Done Sent Done false
23408 6781 Acked Write 3 Done Sent Done false
23409 6782 Acked Write 10 Done Sent Done false
23410 6783 Acked Read 1 Done Sent Done false
23411 6784 Acked Flush 0 Done Sent Done false
23412 6785 Acked Write 7 Done Sent Done false
23413 6786 Acked Read 2 Done Sent Done false
23414 6787 Acked Write 10 Done Sent Done false
23415 6788 Acked Read 8 Done Sent Done false
23416 6789 Acked Write 2 Done Sent Done false
23417 6790 Acked Write 10 Done Sent Done false
23418 6791 Acked Write 1 Done Sent Done false
23419 6792 Acked Write 6 Done Sent Done false
23420 6793 Acked Read 6 Done Sent Done false
23421 6794 Acked Write 6 Done Sent Done false
23422 6795 Acked Flush 0 Done Sent Done false
23423 6796 Acked Read 7 Done Sent Done false
23424 6797 Acked Read 4 Done Sent Done false
23425 6798 Acked Write 4 Done Sent Done false
23426 6799 Acked Write 9 Done Sent Done false
23427 6800 Acked Read 6 Done Sent Done false
23428 6801 Acked Write 5 Done Sent Done false
23429 6802 Acked Write 6 Done Sent Done false
23430 6803 Acked Write 1 Done Sent Done false
23431 6804 Acked Read 2 Done Sent Done false
23432 6805 Acked Write 1 Done Sent Done false
23433 6806 Acked Read 10 Done Sent Done false
23434 6807 Acked Write 3 Done Sent Done false
23435 6808 Acked Write 4 Done Sent Done false
23436 6809 Acked Flush 0 Done Sent Done false
23437 6810 Acked Read 9 Done Sent Done false
23438 6811 Acked Read 6 Done Sent Done false
23439 6812 Acked Read 3 Done Sent Done false
23440 6813 Acked Read 2 Done Sent Done false
23441 6814 Acked Read 7 Done Sent Done false
23442 6815 Acked Read 1 Done Sent Done false
23443 6816 Acked Read 3 Done Sent Done false
23444 6817 Acked Read 6 Done Sent Done false
23445 6818 Acked Write 10 Done Sent Done false
23446 6819 Acked Write 8 Done Sent Done false
23447 6820 Acked Write 7 Done Sent Done false
23448 6821 Acked Read 7 Done Sent Done false
23449 6822 Acked Read 3 Done Sent Done false
23450 6823 Acked Read 8 Done Sent Done false
23451 6824 Acked Read 1 Done Sent Done false
23452 6825 Acked Write 2 Done Sent Done false
23453 6826 Acked Read 7 Done Sent Done false
23454 6827 Acked Read 2 Done Sent Done false
23455 6828 Acked Write 9 Done Sent Done false
23456 6829 Acked Write 1 Done Sent Done false
23457 6830 Acked Write 4 Done Sent Done false
23458 6831 Acked Write 4 Done Sent Done false
23459 6832 Acked Read 2 Done Sent Done false
23460 6833 Acked Read 7 Done Sent Done false
23461 6834 Acked Read 7 Done Sent Done false
23462 6835 Acked Flush 0 Done Sent Done false
23463 6836 Acked Write 3 Done Sent Done false
23464 6837 Acked Read 5 Done Sent Done false
23465 6838 Acked Read 3 Done Sent Done false
23466 6839 Acked Flush 0 Done Sent Done false
23467 6840 Acked Write 8 Done Sent Done false
23468 6841 Acked Read 1 Done Sent Done false
23469 6842 Acked Write 9 Done Sent Done false
23470 6843 Acked Read 9 Done Sent Done false
23471 6844 Acked Write 1 Done Sent Done false
23472 6845 Acked Flush 0 Done Sent Done false
23473 6846 Acked Read 2 Done Sent Done false
23474 6847 Acked Write 1 Done Sent Done false
23475 6848 Acked Flush 0 Done Sent Done false
23476 6849 Acked Read 1 Done Sent Done false
23477 6850 Acked Write 4 Done Sent Done false
23478 6851 Acked Read 7 Done Sent Done false
23479 6852 Acked Read 3 Done Sent Done false
23480 6853 Acked Write 8 Done Sent Done false
23481 6854 Acked Write 10 Done Sent Done false
23482 6855 Acked Write 10 Done Sent Done false
23483 6856 Acked Read 1 Done Sent Done false
23484 6857 Acked Write 6 Done Sent Done false
23485 6858 Acked Write 9 Done Sent Done false
23486 6859 Acked Read 2 Done Sent Done false
23487 6860 Acked Write 7 Done Sent Done false
23488 6861 Acked Write 7 Done Sent Done false
23489 6862 Acked Read 3 Done Sent Done false
23490 6863 Acked Flush 0 Done Sent Done false
23491 6864 Acked Read 9 Done Sent Done false
23492 6865 Acked Write 3 Done Sent Done false
23493 6866 Acked Read 8 Done Sent Done false
23494 6867 Acked Write 5 Done Sent Done false
23495 6868 Acked Write 3 Done Sent Done false
23496 6869 Acked Read 9 Done Sent Done false
23497 6870 Acked Read 8 Done Sent Done false
23498 6871 Acked Write 4 Done Sent Done false
23499 6872 Acked Read 8 Done Sent Done false
23500 6873 Acked Read 2 Done Sent Done false
23501 6874 Acked Flush 0 Done Sent Done false
23502 6875 Acked Read 10 Done Sent Done false
23503 6876 Acked Read 10 Done Sent Done false
23504 6877 Acked Read 6 Done Sent Done false
23505 6878 Acked Write 8 Done Sent Done false
23506 6879 Acked Read 9 Done Sent Done false
23507 6880 Acked Write 1 Done Sent Done false
23508 6881 Acked Flush 0 Done Sent Done false
23509 6882 Acked Flush 0 Done Sent Done false
23510 6883 Acked Read 4 Done Sent Done false
23511 6884 Acked Flush 0 Done Sent Done false
23512 6885 Acked Read 1 Done Sent Done false
23513 6886 Acked Read 4 Done Sent Done false
23514 6887 Acked Read 2 Done Sent Done false
23515 6888 Acked Read 8 Done Sent Done false
23516 6889 Acked Read 2 Done Sent Done false
23517 6890 Acked Flush 0 Done Sent Done false
23518 6891 Acked Read 7 Done Sent Done false
23519 6892 Acked Read 10 Done Sent Done false
23520 6893 Acked Write 10 Done Sent Done false
23521 6894 Acked Read 5 Done Sent Done false
23522 6895 Acked Flush 0 Done Sent Done false
23523 6896 Acked Flush 0 Done Sent Done false
23524 6897 Acked Read 1 Done Sent Done false
23525 6898 Acked Write 8 Done Sent Done false
23526 6899 Acked Read 10 Done Sent Done false
23527 6900 Acked Flush 0 Done Sent Done false
23528 6901 Acked Write 7 Done Sent Done false
23529 6902 Acked Read 8 Done Sent Done false
23530 6903 Acked Read 10 Done Sent Done false
23531 6904 Acked Write 3 Done Sent Done false
23532 6905 Acked Write 5 Done Sent Done false
23533 6906 Acked Read 9 Done Sent Done false
23534 6907 Acked Write 2 Done Sent Done false
23535 6908 Acked Read 4 Done Sent Done false
23536 6909 Acked Read 10 Done Sent Done false
23537 6910 Acked Write 8 Done Sent Done false
23538 6911 Acked Write 2 Done Sent Done false
23539 6912 Acked Write 5 Done Sent Done false
23540 6913 Acked Write 7 Done Sent Done false
23541 6914 Acked Write 1 Done Sent Done false
23542 6915 Acked Read 5 Done Sent Done false
23543 6916 Acked Write 1 Done Sent Done false
23544 6917 Acked Write 2 Done Sent Done false
23545 6918 Acked Write 10 Done Sent Done false
23546 6919 Acked Write 6 Done Sent Done false
23547 6920 Acked Write 1 Done Sent Done false
23548 6921 Acked Write 7 Done Sent Done false
23549 6922 Acked Read 1 Done Sent Done false
23550 6923 Acked Read 4 Done Sent Done false
23551 6924 Acked Flush 0 Done Sent Done false
23552 6925 Acked Read 1 Done Sent Done false
23553 6926 Acked Read 4 Done Sent Done false
23554 6927 Acked Flush 0 Done Sent Done false
23555 6928 Acked Write 7 Done Sent Done false
23556 6929 Acked Write 10 Done Sent Done false
23557 6930 Acked Write 5 Done Sent Done false
23558 6931 Acked Write 8 Done Sent Done false
23559 6932 Acked Read 4 Done Sent Done false
23560 6933 Acked Write 4 Done Sent Done false
23561 6934 Acked Write 6 Done Sent Done false
23562 6935 Acked Write 3 Done Sent Done false
23563 6936 Acked Read 5 Done Sent Done false
23564 6937 Acked Flush 0 Done Sent Done false
23565 6938 Acked Read 5 Done Sent Done false
23566 6939 Acked Write 3 Done Sent Done false
23567 6940 Acked Write 7 Done Sent Done false
23568 6941 Acked Read 1 Done Sent Done false
23569 6942 Acked Write 7 Done Sent Done false
23570 6943 Acked Read 5 Done Sent Done false
23571 6944 Acked Write 1 Done Sent Done false
23572 6945 Acked Write 1 Done Sent Done false
23573 6946 Acked Read 9 Done Sent Done false
23574 6947 Acked Write 6 Done Sent Done false
23575 6948 Acked Flush 0 Done Sent Done false
23576 6949 Acked Write 4 Done Sent Done false
23577 6950 Acked Write 2 Done Sent Done false
23578 6951 Acked Read 2 Done Sent Done false
23579 6952 Acked Flush 0 Done Sent Done false
23580 6953 Acked Write 9 Done Sent Done false
23581 6954 Acked Write 5 Done Sent Done false
23582 6955 Acked Write 7 Done Sent Done false
23583 6956 Acked Flush 0 Done Sent Done false
23584 6957 Acked Flush 0 Done Sent Done false
23585 6958 Acked Read 8 Done Sent Done false
23586 6959 Acked Read 4 Done Sent Done false
23587 6960 Acked Flush 0 Done Sent Done false
23588 6961 Acked Read 4 Done Sent Done false
23589 6962 Acked Read 7 Done Sent Done false
23590 6963 Acked Read 10 Done Sent Done false
23591 6964 Acked Read 2 Done Sent Done false
23592 6965 Acked Flush 0 Done Sent Done false
23593 6966 Acked Read 6 Done Sent Done false
23594 6967 Acked Read 1 Done Sent Done false
23595 6968 Acked Read 10 Done Sent Done false
23596 6969 Acked Read 5 Done Sent Done false
23597 6970 Acked Read 2 Done Sent Done false
23598 6971 Acked Read 4 Done Sent Done false
23599 6972 Acked Write 9 Done Sent Done false
23600 6973 Acked Read 3 Done Sent Done false
23601 6974 Acked Write 6 Done Sent Done false
23602 6975 Acked Write 5 Done Sent Done false
23603 6976 Acked Read 7 Done Sent Done false
23604 6977 Acked Flush 0 Done Sent Done false
23605 6978 Acked Read 2 Done Sent Done false
23606 6979 Acked Write 5 Done Sent Done false
23607 6980 Acked Read 1 Done Sent Done false
23608 6981 Acked Write 4 Done Sent Done false
23609 6982 Acked Read 7 Done Sent Done false
23610 6983 Acked Read 1 Done Sent Done false
23611 6984 Acked Read 1 Done Sent Done false
23612 6985 Acked Read 8 Done Sent Done false
23613 6986 Acked Write 1 Done Sent Done false
23614 6987 Acked Write 3 Done Sent Done false
23615 6988 Acked Write 3 Done Sent Done false
23616 6989 Acked Read 8 Done Sent Done false
23617 6990 Acked Write 2 Done Sent Done false
23618 6991 Acked Read 3 Done Sent Done false
23619 6992 Acked Read 7 Done Sent Done false
23620 6993 Acked Write 7 Done Sent Done false
23621 6994 Acked Write 8 Done Sent Done false
23622 6995 Acked Read 9 Done Sent Done false
23623 6996 Acked Write 6 Done Sent Done false
23624 6997 Acked Write 8 Done Sent Done false
23625 6998 Acked Read 6 Done Sent Done false
23626 6999 Acked Read 3 Done Sent Done false
23627 7000 Acked Read 4 Done Sent Done false
23628 7001 Acked Read 1 Done Sent Done false
23629 7002 Acked Read 10 Done Sent Done false
23630 7003 Acked Flush 0 Done Sent Done false
23631 7004 Acked Read 7 Done Sent Done false
23632 7005 Acked Read 7 Done Sent Done false
23633 7006 Acked Read 7 Done Sent Done false
23634 7007 Acked Write 7 Done Sent Done false
23635 7008 Acked Write 1 Done Sent Done false
23636 7009 Acked Write 10 Done Sent Done false
23637 7010 Acked Flush 0 Done Sent Done false
23638 7011 Acked Read 1 Done Sent Done false
23639 7012 Acked Write 6 Done Sent Done false
23640 7013 Acked Read 4 Done Sent Done false
23641 7014 Acked Flush 0 Done Sent Done false
23642 7015 Acked Read 5 Done Sent Done false
23643 7016 Acked Read 4 Done Sent Done false
23644 7017 Acked Read 2 Done Sent Done false
23645 7018 Acked Read 7 Done Sent Done false
23646 7019 Acked Write 8 Done Sent Done false
23647 7020 Acked Write 6 Done Sent Done false
23648 7021 Acked Flush 0 Done Sent Done false
23649 7022 Acked Read 3 Done Sent Done false
23650 7023 Acked Read 8 Done Sent Done false
23651 7024 Acked Read 3 Done Sent Done false
23652 7025 Acked Read 9 Done Sent Done false
23653 7026 Acked Flush 0 Done Sent Done false
23654 7027 Acked Write 2 Done Sent Done false
23655 7028 Acked Read 6 Done Sent Done false
23656 7029 Acked Read 7 Done Sent Done false
23657 7030 Acked Read 7 Done Sent Done false
23658 7031 Acked Write 1 Done Sent Done false
23659 7032 Acked Read 5 Done Sent Done false
23660 7033 Acked Write 1 Done Sent Done false
23661 7034 Acked Read 1 Done Sent Done false
23662 7035 Acked Write 5 Done Sent Done false
23663 7036 Acked Read 3 Done Sent Done false
23664 STATES DS:0 DS:1 DS:2 TOTAL
23665 Sent 0 301 0 301
23666 Done 301 0 301 602
23667 Skipped 0 0 0 0
23668 Error 0 0 0 0
23669 Last Flush: Some(JobId(7026)) Some(JobId(6735)) Some(JobId(7026))
23670 Downstairs last five completed: 6735 6734 6733 6732 6731
23671 Upstairs last five completed: 7036 7035 7034 7033 7032
23672 CLIENT: Up:0 ds:301 act:2
236732025-03-17T22:21:29.091ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
236742025-03-17T22:21:30.295ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
236752025-03-17T22:21:31.094ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
236762025-03-17T22:21:32.049ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
236772025-03-17T22:21:32.049ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:57677) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
236782025-03-17T22:21:32.050ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
236792025-03-17T22:21:32.050ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
236802025-03-17T22:21:32.050ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
236812025-03-17T22:21:32.050ZINFOcrucible: send last flush ID to this DS: Some(JobId(6735)) = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
236822025-03-17T22:21:32.050ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
236832025-03-17T22:21:32.050ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(6735)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
236842025-03-17T22:21:32.051ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(6735)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
23685 ----------------------------------------------------------------
23686 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:10
23687 States: Active Active Active
23688 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
23689 7027 Acked Write 2 Done Done Done true
23690 7028 Acked Read 6 Done Done Done true
23691 7029 Acked Read 7 Done Done Done true
23692 7030 Acked Read 7 Done Done Done true
23693 7031 Acked Write 1 Done Done Done true
23694 7032 Acked Read 5 Done Done Done true
23695 7033 Acked Write 1 Done Done Done true
23696 7034 Acked Read 1 Done Done Done true
23697 7035 Acked Write 5 Done Done Done true
23698 7036 Acked Read 3 Done Done Done true
23699 STATES DS:0 DS:1 DS:2 TOTAL
23700 Sent 0 0 0 0
23701 Done 10 10 10 30
23702 Skipped 0 0 0 0
23703 Error 0 0 0 0
23704 Last Flush: Some(JobId(7026)) Some(JobId(7026)) Some(JobId(7026))
23705 Downstairs last five completed: 7026 7025 7024 7023 7022
23706 Upstairs last five completed: 7036 7035 7034 7033 7032
23707 CLIENT: Up:0 ds:10 act:3
23708 ----------------------------------------------------------------
23709 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
23710 Downstairs last five completed: 7037 7036 7035 7034 7033
23711 Upstairs last five completed: 7037 7036 7035 7034 7033
23712 CLIENT: Up:0 ds:0 act:3
23713 Replay: All jobs finished
23714 Main task has work to do, go find it
23715 got dsc Stop(2)
23716 stop 2
23717 Mar 17 22:21:36.299 INFO request completed[8830] Got stop action so:false kr:false
23718 , latency_us: 351, response_code: 204, uri: /stop/cid/2, method: POST, req_id: d16da3e6-f08e-4a1b-8ee0-c07724961dab, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
23719 Mar 17 22:21:36.299 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:21:36.299848903Z","hostname":"w-01JPK2S810SFAA9787XEG8GFX5","pid":1206,"":"io task","client":"2","":"downstairs","session_id":"0f8310d7-099e-4ead-8484-91ddc162d307"}
23720 INFO request completed, latency_us: 117, response_code: 200, uri: /state/cid/2, method: GET, req_id: 739fdc7f-ff11-4fe8-a7b1-897c2b93abf4, remote_addr: 127.0.0.1{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:21:36.299916697Z","hostname":"w-01JPK2S810SFAA9787XEG8GFX5","pid":1206,"":"io task","client":"2","":"downstairs","session_id":"0f8310d7-099e-4ead-8484-91ddc162d307"}
23721 :60423, local_addr: 127.0.0.1:9998
237222025-03-17T22:21:36.299ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
237232025-03-17T22:21:36.300ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
23724 [8830] Exited with: ExitStatus(unix_wait_status(9))
23725 [8830][2] reports Exit
23726 Mar 17 22:21:40.301 INFO request completed, latency_us: 218, response_code: 200, uri: /state/cid/2, method: GET, req_id: 3bece39a-b000-44c5-93e1-a2d433533938, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
23727 001/300 Read block 3644 len 24576
23728 002/300 Read block 4513 len 40960
23729 003/300 Write block 3993 len 24576 data: 2 2 2 2 5 6
23730 004/300 Write block 2302 len 20480 data: 5 4 4 4 3
23731 005/300 Read block 2057 len 4096
23732 006/300 Flush
23733 007/300 Read block 292 len 16384
23734 008/300 Read block 2000 len 12288
23735 009/300 Read block 1021 len 12288
23736 010/300 Write block 2708 len 12288 data: 4 3 2
23737 011/300 Read block 742 len 16384
23738 012/300 Read block 4205 len 20480
23739 013/300 Read block 3032 len 12288
23740 014/300 Write block 2977 len 32768 data: 5 5 4 4 4 4 4 4
23741 015/300 Flush
23742 016/300 Read block 3165 len 40960
23743 017/300 Read block 2242 len 16384
23744 018/300 Write block 230 len 36864 data: 3 3 4 6 6 6 6 7 6
23745 019/300 Read block 1707 len 32768
23746 020/300 Read block 3975 len 24576
23747 021/300 Write block 1060 len 32768 data: 4 5 4 5 5 4 6 5
23748 022/300 Read block 2109 len 16384
23749 023/300 Write block 3560 len 32768 data: 5 5 5 5 6 6 6 3
23750 024/300 Write block 4381 len 40960 data: 3 3 4 4 3 3 3 3 3 2
23751 025/300 Read block 4611 len 16384
23752 026/300 Read block 1867 len 20480
23753 027/300 Write block 69 len 36864 data: 4 3 4 4 4 5 6 7 6
23754 028/300 Write block 2000 len 36864 data: 5 5 5 4 3 3 3 4 5
23755 029/300 Write block 3168 len 4096 data: 7
23756 030/300 Write block 1981 len 12288 data: 4 3 3
23757 031/300 Read block 672 len 20480
23758 032/300 Read block 3040 len 28672
23759 033/300 Read block 3058 len 16384
23760 034/300 Write block 4137 len 24576 data: 5 4 4 5 5 5
23761 035/300 Read block 1493 len 24576
23762 036/300 Read block 2102 len 28672
23763 037/300 Read block 1508 len 4096
23764 038/300 Read block 2077 len 12288
23765 039/300 Read block 3226 len 12288
23766 040/300 Write block 91 len 8192 data: 5 4
23767 041/300 Read block 4972 len 20480
23768 042/300 Read block 379 len 20480
23769 043/300 Read block 7 len 24576
23770 044/300 Write block 3006 len 24576 data: 6 6 4 4 5 5
23771 045/300 Write block 3582 len 32768 data: 4 4 4 4 4 4 4 4
23772 046/300 Write block 553 len 40960 data: 5 5 6 7 9 9 9 9 9 9
23773 047/300 Read block 1549 len 36864
23774 048/300 Read block 4221 len 20480
23775 049/300 Write block 3365 len 40960 data: 4 4 4 4 4 4 4 5 5 3
23776 050/300 Flush
23777 051/300 Write block 3551 len 8192 data: 5 5
23778 052/300 Read block 575 len 20480
23779 053/300 Read block 3872 len 40960
23780 054/300 Write block 3095 len 40960 data: 4 4 6 6 5 5 5 6 5 6
23781 055/300 Write block 2858 len 4096 data: 4
23782 056/300 Write block 740 len 20480 data: 5 3 3 3 4
23783 057/300 Write block 3711 len 40960 data: 3 3 3 3 4 3 3 3 3 3
23784 058/300 Read block 610 len 40960
23785 059/300 Read block 699 len 16384
23786 060/300 Read block 4364 len 28672
23787 061/300 Write block 4467 len 8192 data: 7 9
23788 062/300 Read block 3715 len 40960
23789 063/300 Read block 4024 len 40960
23790 064/300 Write block 13 len 4096 data: 7
23791 065/300 Write block 3630 len 8192 data: 3 4
23792 066/300 Write block 721 len 20480 data: 6 5 4 4 4
23793 067/300 Write block 3934 len 36864 data: 3 3 2 2 2 3 4 5 4
23794 068/300 Read block 1690 len 24576
23795 069/300 Read block 4006 len 20480
23796 070/300 Read block 3157 len 24576
23797 071/300 Read block 4004 len 8192
23798 072/300 Write block 1318 len 28672 data: 2 2 2 3 2 4 4
23799 073/300 Write block 156 len 20480 data: 3 4 5 4 6
23800 074/300 Write block 392 len 4096 data: 3
23801 075/300 Write block 2133 len 20480 data: 7 7 7 7 7
23802 076/300 Write block 3557 len 12288 data: 5 5 5
23803 077/300 Write block 4647 len 24576 data: 4 3 3 3 5 4
23804 078/300 Read block 176 len 8192
23805 079/300 Read block 630 len 36864
23806 080/300 Write block 3529 len 8192 data: 4 5
23807 081/300 Read block 1804 len 40960
23808 082/300 Write block 4117 len 28672 data: 4 4 3 3 3 5 4
23809 083/300 Write block 4896 len 32768 data: 4 4 5 7 8 7 8 7
23810 084/300 Read block 560 len 40960
23811 085/300 Write block 4749 len 40960 data: 7 6 6 6 5 4 4 5 5 5
23812 086/300 Read block 4610 len 24576
23813 087/300 Read block 3438 len 32768
23814 088/300 Read block 3038 len 28672
23815 089/300 Write block 1185 len 12288 data: 6 4 4
23816 090/300 Write block 412 len 24576 data: 4 4 4 6 6 7
23817 091/300 Write block 4233 len 36864 data: 5 5 6 4 3 3 6 5 6
23818 092/300 Write block 2031 len 4096 data: 3
23819 093/300 Write block 1371 len 4096 data: 4
23820 094/300 Write block 3133 len 20480 data: 7 7 7 6 4
23821 095/300 Read block 3667 len 28672
23822 096/300 Read block 1425 len 40960
23823 097/300 Write block 931 len 8192 data: 7 6
23824 098/300 Write block 4558 len 16384 data: 3 3 2 2
23825 099/300 Read block 4178 len 24576
23826 100/300 Write block 1510 len 28672 data: 3 4 4 4 3 3 3
23827 101/300 Write block 2977 len 40960 data: 6 6 5 5 5 5 5 5 3 4
23828 102/300 Write block 3941 len 8192 data: 6 5
23829 103/300 Write block 352 len 4096 data: 5
23830 104/300 Read block 1921 len 28672
23831 105/300 Flush
23832 106/300 Write block 3273 len 12288 data: 6 6 7
23833 107/300 Read block 10 len 16384
23834 108/300 Read block 3762 len 4096
23835 109/300 Read block 3902 len 4096
23836 110/300 Write block 1413 len 16384 data: 7 6 5 5
23837 111/300 Read block 2004 len 8192
23838 112/300 Write block 888 len 4096 data: 6
23839 113/300 Write block 4358 len 24576 data: 2 2 3 3 3 3
23840 114/300 Write block 4604 len 28672 data: 7 7 10 8 9 8 8
23841 115/300 Flush
23842 116/300 Read block 1660 len 28672
23843 117/300 Read block 3972 len 4096
23844 118/300 Write block 156 len 12288 data: 4 5 6
23845 119/300 Read block 2990 len 24576
23846 120/300 Read block 2157 len 24576
23847 121/300 Read block 504 len 8192
23848 122/300 Write block 4569 len 8192 data: 6 5
23849 123/300 Write block 68 len 12288 data: 5 5 4
23850 124/300 Write block 3887 len 24576 data: 7 6 5 5 4 5
23851 125/300 Write block 1404 len 24576 data: 5 5 6 5 4 4
23852 126/300 Flush
23853 127/300 Read block 3388 len 32768
23854 128/300 Read block 2123 len 20480
23855 129/300 Write block 2301 len 8192 data: 6 6
23856 130/300 Read block 290 len 36864
23857 131/300 Write block 4680 len 36864 data: 4 5 5 5 7 7 6 6 7
23858 132/300 Write block 3401 len 16384 data: 3 6 8 7
23859 133/300 Read block 3920 len 36864
23860 134/300 Read block 4278 len 32768
23861 135/300 Read block 2197 len 20480
23862 136/300 Read block 1103 len 36864
23863 137/300 Read block 1854 len 24576
23864 138/300 Read block 4794 len 40960
23865 139/300 Read block 350 len 8192
23866 140/300 Write block 2530 len 40960 data: 5 6 5 5 6 5 4 3 4 3
23867 141/300 Write block 963 len 36864 data: 3 3 2 2 2 2 2 3 3
23868 142/300 Read block 4787 len 40960
23869 143/300 Read block 1113 len 12288
23870 144/300 Read block 1285 len 4096
23871 145/300 Write block 2849 len 20480 data: 7 7 6 6 4
23872 146/300 Write block 1140 len 36864 data: 5 4 4 3 3 3 3 3 3
23873 147/300 Read block 4877 len 28672
23874 148/300 Read block 2417 len 24576
23875 149/300 Read block 4632 len 24576
23876 150/300 Read block 845 len 24576
23877 151/300 Write block 2940 len 40960 data: 7 8 7 8 7 8 10 7 6 3
23878 152/300 Read block 3947 len 16384
23879 153/300 Flush
23880 154/300 Write block 531 len 20480 data: 3 2 2 3 4
23881 155/300 Read block 2373 len 32768
23882 156/300 Write block 1118 len 20480 data: 4 4 3 3 2
23883 157/300 Read block 4456 len 4096
23884 158/300 Write block 3647 len 32768 data: 4 4 5 6 5 5 7 5
23885 159/300 Write block 2357 len 12288 data: 4 6 6
23886 160/300 Read block 272 len 28672
23887 161/300 Write block 2351 len 12288 data: 2 3 3
23888 162/300 Read block 746 len 36864
23889 163/300 Write block 3761 len 12288 data: 4 4 2
23890 164/300 Write block 4295 len 36864 data: 2 3 5 6 7 7 9 7 6
23891 165/300 Read block 2555 len 12288
23892 166/300 Read block 3410 len 36864
23893 167/300 Read block 3765 len 16384
23894 168/300 Write block 4598 len 28672 data: 2 2 2 2 5 5 8
23895 169/300 Write block 4705 len 4096 data: 4
23896 170/300 Write block 1150 len 36864 data: 5 5 6 6 5 4 6 6 5
23897 171/300 Read block 1453 len 32768
23898 172/300 Write block 2201 len 36864 data: 7 7 8 6 5 7 8 7 6
23899 173/300 Write block 992 len 28672 data: 5 5 5 4 2 3 3
23900 174/300 Write block 2450 len 36864 data: 4 3 3 3 2 2 2 3 4
23901 175/300 Read block 3855 len 36864
23902 176/300 Write block 447 len 4096 data: 5
23903 177/300 Read block 564 len 28672
23904 178/300 Read block 98 len 16384
23905 179/300 Read block 4667 len 16384
23906 180/300 Write block 4442 len 36864 data: 6 6 6 6 5 5 6 7 7
23907 181/300 Read block 3563 len 16384
23908 182/300 Read block 1883 len 32768
23909 183/300 Read block 4670 len 4096
23910 184/300 Write block 4684 len 28672 data: 8 8 7 7 8 7 7
23911 185/300 Read block 994 len 24576
23912 186/300 Read block 1304 len 36864
23913 187/300 Read block 3204 len 8192
23914 188/300 Read block 3178 len 12288
23915 189/300 Write block 3645 len 12288 data: 6 4 5
23916 190/300 Read block 4979 len 40960
23917 191/300 Read block 2542 len 28672
23918 192/300 Write block 4860 len 32768 data: 9 9 10 10 10 10 9 9
23919 193/300 Write block 1252 len 12288 data: 5 6 7
23920 194/300 Flush
23921 195/300 Flush
23922 196/300 Read block 3366 len 12288
23923 197/300 Write block 4201 len 12288 data: 4 3 3
23924 198/300 Write block 3130 len 16384 data: 8 6 6 8
23925 199/300 Write block 1334 len 8192 data: 4 4
23926 200/300 Write block 483 len 24576 data: 5 7 6 4 4 4
23927 201/300 Read block 2358 len 16384
23928 202/300 Read block 242 len 32768
23929 203/300 Read block 4084 len 20480
23930 204/300 Write block 2567 len 8192 data: 5 5
23931 205/300 Write block 4630 len 24576 data: 5 5 4 6 6 5
23932 206/300 Read block 1777 len 16384
23933 207/300 Write block 4520 len 24576 data: 3 4 4 4 3 3
23934 208/300 Read block 4059 len 36864
23935 209/300 Read block 4074 len 20480
23936 210/300 Read block 499 len 40960
23937 211/300 Write block 1127 len 12288 data: 5 6 4
23938 212/300 Write block 3585 len 16384 data: 5 5 5 5
23939 213/300 Read block 4579 len 24576
23940 214/300 Read block 1750 len 32768
23941 215/300 Write block 3765 len 8192 data: 3 4
23942 216/300 Flush
23943 217/300 Flush
23944 218/300 Read block 357 len 40960
23945 219/300 Read block 4302 len 12288
23946 220/300 Write block 2784 len 28672 data: 6 6 6 5 5 5 5
23947 221/300 Write block 2889 len 8192 data: 6 5
23948 222/300 Flush
23949 223/300 Read block 2166 len 12288
23950 224/300 Write block 2360 len 24576 data: 4 3 3 3 3 3
23951 225/300 Read block 4399 len 8192
23952 226/300 Read block 3912 len 8192
23953 227/300 Read block 990 len 8192
23954 228/300 Flush
23955 229/300 Read block 4135 len 40960
23956 230/300 Read block 1294 len 16384
23957 231/300 Write block 4015 len 36864 data: 3 3 3 3 5 4 4 3 3
23958 232/300 Flush
23959 233/300 Write block 3759 len 36864 data: 3 4 5 5 3 2 4 5 3
23960 234/300 Write block 4135 len 20480 data: 4 3 6 5 5
23961 235/300 Read block 2275 len 40960
23962 236/300 Read block 2355 len 12288
23963 237/300 Read block 4117 len 4096
23964 238/300 Write block 1976 len 36864 data: 4 5 6 4 4 5 4 4 4
23965 239/300 Write block 3827 len 16384 data: 8 9 9 9
23966 240/300 Flush
23967 241/300 Write block 1356 len 12288 data: 7 8 9
23968 242/300 Read block 3593 len 20480
23969 243/300 Write block 612 len 16384 data: 3 5 5 5
23970 244/300 Write block 3412 len 32768 data: 2 2 2 3 5 6 7 5
23971 245/300 Read block 3266 len 8192
23972 246/300 Write block 2988 len 28672 data: 5 5 5 5 5 6 5
23973 247/300 Read block 183 len 32768
23974 248/300 Read block 162 len 4096
23975 249/300 Read block 1240 len 12288
23976 250/300 Read block 1739 len 12288
23977 251/300 Read block 3868 len 36864
23978 252/300 Read block 3583 len 24576
23979 253/300 Read block 3497 len 32768
23980 254/300 Read block 1031 len 20480
23981 255/300 Read block 2716 len 24576
23982 256/300 Write block 1808 len 8192 data: 5 4
23983 257/300 Flush
23984 258/300 Write block 2743 len 32768 data: 3 3 3 2 2 2 2 2
23985 259/300 Read block 3028 len 28672
23986 260/300 Write block 422 len 32768 data: 4 3 4 4 3 3 3 4
23987 261/300 Write block 2105 len 28672 data: 5 7 9 8 8 8 7
23988 262/300 Write block 4500 len 28672 data: 5 4 5 5 3 3 3
23989 263/300 Read block 4223 len 36864
23990 264/300 Flush
23991 265/300 Write block 338 len 32768 data: 5 3 2 3 4 4 3 3
23992 266/300 Read block 3812 len 12288
23993 267/300 Write block 4106 len 36864 data: 6 8 7 6 4 3 4 4 4
23994 268/300 Write block 2214 len 24576 data: 5 6 4 6 6 8
23995 269/300 Read block 4350 len 20480
23996 270/300 Write block 2619 len 4096 data: 5
23997 271/300 Write block 3226 len 12288 data: 6 7 6
23998 272/300 Read block 1854 len 32768
23999 273/300 Flush
24000 274/300 Write block 165 len 32768 data: 6 6 6 6 6 7 7 7
24001 275/300 Read block 2764 len 36864
24002 276/300 Read block 1109 len 36864
24003 277/300 Write block 3761 len 28672 data: 6 6 4 3 5 6 4
24004 278/300 Write block 3487 len 40960 data: 2 2 2 4 6 7 5 6 7 6
24005 279/300 Read block 2633 len 8192
24006 280/300 Write block 4942 len 8192 data: 4 5
24007 281/300 Read block 2130 len 36864
24008 282/300 Write block 1609 len 40960 data: 4 4 5 5 6 6 6 6 6 4
24009 283/300 Write block 1269 len 28672 data: 3 2 2 3 4 6 7
24010 284/300 Write block 185 len 24576 data: 7 6 6 6 5 4
24011 285/300 Read block 4767 len 40960
24012 286/300 Read block 2450 len 36864
24013 287/300 Write block 1470 len 28672 data: 5 4 5 6 5 5 5
24014 288/300 Write block 502 len 8192 data: 7 6
24015 289/300 Flush
24016 290/300 Flush
24017 291/300 Write block 642 len 20480 data: 4 4 4 4 4
24018 292/300 Read block 3046 len 16384
24019 293/300 Read block 2319 len 32768
24020 294/300 Read block 3399 len 12288
24021 295/300 Write block 3675 len 4096 data: 3
24022 296/300 Read block 2616 len 12288
24023 297/300 Write block 137 len 32768 data: 3 4 4 5 4 4 4 5
24024 298/300 Write block 1208 len 32768 data: 3 3 4 4 6 6 6 7
24025 299/300 Read block 271 len 32768
24026 300/300 Read block 1873 len 20480
24027 301/300 Read block 4750 len 36864
24028 Main task has work to do, go find it
24029 got dsc Start(2)
24030 start 2
24031 Mar 17 22:21:40.554 INFO request completed, latency_us: 215, response_code: 204, uri: /start/cid/2, method: POST, req_id: 45cce56a-722d-47f8-ae54-90a87d5182a0, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
24032 Replay: started 2, returned:Ok(())
24033 ----------------------------------------------------------------
24034 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24035 States: Active Active Offline
24036 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24037 7038 Acked Read 6 Done Done Sent false
24038 7039 Acked Read 10 Done Done Sent false
24039 7040 Acked Write 6 Done Done Sent false
24040 7041 Acked Write 5 Done Done Sent false
24041 7042 Acked Read 1 Done Done Sent false
24042 7043 Acked Flush 0 Done Done Sent false
24043 7044 Acked Read 4 Done Done Sent false
24044 7045 Acked Read 3 Done Done Sent false
24045 7046 Acked Read 3 Done Done Sent false
24046 7047 Acked Write 3 Done Done Sent false
24047 7048 Acked Read 4 Done Done Sent false
24048 7049 Acked Read 5 Done Done Sent false
24049 7050 Acked Read 3 Done Done Sent false
24050 7051 Acked Write 8 Done Done Sent false
24051 7052 Acked Flush 0 Done Done Sent false
24052 7053 Acked Read 10 Done Done Sent false
24053 7054 Acked Read 4 Done Done Sent false
24054 7055 Acked Write 9 Done Done Sent false
24055 7056 Acked Read 8 Done Done Sent false
24056 7057 Acked Read 6 Done Done Sent false
24057 7058 Acked Write 8 Done Done Sent false
24058 7059 Acked Read 4 Done Done Sent false
24059 7060 Acked Write 8 Done Done Sent false
24060 7061 Acked Write 10 Done Done Sent false
24061 7062 Acked Read 4 Done Done Sent false
24062 7063 Acked Read 5 Done Done Sent false
24063 7064 Acked Write 9 Done Done Sent false
24064 7065 Acked Write 9 Done Done Sent false
24065 7066 Acked Write 1 Done Done Sent false
24066 7067 Acked Write 3 Done Done Sent false
24067 7068 Acked Read 5 Done Done Sent false
24068 7069 Acked Read 7 Done Done Sent false
24069 7070 Acked Read 4 Done Done Sent false
24070 7071 Acked Write 6 Done Done Sent false
24071 7072 Acked Read 6 Done Done Sent false
24072 7073 Acked Read 7 Done Done Sent false
24073 7074 Acked Read 1 Done Done Sent false
24074 7075 Acked Read 3 Done Done Sent false
24075 7076 Acked Read 3 Done Done Sent false
24076 7077 Acked Write 2 Done Done Sent false
24077 7078 Acked Read 5 Done Done Sent false
24078 7079 Acked Read 5 Done Done Sent false
24079 7080 Acked Read 6 Done Done Sent false
24080 7081 Acked Write 6 Done Done Sent false
24081 7082 Acked Write 8 Done Done Sent false
24082 7083 Acked Write 10 Done Done Sent false
24083 7084 Acked Read 9 Done Done Sent false
24084 7085 Acked Read 5 Done Done Sent false
24085 7086 Acked Write 10 Done Done Sent false
24086 7087 Acked Flush 0 Done Done Sent false
24087 7088 Acked Write 2 Done Done Sent false
24088 7089 Acked Read 5 Done Done Sent false
24089 7090 Acked Read 10 Done Done Sent false
24090 7091 Acked Write 10 Done Done Sent false
24091 7092 Acked Write 1 Done Done Sent false
24092 7093 Acked Write 5 Done Done Sent false
24093 7094 Acked Write 10 Done Done Sent false
24094 7095 Acked Read 10 Done Done Sent false
24095 7096 Acked Read 4 Done Done Sent false
24096 7097 Acked Read 7 Done Done Sent false
24097 7098 Acked Write 2 Done Done Sent false
24098 7099 Acked Read 10 Done Done Sent false
24099 7100 Acked Read 10 Done Done Sent false
24100 7101 Acked Write 1 Done Done Sent false
24101 7102 Acked Write 2 Done Done Sent false
24102 7103 Acked Write 5 Done Done Sent false
24103 7104 Acked Write 9 Done Done Sent false
24104 7105 Acked Read 6 Done Done Sent false
24105 7106 Acked Read 5 Done Done Sent false
24106 7107 Acked Read 6 Done Done Sent false
24107 7108 Acked Read 2 Done Done Sent false
24108 7109 Acked Write 7 Done Done Sent false
24109 7110 Acked Write 5 Done Done Sent false
24110 7111 Acked Write 1 Done Done Sent false
24111 7112 Acked Write 5 Done Done Sent false
24112 7113 Acked Write 3 Done Done Sent false
24113 7114 Acked Write 6 Done Done Sent false
24114 7115 Acked Read 2 Done Done Sent false
24115 7116 Acked Read 9 Done Done Sent false
24116 7117 Acked Write 2 Done Done Sent false
24117 7118 Acked Read 10 Done Done Sent false
24118 7119 Acked Write 7 Done Done Sent false
24119 7120 Acked Write 8 Done Done Sent false
24120 7121 Acked Read 10 Done Done Sent false
24121 7122 Acked Write 10 Done Done Sent false
24122 7123 Acked Read 6 Done Done Sent false
24123 7124 Acked Read 8 Done Done Sent false
24124 7125 Acked Read 7 Done Done Sent false
24125 7126 Acked Write 3 Done Done Sent false
24126 7127 Acked Write 6 Done Done Sent false
24127 7128 Acked Write 9 Done Done Sent false
24128 7129 Acked Write 1 Done Done Sent false
24129 7130 Acked Write 1 Done Done Sent false
24130 7131 Acked Write 5 Done Done Sent false
24131 7132 Acked Read 7 Done Done Sent false
24132 7133 Acked Read 10 Done Done Sent false
24133 7134 Acked Write 2 Done Done Sent false
24134 7135 Acked Write 4 Done Done Sent false
24135 7136 Acked Read 6 Done Done Sent false
24136 7137 Acked Write 7 Done Done Sent false
24137 7138 Acked Write 10 Done Done Sent false
24138 7139 Acked Write 2 Done Done Sent false
24139 7140 Acked Write 1 Done Done Sent false
24140 7141 Acked Read 7 Done Done Sent false
24141 7142 Acked Flush 0 Done Done Sent false
24142 7143 Acked Write 3 Done Done Sent false
24143 7144 Acked Read 4 Done Done Sent false
24144 7145 Acked Read 1 Done Done Sent false
24145 7146 Acked Read 1 Done Done Sent false
24146 7147 Acked Write 4 Done Done Sent false
24147 7148 Acked Read 2 Done Done Sent false
24148 7149 Acked Write 1 Done Done Sent false
24149 7150 Acked Write 6 Done Done Sent false
24150 7151 Acked Write 7 Done Done Sent false
24151 7152 Acked Flush 0 Done Done Sent false
24152 7153 Acked Read 7 Done Done Sent false
24153 7154 Acked Read 1 Done Done Sent false
24154 7155 Acked Write 3 Done Done Sent false
24155 7156 Acked Read 6 Done Done Sent false
24156 7157 Acked Read 6 Done Done Sent false
24157 7158 Acked Read 2 Done Done Sent false
24158 7159 Acked Write 2 Done Done Sent false
24159 7160 Acked Write 3 Done Done Sent false
24160 7161 Acked Write 6 Done Done Sent false
24161 7162 Acked Write 6 Done Done Sent false
24162 7163 Acked Flush 0 Done Done Sent false
24163 7164 Acked Read 8 Done Done Sent false
24164 7165 Acked Read 5 Done Done Sent false
24165 7166 Acked Write 2 Done Done Sent false
24166 7167 Acked Read 9 Done Done Sent false
24167 7168 Acked Write 9 Done Done Sent false
24168 7169 Acked Write 4 Done Done Sent false
24169 7170 Acked Read 9 Done Done Sent false
24170 7171 Acked Read 8 Done Done Sent false
24171 7172 Acked Read 5 Done Done Sent false
24172 7173 Acked Read 9 Done Done Sent false
24173 7174 Acked Read 6 Done Done Sent false
24174 7175 Acked Read 10 Done Done Sent false
24175 7176 Acked Read 2 Done Done Sent false
24176 7177 Acked Write 10 Done Done Sent false
24177 7178 Acked Write 9 Done Done Sent false
24178 7179 Acked Read 10 Done Done Sent false
24179 7180 Acked Read 3 Done Done Sent false
24180 7181 Acked Read 1 Done Done Sent false
24181 7182 Acked Write 5 Done Done Sent false
24182 7183 Acked Write 9 Done Done Sent false
24183 7184 Acked Read 7 Done Done Sent false
24184 7185 Acked Read 6 Done Done Sent false
24185 7186 Acked Read 6 Done Done Sent false
24186 7187 Acked Read 6 Done Done Sent false
24187 7188 Acked Write 10 Done Done Sent false
24188 7189 Acked Read 4 Done Done Sent false
24189 7190 Acked Flush 0 Done Done Sent false
24190 7191 Acked Write 5 Done Done Sent false
24191 7192 Acked Read 8 Done Done Sent false
24192 7193 Acked Write 5 Done Done Sent false
24193 7194 Acked Read 1 Done Done Sent false
24194 7195 Acked Write 8 Done Done Sent false
24195 7196 Acked Write 3 Done Done Sent false
24196 7197 Acked Read 7 Done Done Sent false
24197 7198 Acked Write 3 Done Done Sent false
24198 7199 Acked Read 9 Done Done Sent false
24199 7200 Acked Write 3 Done Done Sent false
24200 7201 Acked Write 9 Done Done Sent false
24201 7202 Acked Read 3 Done Done Sent false
24202 7203 Acked Read 9 Done Done Sent false
24203 7204 Acked Read 4 Done Done Sent false
24204 7205 Acked Write 7 Done Done Sent false
24205 7206 Acked Write 1 Done Done Sent false
24206 7207 Acked Write 9 Done Done Sent false
24207 7208 Acked Read 8 Done Done Sent false
24208 7209 Acked Write 9 Done Done Sent false
24209 7210 Acked Write 7 Done Done Sent false
24210 7211 Acked Write 9 Done Done Sent false
24211 7212 Acked Read 9 Done Done Sent false
24212 7213 Acked Write 1 Done Done Sent false
24213 7214 Acked Read 7 Done Done Sent false
24214 7215 Acked Read 4 Done Done Sent false
24215 7216 Acked Read 4 Done Done Sent false
24216 7217 Acked Write 9 Done Done Sent false
24217 7218 Acked Read 4 Done Done Sent false
24218 7219 Acked Read 8 Done Done Sent false
24219 7220 Acked Read 1 Done Done Sent false
24220 7221 Acked Write 7 Done Done Sent false
24221 7222 Acked Read 6 Done Done Sent false
24222 7223 Acked Read 9 Done Done Sent false
24223 7224 Acked Read 2 Done Done Sent false
24224 7225 Acked Read 3 Done Done Sent false
24225 7226 Acked Write 3 Done Done Sent false
24226 7227 Acked Read 10 Done Done Sent false
24227 7228 Acked Read 7 Done Done Sent false
24228 7229 Acked Write 8 Done Done Sent false
24229 7230 Acked Write 3 Done Done Sent false
24230 7231 Acked Flush 0 Done Done Sent false
24231 7232 Acked Flush 0 Done Done Sent false
24232 7233 Acked Read 3 Done Done Sent false
24233 7234 Acked Write 3 Done Done Sent false
24234 7235 Acked Write 4 Done Done Sent false
24235 7236 Acked Write 2 Done Done Sent false
24236 7237 Acked Write 6 Done Done Sent false
24237 7238 Acked Read 4 Done Done Sent false
24238 7239 Acked Read 8 Done Done Sent false
24239 7240 Acked Read 5 Done Done Sent false
24240 7241 Acked Write 2 Done Done Sent false
24241 7242 Acked Write 6 Done Done Sent false
24242 7243 Acked Read 4 Done Done Sent false
24243 7244 Acked Write 6 Done Done Sent false
24244 7245 Acked Read 9 Done Done Sent false
24245 7246 Acked Read 5 Done Done Sent false
24246 7247 Acked Read 10 Done Done Sent false
24247 7248 Acked Write 3 Done Done Sent false
24248 7249 Acked Write 4 Done Done Sent false
24249 7250 Acked Read 6 Done Done Sent false
24250 7251 Acked Read 8 Done Done Sent false
24251 7252 Acked Write 2 Done Done Sent false
24252 7253 Acked Flush 0 Done Done Sent false
24253 7254 Acked Flush 0 Done Done Sent false
24254 7255 Acked Read 10 Done Done Sent false
24255 7256 Acked Read 3 Done Done Sent false
24256 7257 Acked Write 7 Done Done Sent false
24257 7258 Acked Write 2 Done Done Sent false
24258 7259 Acked Flush 0 Done Done Sent false
24259 7260 Acked Read 3 Done Done Sent false
24260 7261 Acked Write 6 Done Done Sent false
24261 7262 Acked Read 2 Done Done Sent false
24262 7263 Acked Read 2 Done Done Sent false
24263 7264 Acked Read 2 Done Done Sent false
24264 7265 Acked Flush 0 Done Done Sent false
24265 7266 Acked Read 10 Done Done Sent false
24266 7267 Acked Read 4 Done Done Sent false
24267 7268 Acked Write 9 Done Done Sent false
24268 7269 Acked Flush 0 Done Done Sent false
24269 7270 Acked Write 9 Done Done Sent false
24270 7271 Acked Write 5 Done Done Sent false
24271 7272 Acked Read 10 Done Done Sent false
24272 7273 Acked Read 3 Done Done Sent false
24273 7274 Acked Read 1 Done Done Sent false
24274 7275 Acked Write 9 Done Done Sent false
24275 7276 Acked Write 4 Done Done Sent false
24276 7277 Acked Flush 0 Done Done Sent false
24277 7278 Acked Write 3 Done Done Sent false
24278 7279 Acked Read 5 Done Done Sent false
24279 7280 Acked Write 4 Done Done Sent false
24280 7281 Acked Write 8 Done Done Sent false
24281 7282 Acked Read 2 Done Done Sent false
24282 7283 Acked Write 7 Done Done Sent false
24283 7284 Acked Read 8 Done Done Sent false
24284 7285 Acked Read 1 Done Done Sent false
24285 7286 Acked Read 3 Done Done Sent false
24286 7287 Acked Read 3 Done Done Sent false
24287 7288 Acked Read 9 Done Done Sent false
24288 7289 Acked Read 6 Done Done Sent false
24289 7290 Acked Read 8 Done Done Sent false
24290 7291 Acked Read 5 Done Done Sent false
24291 7292 Acked Read 6 Done Done Sent false
24292 7293 Acked Write 2 Done Done Sent false
24293 7294 Acked Flush 0 Done Done Sent false
24294 7295 Acked Write 8 Done Done Sent false
24295 7296 Acked Read 7 Done Done Sent false
24296 7297 Acked Write 8 Done Done Sent false
24297 7298 Acked Write 7 Done Done Sent false
24298 7299 Acked Write 7 Done Done Sent false
24299 7300 Acked Read 9 Done Done Sent false
24300 7301 Acked Flush 0 Done Done Sent false
24301 7302 Acked Write 8 Done Done Sent false
24302 7303 Acked Read 3 Done Done Sent false
24303 7304 Acked Write 9 Done Done Sent false
24304 7305 Acked Write 6 Done Done Sent false
24305 7306 Acked Read 5 Done Done Sent false
24306 7307 Acked Write 1 Done Done Sent false
24307 7308 Acked Write 3 Done Done Sent false
24308 7309 Acked Read 8 Done Done Sent false
24309 7310 Acked Flush 0 Done Done Sent false
24310 7311 Acked Write 8 Done Done Sent false
24311 7312 Acked Read 9 Done Done Sent false
24312 7313 Acked Read 9 Done Done Sent false
24313 7314 Acked Write 7 Done Done Sent false
24314 7315 Acked Write 10 Done Done Sent false
24315 7316 Acked Read 2 Done Done Sent false
24316 7317 Acked Write 2 Done Done Sent false
24317 7318 Acked Read 9 Done Done Sent false
24318 7319 Acked Write 10 Done Done Sent false
24319 7320 Acked Write 7 Done Done Sent false
24320 7321 Acked Write 6 Done Done Sent false
24321 7322 Acked Read 10 Done Done Sent false
24322 7323 Acked Read 9 Done Done Sent false
24323 7324 Acked Write 7 Done Done Sent false
24324 7325 Acked Write 2 Done Done Sent false
24325 7326 Acked Flush 0 Done Done Sent false
24326 7327 Acked Flush 0 Done Done Sent false
24327 7328 Acked Write 5 Done Done Sent false
24328 7329 Acked Read 4 Done Done Sent false
24329 7330 Acked Read 8 Done Done Sent false
24330 7331 Acked Read 3 Done Done Sent false
24331 7332 Acked Write 1 Done Done Sent false
24332 7333 Acked Read 3 Done Done Sent false
24333 7334 Acked Write 8 Done Done Sent false
24334 7335 Acked Write 8 Done Done Sent false
24335 7336 Acked Read 8 Done Done Sent false
24336 7337 Acked Read 5 Done Done Sent false
24337 7338 Acked Read 9 Done Done Sent false
24338 STATES DS:0 DS:1 DS:2 TOTAL
24339 Sent 0 0 301 301
24340 Done 301 301 0 602
24341 Skipped 0 0 0 0
24342 Error 0 0 0 0
24343 Last Flush: Some(JobId(7327)) Some(JobId(7327)) Some(JobId(7037))
24344 Downstairs last five completed: 7037 7036 7035 7034 7033
24345 Upstairs last five completed: 7338 7337 7336 7335 7334
24346 CLIENT: Up:0 ds:301 act:2
243472025-03-17T22:21:41.111ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
243482025-03-17T22:21:42.552ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
243492025-03-17T22:21:43.114ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
243502025-03-17T22:21:44.116ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
24351 [8830] got start action
24352 [8830] I am going to restart
24353 Starting downstairs at port 8830
24354 Make output file at "/tmp/dsc/downstairs-8830.txt"
24355 [8830][2] reports Starting
24356 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1227)
24357 [8830][2] reports Running
24358 ----------------------------------------------------------------
24359 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24360 States: Active Active Offline
24361 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
24362 7038 Acked Read 6 Done Done Sent false
24363 7039 Acked Read 10 Done Done Sent false
24364 7040 Acked Write 6 Done Done Sent false
24365 7041 Acked Write 5 Done Done Sent false
24366 7042 Acked Read 1 Done Done Sent false
24367 7043 Acked Flush 0 Done Done Sent false
24368 7044 Acked Read 4 Done Done Sent false
24369 7045 Acked Read 3 Done Done Sent false
24370 7046 Acked Read 3 Done Done Sent false
24371 7047 Acked Write 3 Done Done Sent false
24372 7048 Acked Read 4 Done Done Sent false
24373 7049 Acked Read 5 Done Done Sent false
24374 7050 Acked Read 3 Done Done Sent false
24375 7051 Acked Write 8 Done Done Sent false
24376 7052 Acked Flush 0 Done Done Sent false
24377 7053 Acked Read 10 Done Done Sent false
24378 7054 Acked Read 4 Done Done Sent false
24379 7055 Acked Write 9 Done Done Sent false
24380 7056 Acked Read 8 Done Done Sent false
24381 7057 Acked Read 6 Done Done Sent false
24382 7058 Acked Write 8 Done Done Sent false
24383 7059 Acked Read 4 Done Done Sent false
24384 7060 Acked Write 8 Done Done Sent false
24385 7061 Acked Write 10 Done Done Sent false
24386 7062 Acked Read 4 Done Done Sent false
24387 7063 Acked Read 5 Done Done Sent false
24388 7064 Acked Write 9 Done Done Sent false
24389 7065 Acked Write 9 Done Done Sent false
24390 7066 Acked Write 1 Done Done Sent false
24391 7067 Acked Write 3 Done Done Sent false
24392 7068 Acked Read 5 Done Done Sent false
24393 7069 Acked Read 7 Done Done Sent false
24394 7070 Acked Read 4 Done Done Sent false
24395 7071 Acked Write 6 Done Done Sent false
24396 7072 Acked Read 6 Done Done Sent false
24397 7073 Acked Read 7 Done Done Sent false
24398 7074 Acked Read 1 Done Done Sent false
24399 7075 Acked Read 3 Done Done Sent false
24400 7076 Acked Read 3 Done Done Sent false
24401 7077 Acked Write 2 Done Done Sent false
24402 7078 Acked Read 5 Done Done Sent false
24403 7079 Acked Read 5 Done Done Sent false
24404 7080 Acked Read 6 Done Done Sent false
24405 7081 Acked Write 6 Done Done Sent false
24406 7082 Acked Write 8 Done Done Sent false
24407 7083 Acked Write 10 Done Done Sent false
24408 7084 Acked Read 9 Done Done Sent false
24409 7085 Acked Read 5 Done Done Sent false
24410 7086 Acked Write 10 Done Done Sent false
24411 7087 Acked Flush 0 Done Done Sent false
24412 7088 Acked Write 2 Done Done Sent false
24413 7089 Acked Read 5 Done Done Sent false
24414 7090 Acked Read 10 Done Done Sent false
24415 7091 Acked Write 10 Done Done Sent false
24416 7092 Acked Write 1 Done Done Sent false
24417 7093 Acked Write 5 Done Done Sent false
24418 7094 Acked Write 10 Done Done Sent false
24419 7095 Acked Read 10 Done Done Sent false
24420 7096 Acked Read 4 Done Done Sent false
24421 7097 Acked Read 7 Done Done Sent false
24422 7098 Acked Write 2 Done Done Sent false
24423 7099 Acked Read 10 Done Done Sent false
24424 7100 Acked Read 10 Done Done Sent false
24425 7101 Acked Write 1 Done Done Sent false
24426 7102 Acked Write 2 Done Done Sent false
24427 7103 Acked Write 5 Done Done Sent false
24428 7104 Acked Write 9 Done Done Sent false
24429 7105 Acked Read 6 Done Done Sent false
24430 7106 Acked Read 5 Done Done Sent false
24431 7107 Acked Read 6 Done Done Sent false
24432 7108 Acked Read 2 Done Done Sent false
24433 7109 Acked Write 7 Done Done Sent false
24434 7110 Acked Write 5 Done Done Sent false
24435 7111 Acked Write 1 Done Done Sent false
24436 7112 Acked Write 5 Done Done Sent false
24437 7113 Acked Write 3 Done Done Sent false
24438 7114 Acked Write 6 Done Done Sent false
24439 7115 Acked Read 2 Done Done Sent false
24440 7116 Acked Read 9 Done Done Sent false
24441 7117 Acked Write 2 Done Done Sent false
24442 7118 Acked Read 10 Done Done Sent false
24443 7119 Acked Write 7 Done Done Sent false
24444 7120 Acked Write 8 Done Done Sent false
24445 7121 Acked Read 10 Done Done Sent false
24446 7122 Acked Write 10 Done Done Sent false
24447 7123 Acked Read 6 Done Done Sent false
24448 7124 Acked Read 8 Done Done Sent false
24449 7125 Acked Read 7 Done Done Sent false
24450 7126 Acked Write 3 Done Done Sent false
24451 7127 Acked Write 6 Done Done Sent false
24452 7128 Acked Write 9 Done Done Sent false
24453 7129 Acked Write 1 Done Done Sent false
24454 7130 Acked Write 1 Done Done Sent false
24455 7131 Acked Write 5 Done Done Sent false
24456 7132 Acked Read 7 Done Done Sent false
24457 7133 Acked Read 10 Done Done Sent false
24458 7134 Acked Write 2 Done Done Sent false
24459 7135 Acked Write 4 Done Done Sent false
24460 7136 Acked Read 6 Done Done Sent false
24461 7137 Acked Write 7 Done Done Sent false
24462 7138 Acked Write 10 Done Done Sent false
24463 7139 Acked Write 2 Done Done Sent false
24464 7140 Acked Write 1 Done Done Sent false
24465 7141 Acked Read 7 Done Done Sent false
24466 7142 Acked Flush 0 Done Done Sent false
24467 7143 Acked Write 3 Done Done Sent false
24468 7144 Acked Read 4 Done Done Sent false
24469 7145 Acked Read 1 Done Done Sent false
24470 7146 Acked Read 1 Done Done Sent false
24471 7147 Acked Write 4 Done Done Sent false
24472 7148 Acked Read 2 Done Done Sent false
24473 7149 Acked Write 1 Done Done Sent false
24474 7150 Acked Write 6 Done Done Sent false
24475 7151 Acked Write 7 Done Done Sent false
24476 7152 Acked Flush 0 Done Done Sent false
24477 7153 Acked Read 7 Done Done Sent false
24478 7154 Acked Read 1 Done Done Sent false
24479 7155 Acked Write 3 Done Done Sent false
24480 7156 Acked Read 6 Done Done Sent false
24481 7157 Acked Read 6 Done Done Sent false
24482 7158 Acked Read 2 Done Done Sent false
24483 7159 Acked Write 2 Done Done Sent false
24484 7160 Acked Write 3 Done Done Sent false
24485 7161 Acked Write 6 Done Done Sent false
24486 7162 Acked Write 6 Done Done Sent false
24487 7163 Acked Flush 0 Done Done Sent false
24488 7164 Acked Read 8 Done Done Sent false
24489 7165 Acked Read 5 Done Done Sent false
24490 7166 Acked Write 2 Done Done Sent false
24491 7167 Acked Read 9 Done Done Sent false
24492 7168 Acked Write 9 Done Done Sent false
24493 7169 Acked Write 4 Done Done Sent false
24494 7170 Acked Read 9 Done Done Sent false
24495 7171 Acked Read 8 Done Done Sent false
24496 7172 Acked Read 5 Done Done Sent false
24497 7173 Acked Read 9 Done Done Sent false
24498 7174 Acked Read 6 Done Done Sent false
24499 7175 Acked Read 10 Done Done Sent false
24500 7176 Acked Read 2 Done Done Sent false
24501 7177 Acked Write 10 Done Done Sent false
24502 7178 Acked Write 9 Done Done Sent false
24503 7179 Acked Read 10 Done Done Sent false
24504 7180 Acked Read 3 Done Done Sent false
24505 7181 Acked Read 1 Done Done Sent false
24506 7182 Acked Write 5 Done Done Sent false
24507 7183 Acked Write 9 Done Done Sent false
24508 7184 Acked Read 7 Done Done Sent false
24509 7185 Acked Read 6 Done Done Sent false
24510 7186 Acked Read 6 Done Done Sent false
24511 7187 Acked Read 6 Done Done Sent false
24512 7188 Acked Write 10 Done Done Sent false
24513 7189 Acked Read 4 Done Done Sent false
24514 7190 Acked Flush 0 Done Done Sent false
24515 7191 Acked Write 5 Done Done Sent false
24516 7192 Acked Read 8 Done Done Sent false
24517 7193 Acked Write 5 Done Done Sent false
24518 7194 Acked Read 1 Done Done Sent false
24519 7195 Acked Write 8 Done Done Sent false
24520 7196 Acked Write 3 Done Done Sent false
24521 7197 Acked Read 7 Done Done Sent false
24522 7198 Acked Write 3 Done Done Sent false
24523 7199 Acked Read 9 Done Done Sent false
24524 7200 Acked Write 3 Done Done Sent false
24525 7201 Acked Write 9 Done Done Sent false
24526 7202 Acked Read 3 Done Done Sent false
24527 7203 Acked Read 9 Done Done Sent false
24528 7204 Acked Read 4 Done Done Sent false
24529 7205 Acked Write 7 Done Done Sent false
24530 7206 Acked Write 1 Done Done Sent false
24531 7207 Acked Write 9 Done Done Sent false
24532 7208 Acked Read 8 Done Done Sent false
24533 7209 Acked Write 9 Done Done Sent false
24534 7210 Acked Write 7 Done Done Sent false
24535 7211 Acked Write 9 Done Done Sent false
24536 7212 Acked Read 9 Done Done Sent false
24537 7213 Acked Write 1 Done Done Sent false
24538 7214 Acked Read 7 Done Done Sent false
24539 7215 Acked Read 4 Done Done Sent false
24540 7216 Acked Read 4 Done Done Sent false
24541 7217 Acked Write 9 Done Done Sent false
24542 7218 Acked Read 4 Done Done Sent false
24543 7219 Acked Read 8 Done Done Sent false
24544 7220 Acked Read 1 Done Done Sent false
24545 7221 Acked Write 7 Done Done Sent false
24546 7222 Acked Read 6 Done Done Sent false
24547 7223 Acked Read 9 Done Done Sent false
24548 7224 Acked Read 2 Done Done Sent false
24549 7225 Acked Read 3 Done Done Sent false
24550 7226 Acked Write 3 Done Done Sent false
24551 7227 Acked Read 10 Done Done Sent false
24552 7228 Acked Read 7 Done Done Sent false
24553 7229 Acked Write 8 Done Done Sent false
24554 7230 Acked Write 3 Done Done Sent false
24555 7231 Acked Flush 0 Done Done Sent false
24556 7232 Acked Flush 0 Done Done Sent false
24557 7233 Acked Read 3 Done Done Sent false
24558 7234 Acked Write 3 Done Done Sent false
24559 7235 Acked Write 4 Done Done Sent false
24560 7236 Acked Write 2 Done Done Sent false
24561 7237 Acked Write 6 Done Done Sent false
24562 7238 Acked Read 4 Done Done Sent false
24563 7239 Acked Read 8 Done Done Sent false
24564 7240 Acked Read 5 Done Done Sent false
24565 7241 Acked Write 2 Done Done Sent false
24566 7242 Acked Write 6 Done Done Sent false
24567 7243 Acked Read 4 Done Done Sent false
24568 7244 Acked Write 6 Done Done Sent false
24569 7245 Acked Read 9 Done Done Sent false
24570 7246 Acked Read 5 Done Done Sent false
24571 7247 Acked Read 10 Done Done Sent false
24572 7248 Acked Write 3 Done Done Sent false
24573 7249 Acked Write 4 Done Done Sent false
24574 7250 Acked Read 6 Done Done Sent false
24575 7251 Acked Read 8 Done Done Sent false
24576 7252 Acked Write 2 Done Done Sent false
24577 7253 Acked Flush 0 Done Done Sent false
24578 7254 Acked Flush 0 Done Done Sent false
24579 7255 Acked Read 10 Done Done Sent false
24580 7256 Acked Read 3 Done Done Sent false
24581 7257 Acked Write 7 Done Done Sent false
24582 7258 Acked Write 2 Done Done Sent false
24583 7259 Acked Flush 0 Done Done Sent false
24584 7260 Acked Read 3 Done Done Sent false
24585 7261 Acked Write 6 Done Done Sent false
24586 7262 Acked Read 2 Done Done Sent false
24587 7263 Acked Read 2 Done Done Sent false
24588 7264 Acked Read 2 Done Done Sent false
24589 7265 Acked Flush 0 Done Done Sent false
24590 7266 Acked Read 10 Done Done Sent false
24591 7267 Acked Read 4 Done Done Sent false
24592 7268 Acked Write 9 Done Done Sent false
24593 7269 Acked Flush 0 Done Done Sent false
24594 7270 Acked Write 9 Done Done Sent false
24595 7271 Acked Write 5 Done Done Sent false
24596 7272 Acked Read 10 Done Done Sent false
24597 7273 Acked Read 3 Done Done Sent false
24598 7274 Acked Read 1 Done Done Sent false
24599 7275 Acked Write 9 Done Done Sent false
24600 7276 Acked Write 4 Done Done Sent false
24601 7277 Acked Flush 0 Done Done Sent false
24602 7278 Acked Write 3 Done Done Sent false
24603 7279 Acked Read 5 Done Done Sent false
24604 7280 Acked Write 4 Done Done Sent false
24605 7281 Acked Write 8 Done Done Sent false
24606 7282 Acked Read 2 Done Done Sent false
24607 7283 Acked Write 7 Done Done Sent false
24608 7284 Acked Read 8 Done Done Sent false
24609 7285 Acked Read 1 Done Done Sent false
24610 7286 Acked Read 3 Done Done Sent false
24611 7287 Acked Read 3 Done Done Sent false
24612 7288 Acked Read 9 Done Done Sent false
24613 7289 Acked Read 6 Done Done Sent false
24614 7290 Acked Read 8 Done Done Sent false
24615 7291 Acked Read 5 Done Done Sent false
24616 7292 Acked Read 6 Done Done Sent false
24617 7293 Acked Write 2 Done Done Sent false
24618 7294 Acked Flush 0 Done Done Sent false
24619 7295 Acked Write 8 Done Done Sent false
24620 7296 Acked Read 7 Done Done Sent false
24621 7297 Acked Write 8 Done Done Sent false
24622 7298 Acked Write 7 Done Done Sent false
24623 7299 Acked Write 7 Done Done Sent false
24624 7300 Acked Read 9 Done Done Sent false
24625 7301 Acked Flush 0 Done Done Sent false
24626 7302 Acked Write 8 Done Done Sent false
24627 7303 Acked Read 3 Done Done Sent false
24628 7304 Acked Write 9 Done Done Sent false
24629 7305 Acked Write 6 Done Done Sent false
24630 7306 Acked Read 5 Done Done Sent false
24631 7307 Acked Write 1 Done Done Sent false
24632 7308 Acked Write 3 Done Done Sent false
24633 7309 Acked Read 8 Done Done Sent false
24634 7310 Acked Flush 0 Done Done Sent false
24635 7311 Acked Write 8 Done Done Sent false
24636 7312 Acked Read 9 Done Done Sent false
24637 7313 Acked Read 9 Done Done Sent false
24638 7314 Acked Write 7 Done Done Sent false
24639 7315 Acked Write 10 Done Done Sent false
24640 7316 Acked Read 2 Done Done Sent false
24641 7317 Acked Write 2 Done Done Sent false
24642 7318 Acked Read 9 Done Done Sent false
24643 7319 Acked Write 10 Done Done Sent false
24644 7320 Acked Write 7 Done Done Sent false
24645 7321 Acked Write 6 Done Done Sent false
24646 7322 Acked Read 10 Done Done Sent false
24647 7323 Acked Read 9 Done Done Sent false
24648 7324 Acked Write 7 Done Done Sent false
24649 7325 Acked Write 2 Done Done Sent false
24650 7326 Acked Flush 0 Done Done Sent false
24651 7327 Acked Flush 0 Done Done Sent false
24652 7328 Acked Write 5 Done Done Sent false
24653 7329 Acked Read 4 Done Done Sent false
24654 7330 Acked Read 8 Done Done Sent false
24655 7331 Acked Read 3 Done Done Sent false
24656 7332 Acked Write 1 Done Done Sent false
24657 7333 Acked Read 3 Done Done Sent false
24658 7334 Acked Write 8 Done Done Sent false
24659 7335 Acked Write 8 Done Done Sent false
24660 7336 Acked Read 8 Done Done Sent false
24661 7337 Acked Read 5 Done Done Sent false
24662 7338 Acked Read 9 Done Done Sent false
24663 STATES DS:0 DS:1 DS:2 TOTAL
24664 Sent 0 0 301 301
24665 Done 301 301 0 602
24666 Skipped 0 0 0 0
24667 Error 0 0 0 0
24668 Last Flush: Some(JobId(7327)) Some(JobId(7327)) Some(JobId(7037))
24669 Downstairs last five completed: 7037 7036 7035 7034 7033
24670 Upstairs last five completed: 7338 7337 7336 7335 7334
24671 CLIENT: Up:0 ds:301 act:2
246722025-03-17T22:21:45.117ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
246732025-03-17T22:21:46.118ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
246742025-03-17T22:21:46.300ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
246752025-03-17T22:21:46.300ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:33705) = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
246762025-03-17T22:21:46.301ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
246772025-03-17T22:21:46.302ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
246782025-03-17T22:21:46.302ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
246792025-03-17T22:21:46.302ZINFOcrucible: send last flush ID to this DS: Some(JobId(7037)) = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
246802025-03-17T22:21:46.302ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
246812025-03-17T22:21:46.302ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(7037)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
246822025-03-17T22:21:46.302ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(7037)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
24683 ----------------------------------------------------------------
24684 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
24685 Downstairs last five completed: 7339 7338 7337 7336 7335
24686 Upstairs last five completed: 7339 7338 7337 7336 7335
24687 CLIENT: Up:0 ds:0 act:3
24688 Replay: All jobs finished
24689 Main task has work to do, go find it
24690 got dsc Stop(2)
24691 stop 2
24692 Mar 17 22:21:48.559 INFO request completed, latency_us: 332, response_code[8830] Got stop action so:false kr:false
24693 : 204, uri: /stop/cid/2, method: POST, req_id: 1de7f0ae-b946-41f6-8bf7-d4ecba3131f6, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
246942025-03-17T22:21:48.560ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
246952025-03-17T22:21:48.560ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
246962025-03-17T22:21:48.560ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
24697 Mar 17 22:21:48.560 INFO request completed, latency_us: 177, response_code: 200, uri: /state/cid/2, method: GET, req_id: 4b8b52ea-8a5a-40cc-8370-1dee92759b5d, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
246982025-03-17T22:21:48.560ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
24699 [8830] Exited with: ExitStatus(unix_wait_status(9))
24700 [8830][2] reports Exit
24701 Mar 17 22:21:52.562 INFO request completed, latency_us: 237, response_code: 200, uri: /state/cid/2, method: GET, req_id: 1d05afba-489e-4462-af8e-c6204e19b468, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
24702 001/300 Write block 2899 len 32768 data: 5 7 5 4 4 5 5 5
24703 002/300 Read block 3307 len 12288
24704 003/300 Flush
24705 004/300 Write block 2697 len 16384 data: 10 10 7 7
24706 005/300 Read block 816 len 4096
24707 006/300 Write block 3182 len 4096 data: 4
24708 007/300 Write block 4483 len 28672 data: 3 4 5 7 6 5 5
24709 008/300 Read block 362 len 4096
24710 009/300 Flush
24711 010/300 Write block 4307 len 32768 data: 2 2 2 2 2 2 3 4
24712 011/300 Read block 2552 len 32768
24713 012/300 Write block 4159 len 40960 data: 5 6 6 6 6 5 3 5 4 5
24714 013/300 Write block 4294 len 36864 data: 3 3 4 6 7 8 8 10 8
24715 014/300 Read block 4659 len 36864
24716 015/300 Read block 2050 len 32768
24717 016/300 Read block 3759 len 4096
24718 017/300 Read block 1921 len 8192
24719 018/300 Read block 4932 len 32768
24720 019/300 Write block 3778 len 28672 data: 3 3 4 4 4 3 2
24721 020/300 Flush
24722 021/300 Read block 4626 len 12288
24723 022/300 Read block 2623 len 4096
24724 023/300 Read block 110 len 36864
24725 024/300 Write block 985 len 40960 data: 4 4 4 6 6 5 5 6 6 6
24726 025/300 Read block 2352 len 4096
24727 026/300 Write block 2166 len 32768 data: 5 4 5 4 5 5 5 5
24728 027/300 Flush
24729 028/300 Read block 655 len 32768
24730 029/300 Flush
24731 030/300 Write block 4986 len 40960 data: 5 3 3 2 3 4 4 4 4 4
24732 031/300 Write block 4620 len 40960 data: 3 4 4 4 3 5 5 4 5 5
24733 032/300 Read block 1283 len 32768
24734 033/300 Write block 3567 len 40960 data: 4 3 3 4 3 3 3 3 2 2
24735 034/300 Read block 556 len 16384
24736 035/300 Read block 2687 len 36864
24737 036/300 Write block 25 len 24576 data: 6 7 7 8 6 5
24738 037/300 Read block 2695 len 24576
24739 038/300 Write block 1320 len 36864 data: 3 4 3 5 5 4 7 7 7
24740 039/300 Write block 4097 len 12288 data: 4 5 6
24741 040/300 Read block 1756 len 36864
24742 041/300 Write block 3490 len 16384 data: 5 7 8 6
24743 042/300 Write block 4186 len 12288 data: 5 5 4
24744 043/300 Flush
24745 044/300 Write block 4170 len 40960 data: 5 5 5 4 4 2 3 4 4 4
24746 045/300 Write block 4070 len 20480 data: 4 4 4 5 5
24747 046/300 Read block 2003 len 28672
24748 047/300 Write block 3991 len 24576 data: 3 2 3 3 3 3
24749 048/300 Read block 1019 len 12288
24750 049/300 Write block 3759 len 12288 data: 4 5 7
24751 050/300 Flush
24752 051/300 Write block 4373 len 20480 data: 6 5 5 5 6
24753 052/300 Read block 4860 len 36864
24754 053/300 Flush
24755 054/300 Write block 750 len 28672 data: 3 3 3 3 4 4 4
24756 055/300 Read block 2546 len 28672
24757 056/300 Read block 4775 len 8192
24758 057/300 Read block 4632 len 12288
24759 058/300 Read block 976 len 16384
24760 059/300 Read block 1439 len 8192
24761 060/300 Read block 2468 len 20480
24762 061/300 Read block 63 len 24576
24763 062/300 Write block 4292 len 8192 data: 3 3
24764 063/300 Read block 4154 len 32768
24765 064/300 Read block 1427 len 20480
24766 065/300 Read block 3235 len 4096
24767 066/300 Read block 3291 len 4096
24768 067/300 Write block 1983 len 40960 data: 5 5 6 8 7 6 9 8 7 8
24769 068/300 Write block 4128 len 8192 data: 6 4
24770 069/300 Read block 2293 len 4096
24771 070/300 Write block 401 len 28672 data: 7 6 8 8 6 5 6
24772 071/300 Read block 3493 len 28672
24773 072/300 Read block 2604 len 28672
24774 073/300 Read block 1638 len 24576
24775 074/300 Read block 4228 len 16384
24776 075/300 Write block 4270 len 20480 data: 5 6 5 5 5
24777 076/300 Flush
24778 077/300 Read block 564 len 12288
24779 078/300 Flush
24780 079/300 Write block 1657 len 12288 data: 4 4 4
24781 080/300 Write block 3641 len 36864 data: 7 9 8 8 7 5 6 5 6
24782 081/300 Read block 3019 len 20480
24783 082/300 Write block 2339 len 28672 data: 5 3 3 3 3 3 3
24784 083/300 Read block 2856 len 20480
24785 084/300 Read block 1391 len 24576
24786 085/300 Read block 1427 len 28672
24787 086/300 Write block 1979 len 36864 data: 5 5 6 5 6 6 7 9 8
24788 087/300 Read block 1424 len 4096
24789 088/300 Read block 2402 len 20480
24790 089/300 Flush
24791 090/300 Read block 3898 len 4096
24792 091/300 Write block 1262 len 8192 data: 4 4
24793 092/300 Flush
24794 093/300 Read block 870 len 40960
24795 094/300 Write block 1222 len 28672 data: 9 8 8 7 6 5 6
24796 095/300 Read block 2027 len 28672
24797 096/300 Read block 1579 len 20480
24798 097/300 Write block 4105 len 16384 data: 5 7 9 8
24799 098/300 Read block 1240 len 8192
24800 099/300 Write block 2703 len 28672 data: 7 8 6 3 3 5 4
24801 100/300 Read block 2778 len 32768
24802 101/300 Write block 2494 len 32768 data: 5 7 6 6 7 7 6 4
24803 102/300 Read block 524 len 32768
24804 103/300 Write block 4857 len 4096 data: 6
24805 104/300 Flush
24806 105/300 Read block 3885 len 20480
24807 106/300 Write block 1433 len 4096 data: 4
24808 107/300 Flush
24809 108/300 Write block 584 len 20480 data: 3 4 3 4 5
24810 109/300 Read block 3291 len 16384
24811 110/300 Write block 125 len 28672 data: 6 7 6 6 6 6 6
24812 111/300 Read block 4417 len 36864
24813 112/300 Write block 2154 len 16384 data: 3 4 2 3
24814 113/300 Write block 2782 len 16384 data: 4 4 7 7
24815 114/300 Read block 2601 len 20480
24816 115/300 Write block 3006 len 36864 data: 7 7 5 5 6 6 5 6 6
24817 116/300 Read block 4204 len 24576
24818 117/300 Write block 4641 len 4096 data: 3
24819 118/300 Read block 4559 len 12288
24820 119/300 Read block 3302 len 36864
24821 120/300 Flush
24822 121/300 Read block 4531 len 32768
24823 122/300 Read block 3014 len 20480
24824 123/300 Read block 1400 len 28672
24825 124/300 Read block 1924 len 8192
24826 125/300 Read block 371 len 40960
24827 126/300 Read block 77 len 40960
24828 127/300 Write block 2178 len 40960 data: 9 8 7 8 7 7 6 6 6 5
24829 128/300 Write block 739 len 32768 data: 5 6 4 4 4 5 4 4
24830 129/300 Read block 3770 len 12288
24831 130/300 Read block 2808 len 16384
24832 131/300 Read block 3051 len 40960
24833 132/300 Write block 1295 len 32768 data: 4 4 4 5 5 5 5 4
24834 133/300 Write block 2766 len 4096 data: 4
24835 134/300 Read block 4800 len 8192
24836 135/300 Read block 2029 len 24576
24837 136/300 Read block 4735 len 36864
24838 137/300 Read block 4003 len 20480
24839 138/300 Write block 4621 len 20480 data: 5 5 5 4 6
24840 139/300 Write block 29 len 4096 data: 7
24841 140/300 Read block 1545 len 4096
24842 141/300 Write block 2419 len 28672 data: 6 5 6 5 4 4 4
24843 142/300 Write block 24 len 28672 data: 5 7 8 8 9 8 6
24844 143/300 Flush
24845 144/300 Read block 4415 len 40960
24846 145/300 Write block 4015 len 40960 data: 4 4 4 4 6 5 5 4 4 2
24847 146/300 Write block 4341 len 16384 data: 4 6 5 5
24848 147/300 Write block 1193 len 12288 data: 2 2 2
24849 148/300 Read block 1211 len 8192
24850 149/300 Read block 3493 len 12288
24851 150/300 Write block 427 len 16384 data: 4 4 5 3
24852 151/300 Write block 2272 len 28672 data: 5 5 6 7 6 5 7
24853 152/300 Read block 3951 len 28672
24854 153/300 Write block 3355 len 12288 data: 7 7 5
24855 154/300 Flush
24856 155/300 Read block 1985 len 24576
24857 156/300 Write block 1032 len 36864 data: 7 8 9 7 5 5 5 8 9
24858 157/300 Read block 3644 len 12288
24859 158/300 Write block 3164 len 8192 data: 4 4
24860 159/300 Read block 550 len 4096
24861 160/300 Read block 3126 len 40960
24862 161/300 Read block 2077 len 24576
24863 162/300 Read block 4202 len 8192
24864 163/300 Read block 4026 len 28672
24865 164/300 Write block 860 len 8192 data: 5 5
24866 165/300 Write block 4420 len 32768 data: 6 9 8 8 7 7 6 6
24867 166/300 Write block 373 len 40960 data: 7 7 6 5 4 3 3 3 2 2
24868 167/300 Write block 2865 len 8192 data: 6 5
24869 168/300 Read block 4353 len 8192
24870 169/300 Read block 2675 len 12288
24871 170/300 Read block 4693 len 16384
24872 171/300 Read block 856 len 8192
24873 172/300 Read block 1883 len 16384
24874 173/300 Read block 374 len 32768
24875 174/300 Write block 1908 len 4096 data: 2
24876 175/300 Read block 2823 len 24576
24877 176/300 Write block 2209 len 40960 data: 7 6 5 5 6 6 7 5 7 7
24878 177/300 Flush
24879 178/300 Write block 2493 len 4096 data: 4
24880 179/300 Write block 2331 len 32768 data: 4 4 3 4 6 6 5 5
24881 180/300 Flush
24882 181/300 Read block 2060 len 12288
24883 182/300 Flush
24884 183/300 Read block 4494 len 16384
24885 184/300 Read block 3666 len 20480
24886 185/300 Write block 1292 len 32768 data: 4 4 3 5 5 5 6 6
24887 186/300 Write block 4157 len 16384 data: 5 4 6 7
24888 187/300 Read block 104 len 40960
24889 188/300 Write block 3960 len 28672 data: 3 2 2 2 2 2 3
24890 189/300 Write block 1839 len 16384 data: 3 3 3 5
24891 190/300 Write block 572 len 28672 data: 2 2 2 3 4 4 4
24892 191/300 Flush
24893 192/300 Flush
24894 193/300 Read block 496 len 16384
24895 194/300 Write block 4542 len 24576 data: 5 5 4 8 6 8
24896 195/300 Write block 4432 len 8192 data: 5 5
24897 196/300 Read block 3376 len 4096
24898 197/300 Write block 2881 len 12288 data: 5 2 3
24899 198/300 Read block 1052 len 8192
24900 199/300 Write block 3173 len 32768 data: 6 6 5 4 4 4 3 4
24901 200/300 Write block 89 len 20480 data: 4 4 6 5 4
24902 201/300 Read block 4735 len 36864
24903 202/300 Write block 64 len 24576 data: 7 7 6 5 6 6
24904 203/300 Write block 2357 len 40960 data: 5 7 7 5 4 4 4 4 4 4
24905 204/300 Read block 827 len 4096
24906 205/300 Read block 1194 len 24576
24907 206/300 Write block 3528 len 16384 data: 3 5 6 5
24908 207/300 Write block 419 len 16384 data: 6 6 5 5
24909 208/300 Read block 833 len 28672
24910 209/300 Write block 4533 len 24576 data: 8 7 7 6 6 6
24911 210/300 Write block 4755 len 8192 data: 5 6
24912 211/300 Read block 2416 len 8192
24913 212/300 Write block 2487 len 36864 data: 6 6 6 6 6 5 5 6 8
24914 213/300 Write block 2293 len 32768 data: 5 4 5 6 6 7 7 7
24915 214/300 Write block 1122 len 40960 data: 3 2 2 3 4 6 7 5 4 5
24916 215/300 Flush
24917 216/300 Read block 3297 len 4096
24918 217/300 Write block 994 len 32768 data: 7 5 3 4 4 2 2 2
24919 218/300 Write block 2267 len 12288 data: 4 4 4
24920 219/300 Read block 695 len 20480
24921 220/300 Write block 2206 len 20480 data: 8 9 8 8 7
24922 221/300 Write block 3398 len 32768 data: 4 4 3 4 7 9 8 7
24923 222/300 Write block 849 len 20480 data: 7 8 8 8 8
24924 223/300 Read block 3425 len 16384
24925 224/300 Write block 2324 len 20480 data: 4 4 6 5 5
24926 225/300 Flush
24927 226/300 Read block 16 len 4096
24928 227/300 Read block 2296 len 8192
24929 228/300 Flush
24930 229/300 Read block 298 len 16384
24931 230/300 Read block 508 len 4096
24932 231/300 Read block 1080 len 40960
24933 232/300 Read block 794 len 8192
24934 233/300 Read block 2073 len 32768
24935 234/300 Read block 4651 len 8192
24936 235/300 Write block 779 len 36864 data: 3 4 3 3 3 4 5 4 4
24937 236/300 Read block 3560 len 28672
24938 237/300 Read block 1668 len 8192
24939 238/300 Read block 1259 len 8192
24940 239/300 Read block 2450 len 24576
24941 240/300 Read block 4596 len 4096
24942 241/300 Read block 1906 len 24576
24943 242/300 Read block 2001 len 28672
24944 243/300 Read block 4580 len 8192
24945 244/300 Flush
24946 245/300 Read block 1219 len 36864
24947 246/300 Write block 2470 len 8192 data: 4 4
24948 247/300 Write block 3462 len 12288 data: 6 6 6
24949 248/300 Write block 4261 len 40960 data: 5 4 3 2 3 5 4 4 5 6
24950 249/300 Read block 780 len 4096
24951 250/300 Write block 4212 len 24576 data: 5 4 5 4 4 5
24952 251/300 Write block 2620 len 12288 data: 5 5 4
24953 252/300 Read block 3553 len 24576
24954 253/300 Read block 2353 len 32768
24955 254/300 Write block 4335 len 24576 data: 5 4 8 8 8 7
24956 255/300 Write block 3433 len 20480 data: 7 8 7 6 7
24957 256/300 Read block 4860 len 16384
24958 257/300 Read block 816 len 40960
24959 258/300 Read block 3814 len 32768
24960 259/300 Flush
24961 260/300 Read block 814 len 24576
24962 261/300 Write block 4016 len 16384 data: 5 5 5 7
24963 262/300 Read block 1051 len 8192
24964 263/300 Read block 83 len 4096
24965 264/300 Write block 197 len 24576 data: 3 3 4 5 5 5
24966 265/300 Write block 851 len 16384 data: 9 9 9 9
24967 266/300 Flush
24968 267/300 Read block 3262 len 4096
24969 268/300 Read block 1318 len 40960
24970 269/300 Write block 1452 len 20480 data: 4 4 4 3 4
24971 270/300 Read block 3893 len 36864
24972 271/300 Read block 3420 len 4096
24973 272/300 Flush
24974 273/300 Flush
24975 274/300 Write block 51 len 4096 data: 5
24976 275/300 Read block 4822 len 28672
24977 276/300 Read block 361 len 8192
24978 277/300 Write block 1358 len 20480 data: 10 9 9 7 7
24979 278/300 Flush
24980 279/300 Read block 564 len 40960
24981 280/300 Write block 4764 len 16384 data: 5 5 5 5
24982 281/300 Write block 74 len 40960 data: 6 7 8 7 6 6 6 4 4 3
24983 282/300 Write block 2382 len 40960 data: 6 5 5 5 4 2 4 5 5 6
24984 283/300 Read block 4719 len 40960
24985 284/300 Write block 2987 len 24576 data: 5 6 6 6 6 6
24986 285/300 Read block 2049 len 36864
24987 286/300 Read block 4727 len 16384
24988 287/300 Read block 2629 len 28672
24989 288/300 Write block 2387 len 28672 data: 3 5 6 6 7 5 5
24990 289/300 Read block 1362 len 4096
24991 290/300 Read block 1836 len 24576
24992 291/300 Write block 4424 len 36864 data: 8 8 7 7 5 5 6 5 6
24993 292/300 Read block 594 len 40960
24994 293/300 Write block 2921 len 40960 data: 6 6 4 3 3 5 5 6 7 10
24995 294/300 Write block 592 len 32768 data: 9 7 9 9 8 6 6 5
24996 295/300 Flush
24997 296/300 Read block 2065 len 32768
24998 297/300 Write block 3738 len 24576 data: 3 4 5 6 6 4
24999 298/300 Flush
25000 299/300 Write block 2483 len 16384 data: 5 5 6 6
25001 300/300 Read block 1900 len 28672
25002 301/300 Read block 3728 len 12288
25003 Main task has work to do, go find it
25004 got dsc Start(2)
25005 start 2
25006 Mar 17 22:21:52.811 INFO request completed, latency_us: 216, response_code: 204, uri: /start/cid/2, method: POST, req_id: fcbea61a-55b5-4912-bc5e-de14e922960b, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
25007 Replay: started 2, returned:Ok(())
25008 ----------------------------------------------------------------
25009 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25010 States: Active Active Offline
25011 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25012 7340 Acked Write 8 Done Done Sent false
25013 7341 Acked Read 3 Done Done Sent false
25014 7342 Acked Flush 0 Done Done Sent false
25015 7343 Acked Write 4 Done Done Sent false
25016 7344 Acked Read 1 Done Done Sent false
25017 7345 Acked Write 1 Done Done Sent false
25018 7346 Acked Write 7 Done Done Sent false
25019 7347 Acked Read 1 Done Done Sent false
25020 7348 Acked Flush 0 Done Done Sent false
25021 7349 Acked Write 8 Done Done Sent false
25022 7350 Acked Read 8 Done Done Sent false
25023 7351 Acked Write 10 Done Done Sent false
25024 7352 Acked Write 9 Done Done Sent false
25025 7353 Acked Read 9 Done Done Sent false
25026 7354 Acked Read 8 Done Done Sent false
25027 7355 Acked Read 1 Done Done Sent false
25028 7356 Acked Read 2 Done Done Sent false
25029 7357 Acked Read 8 Done Done Sent false
25030 7358 Acked Write 7 Done Done Sent false
25031 7359 Acked Flush 0 Done Done Sent false
25032 7360 Acked Read 3 Done Done Sent false
25033 7361 Acked Read 1 Done Done Sent false
25034 7362 Acked Read 9 Done Done Sent false
25035 7363 Acked Write 10 Done Done Sent false
25036 7364 Acked Read 1 Done Done Sent false
25037 7365 Acked Write 8 Done Done Sent false
25038 7366 Acked Flush 0 Done Done Sent false
25039 7367 Acked Read 8 Done Done Sent false
25040 7368 Acked Flush 0 Done Done Sent false
25041 7369 Acked Write 10 Done Done Sent false
25042 7370 Acked Write 10 Done Done Sent false
25043 7371 Acked Read 8 Done Done Sent false
25044 7372 Acked Write 10 Done Done Sent false
25045 7373 Acked Read 4 Done Done Sent false
25046 7374 Acked Read 9 Done Done Sent false
25047 7375 Acked Write 6 Done Done Sent false
25048 7376 Acked Read 6 Done Done Sent false
25049 7377 Acked Write 9 Done Done Sent false
25050 7378 Acked Write 3 Done Done Sent false
25051 7379 Acked Read 9 Done Done Sent false
25052 7380 Acked Write 4 Done Done Sent false
25053 7381 Acked Write 3 Done Done Sent false
25054 7382 Acked Flush 0 Done Done Sent false
25055 7383 Acked Write 10 Done Done Sent false
25056 7384 Acked Write 5 Done Done Sent false
25057 7385 Acked Read 7 Done Done Sent false
25058 7386 Acked Write 6 Done Done Sent false
25059 7387 Acked Read 3 Done Done Sent false
25060 7388 Acked Write 3 Done Done Sent false
25061 7389 Acked Flush 0 Done Done Sent false
25062 7390 Acked Write 5 Done Done Sent false
25063 7391 Acked Read 9 Done Done Sent false
25064 7392 Acked Flush 0 Done Done Sent false
25065 7393 Acked Write 7 Done Done Sent false
25066 7394 Acked Read 7 Done Done Sent false
25067 7395 Acked Read 2 Done Done Sent false
25068 7396 Acked Read 3 Done Done Sent false
25069 7397 Acked Read 4 Done Done Sent false
25070 7398 Acked Read 2 Done Done Sent false
25071 7399 Acked Read 5 Done Done Sent false
25072 7400 Acked Read 6 Done Done Sent false
25073 7401 Acked Write 2 Done Done Sent false
25074 7402 Acked Read 8 Done Done Sent false
25075 7403 Acked Read 5 Done Done Sent false
25076 7404 Acked Read 1 Done Done Sent false
25077 7405 Acked Read 1 Done Done Sent false
25078 7406 Acked Write 10 Done Done Sent false
25079 7407 Acked Write 2 Done Done Sent false
25080 7408 Acked Read 1 Done Done Sent false
25081 7409 Acked Write 7 Done Done Sent false
25082 7410 Acked Read 7 Done Done Sent false
25083 7411 Acked Read 7 Done Done Sent false
25084 7412 Acked Read 6 Done Done Sent false
25085 7413 Acked Read 4 Done Done Sent false
25086 7414 Acked Write 5 Done Done Sent false
25087 7415 Acked Flush 0 Done Done Sent false
25088 7416 Acked Read 3 Done Done Sent false
25089 7417 Acked Flush 0 Done Done Sent false
25090 7418 Acked Write 3 Done Done Sent false
25091 7419 Acked Write 9 Done Done Sent false
25092 7420 Acked Read 5 Done Done Sent false
25093 7421 Acked Write 7 Done Done Sent false
25094 7422 Acked Read 5 Done Done Sent false
25095 7423 Acked Read 6 Done Done Sent false
25096 7424 Acked Read 7 Done Done Sent false
25097 7425 Acked Write 9 Done Done Sent false
25098 7426 Acked Read 1 Done Done Sent false
25099 7427 Acked Read 5 Done Done Sent false
25100 7428 Acked Flush 0 Done Done Sent false
25101 7429 Acked Read 1 Done Done Sent false
25102 7430 Acked Write 2 Done Done Sent false
25103 7431 Acked Flush 0 Done Done Sent false
25104 7432 Acked Read 10 Done Done Sent false
25105 7433 Acked Write 7 Done Done Sent false
25106 7434 Acked Read 7 Done Done Sent false
25107 7435 Acked Read 5 Done Done Sent false
25108 7436 Acked Write 4 Done Done Sent false
25109 7437 Acked Read 2 Done Done Sent false
25110 7438 Acked Write 7 Done Done Sent false
25111 7439 Acked Read 8 Done Done Sent false
25112 7440 Acked Write 8 Done Done Sent false
25113 7441 Acked Read 8 Done Done Sent false
25114 7442 Acked Write 1 Done Done Sent false
25115 7443 Acked Flush 0 Done Done Sent false
25116 7444 Acked Read 5 Done Done Sent false
25117 7445 Acked Write 1 Done Done Sent false
25118 7446 Acked Flush 0 Done Done Sent false
25119 7447 Acked Write 5 Done Done Sent false
25120 7448 Acked Read 4 Done Done Sent false
25121 7449 Acked Write 7 Done Done Sent false
25122 7450 Acked Read 9 Done Done Sent false
25123 7451 Acked Write 4 Done Done Sent false
25124 7452 Acked Write 4 Done Done Sent false
25125 7453 Acked Read 5 Done Done Sent false
25126 7454 Acked Write 9 Done Done Sent false
25127 7455 Acked Read 6 Done Done Sent false
25128 7456 Acked Write 1 Done Done Sent false
25129 7457 Acked Read 3 Done Done Sent false
25130 7458 Acked Read 9 Done Done Sent false
25131 7459 Acked Flush 0 Done Done Sent false
25132 7460 Acked Read 8 Done Done Sent false
25133 7461 Acked Read 5 Done Done Sent false
25134 7462 Acked Read 7 Done Done Sent false
25135 7463 Acked Read 2 Done Done Sent false
25136 7464 Acked Read 10 Done Done Sent false
25137 7465 Acked Read 10 Done Done Sent false
25138 7466 Acked Write 10 Done Done Sent false
25139 7467 Acked Write 8 Done Done Sent false
25140 7468 Acked Read 3 Done Done Sent false
25141 7469 Acked Read 4 Done Done Sent false
25142 7470 Acked Read 10 Done Done Sent false
25143 7471 Acked Write 8 Done Done Sent false
25144 7472 Acked Write 1 Done Done Sent false
25145 7473 Acked Read 2 Done Done Sent false
25146 7474 Acked Read 6 Done Done Sent false
25147 7475 Acked Read 9 Done Done Sent false
25148 7476 Acked Read 5 Done Done Sent false
25149 7477 Acked Write 5 Done Done Sent false
25150 7478 Acked Write 1 Done Done Sent false
25151 7479 Acked Read 1 Done Done Sent false
25152 7480 Acked Write 7 Done Done Sent false
25153 7481 Acked Write 7 Done Done Sent false
25154 7482 Acked Flush 0 Done Done Sent false
25155 7483 Acked Read 10 Done Done Sent false
25156 7484 Acked Write 10 Done Done Sent false
25157 7485 Acked Write 4 Done Done Sent false
25158 7486 Acked Write 3 Done Done Sent false
25159 7487 Acked Read 2 Done Done Sent false
25160 7488 Acked Read 3 Done Done Sent false
25161 7489 Acked Write 4 Done Done Sent false
25162 7490 Acked Write 7 Done Done Sent false
25163 7491 Acked Read 7 Done Done Sent false
25164 7492 Acked Write 3 Done Done Sent false
25165 7493 Acked Flush 0 Done Done Sent false
25166 7494 Acked Read 6 Done Done Sent false
25167 7495 Acked Write 9 Done Done Sent false
25168 7496 Acked Read 3 Done Done Sent false
25169 7497 Acked Write 2 Done Done Sent false
25170 7498 Acked Read 1 Done Done Sent false
25171 7499 Acked Read 10 Done Done Sent false
25172 7500 Acked Read 6 Done Done Sent false
25173 7501 Acked Read 2 Done Done Sent false
25174 7502 Acked Read 7 Done Done Sent false
25175 7503 Acked Write 2 Done Done Sent false
25176 7504 Acked Write 8 Done Done Sent false
25177 7505 Acked Write 10 Done Done Sent false
25178 7506 Acked Write 2 Done Done Sent false
25179 7507 Acked Read 2 Done Done Sent false
25180 7508 Acked Read 3 Done Done Sent false
25181 7509 Acked Read 4 Done Done Sent false
25182 7510 Acked Read 2 Done Done Sent false
25183 7511 Acked Read 4 Done Done Sent false
25184 7512 Acked Read 8 Done Done Sent false
25185 7513 Acked Write 1 Done Done Sent false
25186 7514 Acked Read 6 Done Done Sent false
25187 7515 Acked Write 10 Done Done Sent false
25188 7516 Acked Flush 0 Done Done Sent false
25189 7517 Acked Write 1 Done Done Sent false
25190 7518 Acked Write 8 Done Done Sent false
25191 7519 Acked Flush 0 Done Done Sent false
25192 7520 Acked Read 3 Done Done Sent false
25193 7521 Acked Flush 0 Done Done Sent false
25194 7522 Acked Read 4 Done Done Sent false
25195 7523 Acked Read 5 Done Done Sent false
25196 7524 Acked Write 8 Done Done Sent false
25197 7525 Acked Write 4 Done Done Sent false
25198 7526 Acked Read 10 Done Done Sent false
25199 7527 Acked Write 7 Done Done Sent false
25200 7528 Acked Write 4 Done Done Sent false
25201 7529 Acked Write 7 Done Done Sent false
25202 7530 Acked Flush 0 Done Done Sent false
25203 7531 Acked Flush 0 Done Done Sent false
25204 7532 Acked Read 4 Done Done Sent false
25205 7533 Acked Write 6 Done Done Sent false
25206 7534 Acked Write 2 Done Done Sent false
25207 7535 Acked Read 1 Done Done Sent false
25208 7536 Acked Write 3 Done Done Sent false
25209 7537 Acked Read 2 Done Done Sent false
25210 7538 Acked Write 8 Done Done Sent false
25211 7539 Acked Write 5 Done Done Sent false
25212 7540 Acked Read 9 Done Done Sent false
25213 7541 Acked Write 6 Done Done Sent false
25214 7542 Acked Write 10 Done Done Sent false
25215 7543 Acked Read 1 Done Done Sent false
25216 7544 Acked Read 6 Done Done Sent false
25217 7545 Acked Write 4 Done Done Sent false
25218 7546 Acked Write 4 Done Done Sent false
25219 7547 Acked Read 7 Done Done Sent false
25220 7548 Acked Write 6 Done Done Sent false
25221 7549 Acked Write 2 Done Done Sent false
25222 7550 Acked Read 2 Done Done Sent false
25223 7551 Acked Write 9 Done Done Sent false
25224 7552 Acked Write 8 Done Done Sent false
25225 7553 Acked Write 10 Done Done Sent false
25226 7554 Acked Flush 0 Done Done Sent false
25227 7555 Acked Read 1 Done Done Sent false
25228 7556 Acked Write 8 Done Done Sent false
25229 7557 Acked Write 3 Done Done Sent false
25230 7558 Acked Read 5 Done Done Sent false
25231 7559 Acked Write 5 Done Done Sent false
25232 7560 Acked Write 8 Done Done Sent false
25233 7561 Acked Write 5 Done Done Sent false
25234 7562 Acked Read 4 Done Done Sent false
25235 7563 Acked Write 5 Done Done Sent false
25236 7564 Acked Flush 0 Done Done Sent false
25237 7565 Acked Read 1 Done Done Sent false
25238 7566 Acked Read 2 Done Done Sent false
25239 7567 Acked Flush 0 Done Done Sent false
25240 7568 Acked Read 4 Done Done Sent false
25241 7569 Acked Read 1 Done Done Sent false
25242 7570 Acked Read 10 Done Done Sent false
25243 7571 Acked Read 2 Done Done Sent false
25244 7572 Acked Read 8 Done Done Sent false
25245 7573 Acked Read 2 Done Done Sent false
25246 7574 Acked Write 9 Done Done Sent false
25247 7575 Acked Read 7 Done Done Sent false
25248 7576 Acked Read 2 Done Done Sent false
25249 7577 Acked Read 2 Done Done Sent false
25250 7578 Acked Read 6 Done Done Sent false
25251 7579 Acked Read 1 Done Done Sent false
25252 7580 Acked Read 6 Done Done Sent false
25253 7581 Acked Read 7 Done Done Sent false
25254 7582 Acked Read 2 Done Done Sent false
25255 7583 Acked Flush 0 Done Done Sent false
25256 7584 Acked Read 9 Done Done Sent false
25257 7585 Acked Write 2 Done Done Sent false
25258 7586 Acked Write 3 Done Done Sent false
25259 7587 Acked Write 10 Done Done Sent false
25260 7588 Acked Read 1 Done Done Sent false
25261 7589 Acked Write 6 Done Done Sent false
25262 7590 Acked Write 3 Done Done Sent false
25263 7591 Acked Read 6 Done Done Sent false
25264 7592 Acked Read 8 Done Done Sent false
25265 7593 Acked Write 6 Done Done Sent false
25266 7594 Acked Write 5 Done Done Sent false
25267 7595 Acked Read 4 Done Done Sent false
25268 7596 Acked Read 10 Done Done Sent false
25269 7597 Acked Read 8 Done Done Sent false
25270 7598 Acked Flush 0 Done Done Sent false
25271 7599 Acked Read 6 Done Done Sent false
25272 7600 Acked Write 4 Done Done Sent false
25273 7601 Acked Read 2 Done Done Sent false
25274 7602 Acked Read 1 Done Done Sent false
25275 7603 Acked Write 6 Done Done Sent false
25276 7604 Acked Write 4 Done Done Sent false
25277 7605 Acked Flush 0 Done Done Sent false
25278 7606 Acked Read 1 Done Done Sent false
25279 7607 Acked Read 10 Done Done Sent false
25280 7608 Acked Write 5 Done Done Sent false
25281 7609 Acked Read 9 Done Done Sent false
25282 7610 Acked Read 1 Done Done Sent false
25283 7611 Acked Flush 0 Done Done Sent false
25284 7612 Acked Flush 0 Done Done Sent false
25285 7613 Acked Write 1 Done Done Sent false
25286 7614 Acked Read 7 Done Done Sent false
25287 7615 Acked Read 2 Done Done Sent false
25288 7616 Acked Write 5 Done Done Sent false
25289 7617 Acked Flush 0 Done Done Sent false
25290 7618 Acked Read 10 Done Done Sent false
25291 7619 Acked Write 4 Done Done Sent false
25292 7620 Acked Write 10 Done Done Sent false
25293 7621 Acked Write 10 Done Done Sent false
25294 7622 Acked Read 10 Done Done Sent false
25295 7623 Acked Write 6 Done Done Sent false
25296 7624 Acked Read 9 Done Done Sent false
25297 7625 Acked Read 4 Done Done Sent false
25298 7626 Acked Read 7 Done Done Sent false
25299 7627 Acked Write 7 Done Done Sent false
25300 7628 Acked Read 1 Done Done Sent false
25301 7629 Acked Read 6 Done Done Sent false
25302 7630 Acked Write 9 Done Done Sent false
25303 7631 Acked Read 10 Done Done Sent false
25304 7632 Acked Write 10 Done Done Sent false
25305 7633 Acked Write 8 Done Done Sent false
25306 7634 Acked Flush 0 Done Done Sent false
25307 7635 Acked Read 8 Done Done Sent false
25308 7636 Acked Write 6 Done Done Sent false
25309 7637 Acked Flush 0 Done Done Sent false
25310 7638 Acked Write 4 Done Done Sent false
25311 7639 Acked Read 7 Done Done Sent false
25312 7640 Acked Read 3 Done Done Sent false
25313 STATES DS:0 DS:1 DS:2 TOTAL
25314 Sent 0 0 301 301
25315 Done 301 301 0 602
25316 Skipped 0 0 0 0
25317 Error 0 0 0 0
25318 Last Flush: Some(JobId(7637)) Some(JobId(7637)) Some(JobId(7339))
25319 Downstairs last five completed: 7339 7338 7337 7336 7335
25320 Upstairs last five completed: 7640 7639 7638 7637 7636
25321 CLIENT: Up:0 ds:301 act:2
253222025-03-17T22:21:53.313ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
253232025-03-17T22:21:54.130ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
25324 [8830] got start action
25325 [8830] I am going to restart
25326 Starting downstairs at port 8830
25327 Make output file at "/tmp/dsc/downstairs-8830.txt"
25328 [8830][2] reports Starting
25329 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1228)
25330 [8830][2] reports Running
253312025-03-17T22:21:55.132ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
253322025-03-17T22:21:56.134ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
25333 ----------------------------------------------------------------
25334 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25335 States: Active Active Offline
25336 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25337 7340 Acked Write 8 Done Done Sent false
25338 7341 Acked Read 3 Done Done Sent false
25339 7342 Acked Flush 0 Done Done Sent false
25340 7343 Acked Write 4 Done Done Sent false
25341 7344 Acked Read 1 Done Done Sent false
25342 7345 Acked Write 1 Done Done Sent false
25343 7346 Acked Write 7 Done Done Sent false
25344 7347 Acked Read 1 Done Done Sent false
25345 7348 Acked Flush 0 Done Done Sent false
25346 7349 Acked Write 8 Done Done Sent false
25347 7350 Acked Read 8 Done Done Sent false
25348 7351 Acked Write 10 Done Done Sent false
25349 7352 Acked Write 9 Done Done Sent false
25350 7353 Acked Read 9 Done Done Sent false
25351 7354 Acked Read 8 Done Done Sent false
25352 7355 Acked Read 1 Done Done Sent false
25353 7356 Acked Read 2 Done Done Sent false
25354 7357 Acked Read 8 Done Done Sent false
25355 7358 Acked Write 7 Done Done Sent false
25356 7359 Acked Flush 0 Done Done Sent false
25357 7360 Acked Read 3 Done Done Sent false
25358 7361 Acked Read 1 Done Done Sent false
25359 7362 Acked Read 9 Done Done Sent false
25360 7363 Acked Write 10 Done Done Sent false
25361 7364 Acked Read 1 Done Done Sent false
25362 7365 Acked Write 8 Done Done Sent false
25363 7366 Acked Flush 0 Done Done Sent false
25364 7367 Acked Read 8 Done Done Sent false
25365 7368 Acked Flush 0 Done Done Sent false
25366 7369 Acked Write 10 Done Done Sent false
25367 7370 Acked Write 10 Done Done Sent false
25368 7371 Acked Read 8 Done Done Sent false
25369 7372 Acked Write 10 Done Done Sent false
25370 7373 Acked Read 4 Done Done Sent false
25371 7374 Acked Read 9 Done Done Sent false
25372 7375 Acked Write 6 Done Done Sent false
25373 7376 Acked Read 6 Done Done Sent false
25374 7377 Acked Write 9 Done Done Sent false
25375 7378 Acked Write 3 Done Done Sent false
25376 7379 Acked Read 9 Done Done Sent false
25377 7380 Acked Write 4 Done Done Sent false
25378 7381 Acked Write 3 Done Done Sent false
25379 7382 Acked Flush 0 Done Done Sent false
25380 7383 Acked Write 10 Done Done Sent false
25381 7384 Acked Write 5 Done Done Sent false
25382 7385 Acked Read 7 Done Done Sent false
25383 7386 Acked Write 6 Done Done Sent false
25384 7387 Acked Read 3 Done Done Sent false
25385 7388 Acked Write 3 Done Done Sent false
25386 7389 Acked Flush 0 Done Done Sent false
25387 7390 Acked Write 5 Done Done Sent false
25388 7391 Acked Read 9 Done Done Sent false
25389 7392 Acked Flush 0 Done Done Sent false
25390 7393 Acked Write 7 Done Done Sent false
25391 7394 Acked Read 7 Done Done Sent false
25392 7395 Acked Read 2 Done Done Sent false
25393 7396 Acked Read 3 Done Done Sent false
25394 7397 Acked Read 4 Done Done Sent false
25395 7398 Acked Read 2 Done Done Sent false
25396 7399 Acked Read 5 Done Done Sent false
25397 7400 Acked Read 6 Done Done Sent false
25398 7401 Acked Write 2 Done Done Sent false
25399 7402 Acked Read 8 Done Done Sent false
25400 7403 Acked Read 5 Done Done Sent false
25401 7404 Acked Read 1 Done Done Sent false
25402 7405 Acked Read 1 Done Done Sent false
25403 7406 Acked Write 10 Done Done Sent false
25404 7407 Acked Write 2 Done Done Sent false
25405 7408 Acked Read 1 Done Done Sent false
25406 7409 Acked Write 7 Done Done Sent false
25407 7410 Acked Read 7 Done Done Sent false
25408 7411 Acked Read 7 Done Done Sent false
25409 7412 Acked Read 6 Done Done Sent false
25410 7413 Acked Read 4 Done Done Sent false
25411 7414 Acked Write 5 Done Done Sent false
25412 7415 Acked Flush 0 Done Done Sent false
25413 7416 Acked Read 3 Done Done Sent false
25414 7417 Acked Flush 0 Done Done Sent false
25415 7418 Acked Write 3 Done Done Sent false
25416 7419 Acked Write 9 Done Done Sent false
25417 7420 Acked Read 5 Done Done Sent false
25418 7421 Acked Write 7 Done Done Sent false
25419 7422 Acked Read 5 Done Done Sent false
25420 7423 Acked Read 6 Done Done Sent false
25421 7424 Acked Read 7 Done Done Sent false
25422 7425 Acked Write 9 Done Done Sent false
25423 7426 Acked Read 1 Done Done Sent false
25424 7427 Acked Read 5 Done Done Sent false
25425 7428 Acked Flush 0 Done Done Sent false
25426 7429 Acked Read 1 Done Done Sent false
25427 7430 Acked Write 2 Done Done Sent false
25428 7431 Acked Flush 0 Done Done Sent false
25429 7432 Acked Read 10 Done Done Sent false
25430 7433 Acked Write 7 Done Done Sent false
25431 7434 Acked Read 7 Done Done Sent false
25432 7435 Acked Read 5 Done Done Sent false
25433 7436 Acked Write 4 Done Done Sent false
25434 7437 Acked Read 2 Done Done Sent false
25435 7438 Acked Write 7 Done Done Sent false
25436 7439 Acked Read 8 Done Done Sent false
25437 7440 Acked Write 8 Done Done Sent false
25438 7441 Acked Read 8 Done Done Sent false
25439 7442 Acked Write 1 Done Done Sent false
25440 7443 Acked Flush 0 Done Done Sent false
25441 7444 Acked Read 5 Done Done Sent false
25442 7445 Acked Write 1 Done Done Sent false
25443 7446 Acked Flush 0 Done Done Sent false
25444 7447 Acked Write 5 Done Done Sent false
25445 7448 Acked Read 4 Done Done Sent false
25446 7449 Acked Write 7 Done Done Sent false
25447 7450 Acked Read 9 Done Done Sent false
25448 7451 Acked Write 4 Done Done Sent false
25449 7452 Acked Write 4 Done Done Sent false
25450 7453 Acked Read 5 Done Done Sent false
25451 7454 Acked Write 9 Done Done Sent false
25452 7455 Acked Read 6 Done Done Sent false
25453 7456 Acked Write 1 Done Done Sent false
25454 7457 Acked Read 3 Done Done Sent false
25455 7458 Acked Read 9 Done Done Sent false
25456 7459 Acked Flush 0 Done Done Sent false
25457 7460 Acked Read 8 Done Done Sent false
25458 7461 Acked Read 5 Done Done Sent false
25459 7462 Acked Read 7 Done Done Sent false
25460 7463 Acked Read 2 Done Done Sent false
25461 7464 Acked Read 10 Done Done Sent false
25462 7465 Acked Read 10 Done Done Sent false
25463 7466 Acked Write 10 Done Done Sent false
25464 7467 Acked Write 8 Done Done Sent false
25465 7468 Acked Read 3 Done Done Sent false
25466 7469 Acked Read 4 Done Done Sent false
25467 7470 Acked Read 10 Done Done Sent false
25468 7471 Acked Write 8 Done Done Sent false
25469 7472 Acked Write 1 Done Done Sent false
25470 7473 Acked Read 2 Done Done Sent false
25471 7474 Acked Read 6 Done Done Sent false
25472 7475 Acked Read 9 Done Done Sent false
25473 7476 Acked Read 5 Done Done Sent false
25474 7477 Acked Write 5 Done Done Sent false
25475 7478 Acked Write 1 Done Done Sent false
25476 7479 Acked Read 1 Done Done Sent false
25477 7480 Acked Write 7 Done Done Sent false
25478 7481 Acked Write 7 Done Done Sent false
25479 7482 Acked Flush 0 Done Done Sent false
25480 7483 Acked Read 10 Done Done Sent false
25481 7484 Acked Write 10 Done Done Sent false
25482 7485 Acked Write 4 Done Done Sent false
25483 7486 Acked Write 3 Done Done Sent false
25484 7487 Acked Read 2 Done Done Sent false
25485 7488 Acked Read 3 Done Done Sent false
25486 7489 Acked Write 4 Done Done Sent false
25487 7490 Acked Write 7 Done Done Sent false
25488 7491 Acked Read 7 Done Done Sent false
25489 7492 Acked Write 3 Done Done Sent false
25490 7493 Acked Flush 0 Done Done Sent false
25491 7494 Acked Read 6 Done Done Sent false
25492 7495 Acked Write 9 Done Done Sent false
25493 7496 Acked Read 3 Done Done Sent false
25494 7497 Acked Write 2 Done Done Sent false
25495 7498 Acked Read 1 Done Done Sent false
25496 7499 Acked Read 10 Done Done Sent false
25497 7500 Acked Read 6 Done Done Sent false
25498 7501 Acked Read 2 Done Done Sent false
25499 7502 Acked Read 7 Done Done Sent false
25500 7503 Acked Write 2 Done Done Sent false
25501 7504 Acked Write 8 Done Done Sent false
25502 7505 Acked Write 10 Done Done Sent false
25503 7506 Acked Write 2 Done Done Sent false
25504 7507 Acked Read 2 Done Done Sent false
25505 7508 Acked Read 3 Done Done Sent false
25506 7509 Acked Read 4 Done Done Sent false
25507 7510 Acked Read 2 Done Done Sent false
25508 7511 Acked Read 4 Done Done Sent false
25509 7512 Acked Read 8 Done Done Sent false
25510 7513 Acked Write 1 Done Done Sent false
25511 7514 Acked Read 6 Done Done Sent false
25512 7515 Acked Write 10 Done Done Sent false
25513 7516 Acked Flush 0 Done Done Sent false
25514 7517 Acked Write 1 Done Done Sent false
25515 7518 Acked Write 8 Done Done Sent false
25516 7519 Acked Flush 0 Done Done Sent false
25517 7520 Acked Read 3 Done Done Sent false
25518 7521 Acked Flush 0 Done Done Sent false
25519 7522 Acked Read 4 Done Done Sent false
25520 7523 Acked Read 5 Done Done Sent false
25521 7524 Acked Write 8 Done Done Sent false
25522 7525 Acked Write 4 Done Done Sent false
25523 7526 Acked Read 10 Done Done Sent false
25524 7527 Acked Write 7 Done Done Sent false
25525 7528 Acked Write 4 Done Done Sent false
25526 7529 Acked Write 7 Done Done Sent false
25527 7530 Acked Flush 0 Done Done Sent false
25528 7531 Acked Flush 0 Done Done Sent false
25529 7532 Acked Read 4 Done Done Sent false
25530 7533 Acked Write 6 Done Done Sent false
25531 7534 Acked Write 2 Done Done Sent false
25532 7535 Acked Read 1 Done Done Sent false
25533 7536 Acked Write 3 Done Done Sent false
25534 7537 Acked Read 2 Done Done Sent false
25535 7538 Acked Write 8 Done Done Sent false
25536 7539 Acked Write 5 Done Done Sent false
25537 7540 Acked Read 9 Done Done Sent false
25538 7541 Acked Write 6 Done Done Sent false
25539 7542 Acked Write 10 Done Done Sent false
25540 7543 Acked Read 1 Done Done Sent false
25541 7544 Acked Read 6 Done Done Sent false
25542 7545 Acked Write 4 Done Done Sent false
25543 7546 Acked Write 4 Done Done Sent false
25544 7547 Acked Read 7 Done Done Sent false
25545 7548 Acked Write 6 Done Done Sent false
25546 7549 Acked Write 2 Done Done Sent false
25547 7550 Acked Read 2 Done Done Sent false
25548 7551 Acked Write 9 Done Done Sent false
25549 7552 Acked Write 8 Done Done Sent false
25550 7553 Acked Write 10 Done Done Sent false
25551 7554 Acked Flush 0 Done Done Sent false
25552 7555 Acked Read 1 Done Done Sent false
25553 7556 Acked Write 8 Done Done Sent false
25554 7557 Acked Write 3 Done Done Sent false
25555 7558 Acked Read 5 Done Done Sent false
25556 7559 Acked Write 5 Done Done Sent false
25557 7560 Acked Write 8 Done Done Sent false
25558 7561 Acked Write 5 Done Done Sent false
25559 7562 Acked Read 4 Done Done Sent false
25560 7563 Acked Write 5 Done Done Sent false
25561 7564 Acked Flush 0 Done Done Sent false
25562 7565 Acked Read 1 Done Done Sent false
25563 7566 Acked Read 2 Done Done Sent false
25564 7567 Acked Flush 0 Done Done Sent false
25565 7568 Acked Read 4 Done Done Sent false
25566 7569 Acked Read 1 Done Done Sent false
25567 7570 Acked Read 10 Done Done Sent false
25568 7571 Acked Read 2 Done Done Sent false
25569 7572 Acked Read 8 Done Done Sent false
25570 7573 Acked Read 2 Done Done Sent false
25571 7574 Acked Write 9 Done Done Sent false
25572 7575 Acked Read 7 Done Done Sent false
25573 7576 Acked Read 2 Done Done Sent false
25574 7577 Acked Read 2 Done Done Sent false
25575 7578 Acked Read 6 Done Done Sent false
25576 7579 Acked Read 1 Done Done Sent false
25577 7580 Acked Read 6 Done Done Sent false
25578 7581 Acked Read 7 Done Done Sent false
25579 7582 Acked Read 2 Done Done Sent false
25580 7583 Acked Flush 0 Done Done Sent false
25581 7584 Acked Read 9 Done Done Sent false
25582 7585 Acked Write 2 Done Done Sent false
25583 7586 Acked Write 3 Done Done Sent false
25584 7587 Acked Write 10 Done Done Sent false
25585 7588 Acked Read 1 Done Done Sent false
25586 7589 Acked Write 6 Done Done Sent false
25587 7590 Acked Write 3 Done Done Sent false
25588 7591 Acked Read 6 Done Done Sent false
25589 7592 Acked Read 8 Done Done Sent false
25590 7593 Acked Write 6 Done Done Sent false
25591 7594 Acked Write 5 Done Done Sent false
25592 7595 Acked Read 4 Done Done Sent false
25593 7596 Acked Read 10 Done Done Sent false
25594 7597 Acked Read 8 Done Done Sent false
25595 7598 Acked Flush 0 Done Done Sent false
25596 7599 Acked Read 6 Done Done Sent false
25597 7600 Acked Write 4 Done Done Sent false
25598 7601 Acked Read 2 Done Done Sent false
25599 7602 Acked Read 1 Done Done Sent false
25600 7603 Acked Write 6 Done Done Sent false
25601 7604 Acked Write 4 Done Done Sent false
25602 7605 Acked Flush 0 Done Done Sent false
25603 7606 Acked Read 1 Done Done Sent false
25604 7607 Acked Read 10 Done Done Sent false
25605 7608 Acked Write 5 Done Done Sent false
25606 7609 Acked Read 9 Done Done Sent false
25607 7610 Acked Read 1 Done Done Sent false
25608 7611 Acked Flush 0 Done Done Sent false
25609 7612 Acked Flush 0 Done Done Sent false
25610 7613 Acked Write 1 Done Done Sent false
25611 7614 Acked Read 7 Done Done Sent false
25612 7615 Acked Read 2 Done Done Sent false
25613 7616 Acked Write 5 Done Done Sent false
25614 7617 Acked Flush 0 Done Done Sent false
25615 7618 Acked Read 10 Done Done Sent false
25616 7619 Acked Write 4 Done Done Sent false
25617 7620 Acked Write 10 Done Done Sent false
25618 7621 Acked Write 10 Done Done Sent false
25619 7622 Acked Read 10 Done Done Sent false
25620 7623 Acked Write 6 Done Done Sent false
25621 7624 Acked Read 9 Done Done Sent false
25622 7625 Acked Read 4 Done Done Sent false
25623 7626 Acked Read 7 Done Done Sent false
25624 7627 Acked Write 7 Done Done Sent false
25625 7628 Acked Read 1 Done Done Sent false
25626 7629 Acked Read 6 Done Done Sent false
25627 7630 Acked Write 9 Done Done Sent false
25628 7631 Acked Read 10 Done Done Sent false
25629 7632 Acked Write 10 Done Done Sent false
25630 7633 Acked Write 8 Done Done Sent false
25631 7634 Acked Flush 0 Done Done Sent false
25632 7635 Acked Read 8 Done Done Sent false
25633 7636 Acked Write 6 Done Done Sent false
25634 7637 Acked Flush 0 Done Done Sent false
25635 7638 Acked Write 4 Done Done Sent false
25636 7639 Acked Read 7 Done Done Sent false
25637 7640 Acked Read 3 Done Done Sent false
25638 STATES DS:0 DS:1 DS:2 TOTAL
25639 Sent 0 0 301 301
25640 Done 301 301 0 602
25641 Skipped 0 0 0 0
25642 Error 0 0 0 0
25643 Last Flush: Some(JobId(7637)) Some(JobId(7637)) Some(JobId(7339))
25644 Downstairs last five completed: 7339 7338 7337 7336 7335
25645 Upstairs last five completed: 7640 7639 7638 7637 7636
25646 CLIENT: Up:0 ds:301 act:2
256472025-03-17T22:21:57.555ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
256482025-03-17T22:21:58.136ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
256492025-03-17T22:21:58.561ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
256502025-03-17T22:21:58.561ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:33551) = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
256512025-03-17T22:21:58.562ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
256522025-03-17T22:21:58.563ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
256532025-03-17T22:21:58.563ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
256542025-03-17T22:21:58.563ZINFOcrucible: send last flush ID to this DS: Some(JobId(7339)) = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
256552025-03-17T22:21:58.563ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
256562025-03-17T22:21:58.563ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(7339)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
256572025-03-17T22:21:58.563ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(7339)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
25658 ----------------------------------------------------------------
25659 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25660 Downstairs last five completed: 7641 7640 7639 7638 7637
25661 Upstairs last five completed: 7641 7640 7639 7638 7637
25662 CLIENT: Up:0 ds:0 act:3
25663 Replay: All jobs finished
25664 Main task has work to do, go find it
25665 got dsc Stop(0)
25666 stop 0
25667 Mar 17 22:22:00.816 INFO request completed, latency_us: 368, response_code: 204[8810] Got stop action so:false kr:false
25668 , uri: /stop/cid/0, method: POST, req_id: 49b6f090-7198-40f6-bbaf-99b464616b80, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
256692025-03-17T22:22:00.817ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
256702025-03-17T22:22:00.817ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
25671 Mar 17 22:22:00.817 INFO request completed, latency_us: 122, response_code: 200{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:22:00.817406534Z","hostname":"w-01JPK2S810SFAA9787XEG8GFX5","pid":1206,"session_id":"0f8310d7-099e-4ead-8484-91ddc162d307"}
25672 , uri: /state/cid/0, method: GET, req_id: 3548626a-6841-4f97-bdd8-2370bea0e08e, remote_addr: 127.0{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:22:00.817470819Z","hostname":"w-01JPK2S810SFAA9787XEG8GFX5","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"0f8310d7-099e-4ead-8484-91ddc162d307"}
25673 .0.1:60423, local_addr: 127.0.0.1:9998
25674 [8810] Exited with: ExitStatus(unix_wait_status(9))
25675 [8810][0] reports Exit
25676 Mar 17 22:22:04.819 INFO request completed, latency_us: 217, response_code: 200, uri: /state/cid/0, method: GET, req_id: f100350b-060f-4f02-8297-7c0fa3e8cb45, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
25677 001/300 Write block 1646 len 36864 data: 7 7 6 4 2 3 3 4 4
25678 002/300 Write block 2770 len 4096 data: 7
25679 003/300 Write block 2935 len 36864 data: 6 6 5 5 6 8 9 8 9
25680 004/300 Read block 1133 len 36864
25681 005/300 Write block 1628 len 8192 data: 2 2
25682 006/300 Write block 2643 len 8192 data: 3 3
25683 007/300 Write block 650 len 16384 data: 5 3 4 3
25684 008/300 Write block 1510 len 4096 data: 4
25685 009/300 Read block 232 len 36864
25686 010/300 Write block 234 len 20480 data: 7 7 7 8 7
25687 011/300 Write block 425 len 12288 data: 5 4 5
25688 012/300 Write block 2311 len 40960 data: 4 4 4 3 4 4 3 2 2 3
25689 013/300 Flush
25690 014/300 Write block 1396 len 28672 data: 6 6 6 7 7 7 7
25691 015/300 Write block 431 len 8192 data: 3 3
25692 016/300 Read block 4924 len 40960
25693 017/300 Write block 2191 len 20480 data: 4 4 4 5 5
25694 018/300 Read block 4924 len 20480
25695 019/300 Read block 4476 len 40960
25696 020/300 Write block 4742 len 16384 data: 4 3 4 5
25697 021/300 Write block 1916 len 24576 data: 4 4 4 4 3 6
25698 022/300 Read block 1264 len 16384
25699 023/300 Write block 4108 len 28672 data: 9 7 5 4 5 5 5
25700 024/300 Read block 3935 len 12288
25701 025/300 Read block 1606 len 8192
25702 026/300 Flush
25703 027/300 Flush
25704 028/300 Write block 2826 len 24576 data: 10 11 8 8 8 5
25705 029/300 Read block 2779 len 40960
25706 030/300 Read block 1838 len 40960
25707 031/300 Write block 1580 len 32768 data: 7 6 6 6 6 6 6 3
25708 032/300 Write block 3168 len 4096 data: 8
25709 033/300 Read block 2523 len 8192
25710 034/300 Read block 3518 len 28672
25711 035/300 Read block 2487 len 28672
25712 036/300 Flush
25713 037/300 Read block 3752 len 16384
25714 038/300 Write block 2128 len 16384 data: 3 3 4 6
25715 039/300 Flush
25716 040/300 Write block 4661 len 36864 data: 4 4 5 4 4 4 4 3 3
25717 041/300 Read block 3946 len 36864
25718 042/300 Flush
25719 043/300 Flush
25720 044/300 Read block 3336 len 24576
25721 045/300 Flush
25722 046/300 Write block 4452 len 8192 data: 6 6
25723 047/300 Write block 2979 len 40960 data: 6 6 6 6 6 6 4 5 6 7
25724 048/300 Write block 4695 len 12288 data: 3 3 2
25725 049/300 Read block 4649 len 8192
25726 050/300 Read block 3921 len 28672
25727 051/300 Read block 4829 len 4096
25728 052/300 Read block 1119 len 4096
25729 053/300 Read block 4095 len 28672
25730 054/300 Write block 2506 len 12288 data: 3 3 5
25731 055/300 Read block 3894 len 12288
25732 056/300 Read block 4488 len 32768
25733 057/300 Write block 2897 len 28672 data: 7 7 6 8 6 5 5
25734 058/300 Read block 342 len 16384
25735 059/300 Read block 3496 len 28672
25736 060/300 Read block 417 len 24576
25737 061/300 Read block 2576 len 32768
25738 062/300 Read block 682 len 20480
25739 063/300 Read block 1345 len 24576
25740 064/300 Write block 1669 len 24576 data: 6 6 5 7 7 7
25741 065/300 Read block 3124 len 8192
25742 066/300 Read block 2053 len 32768
25743 067/300 Read block 1057 len 36864
25744 068/300 Read block 612 len 8192
25745 069/300 Read block 2246 len 32768
25746 070/300 Read block 3587 len 20480
25747 071/300 Write block 4636 len 36864 data: 5 5 5 5 3 4 4 3 4
25748 072/300 Write block 2497 len 16384 data: 7 8 8 7
25749 073/300 Read block 4235 len 36864
25750 074/300 Read block 803 len 16384
25751 075/300 Flush
25752 076/300 Read block 3306 len 28672
25753 077/300 Read block 2053 len 20480
25754 078/300 Read block 4343 len 36864
25755 079/300 Write block 2807 len 36864 data: 3 4 3 2 2 3 3 3 3
25756 080/300 Write block 330 len 8192 data: 5 5
25757 081/300 Read block 4283 len 36864
25758 082/300 Read block 755 len 28672
25759 083/300 Read block 3708 len 24576
25760 084/300 Read block 1190 len 32768
25761 085/300 Write block 4345 len 24576 data: 4 4 4 4 5 3
25762 086/300 Write block 4805 len 12288 data: 6 5 7
25763 087/300 Flush
25764 088/300 Flush
25765 089/300 Write block 4984 len 16384 data: 4 5 6 4
25766 090/300 Read block 3997 len 12288
25767 091/300 Write block 1152 len 20480 data: 7 7 6 5 7
25768 092/300 Read block 846 len 16384
25769 093/300 Read block 2711 len 16384
25770 094/300 Write block 3572 len 4096 data: 4
25771 095/300 Read block 2860 len 36864
25772 096/300 Read block 3679 len 40960
25773 097/300 Read block 4509 len 32768
25774 098/300 Read block 3534 len 28672
25775 099/300 Write block 1191 len 16384 data: 2 2 3 3
25776 100/300 Read block 2525 len 8192
25777 101/300 Write block 4398 len 4096 data: 6
25778 102/300 Flush
25779 103/300 Write block 33 len 40960 data: 7 6 6 6 4 4 4 4 4 4
25780 104/300 Read block 4543 len 12288
25781 105/300 Read block 3955 len 20480
25782 106/300 Read block 4335 len 8192
25783 107/300 Read block 4063 len 16384
25784 108/300 Write block 4093 len 40960 data: 6 5 7 5 5 6 7 4 4 3
25785 109/300 Write block 1502 len 16384 data: 5 6 7 6
25786 110/300 Read block 9 len 8192
25787 111/300 Read block 3976 len 36864
25788 112/300 Read block 628 len 40960
25789 113/300 Read block 1832 len 28672
25790 114/300 Write block 4261 len 28672 data: 6 5 4 3 4 6 5
25791 115/300 Write block 3242 len 28672 data: 5 5 5 4 4 6 7
25792 116/300 Read block 911 len 16384
25793 117/300 Write block 4142 len 24576 data: 6 5 5 6 7 8
25794 118/300 Write block 491 len 32768 data: 8 7 7 6 5 6 7 7
25795 119/300 Write block 3608 len 36864 data: 3 3 4 5 6 7 6 6 6
25796 120/300 Read block 173 len 24576
25797 121/300 Read block 4895 len 8192
25798 122/300 Read block 63 len 12288
25799 123/300 Flush
25800 124/300 Read block 2377 len 20480
25801 125/300 Write block 4315 len 8192 data: 4 4
25802 126/300 Write block 3870 len 40960 data: 4 5 4 3 3 3 3 3 3 3
25803 127/300 Write block 3525 len 4096 data: 4
25804 128/300 Flush
25805 129/300 Write block 2040 len 8192 data: 7 8
25806 130/300 Read block 4642 len 20480
25807 131/300 Read block 4551 len 40960
25808 132/300 Read block 4055 len 16384
25809 133/300 Write block 2261 len 36864 data: 2 3 3 3 4 4 5 5 5
25810 134/300 Read block 3275 len 28672
25811 135/300 Write block 4051 len 24576 data: 4 4 5 5 4 7
25812 136/300 Flush
25813 137/300 Read block 1420 len 28672
25814 138/300 Read block 4693 len 8192
25815 139/300 Write block 2888 len 24576 data: 6 7 6 4 5 5
25816 140/300 Write block 4495 len 4096 data: 3
25817 141/300 Write block 1850 len 40960 data: 5 6 6 6 4 4 4 5 6 5
25818 142/300 Read block 4664 len 4096
25819 143/300 Read block 3965 len 28672
25820 144/300 Write block 3352 len 8192 data: 6 7
25821 145/300 Read block 2405 len 8192
25822 146/300 Read block 2233 len 8192
25823 147/300 Read block 163 len 8192
25824 148/300 Read block 2181 len 12288
25825 149/300 Read block 1911 len 24576
25826 150/300 Read block 2643 len 24576
25827 151/300 Write block 377 len 20480 data: 5 4 4 4 3
25828 152/300 Read block 4510 len 20480
25829 153/300 Read block 2792 len 4096
25830 154/300 Read block 2838 len 16384
25831 155/300 Read block 2150 len 12288
25832 156/300 Flush
25833 157/300 Read block 3323 len 24576
25834 158/300 Read block 4392 len 16384
25835 159/300 Read block 3825 len 40960
25836 160/300 Flush
25837 161/300 Read block 601 len 24576
25838 162/300 Write block 4825 len 4096 data: 6
25839 163/300 Write block 815 len 32768 data: 5 4 3 3 3 2 2 2
25840 164/300 Write block 3742 len 20480 data: 7 5 4 6 6
25841 165/300 Write block 4062 len 4096 data: 7
25842 166/300 Write block 70 len 36864 data: 5 5 5 5 7 8 9 8 7
25843 167/300 Write block 3883 len 4096 data: 6
25844 168/300 Read block 2664 len 24576
25845 169/300 Write block 3471 len 4096 data: 3
25846 170/300 Read block 3677 len 36864
25847 171/300 Flush
25848 172/300 Read block 144 len 12288
25849 173/300 Read block 3846 len 20480
25850 174/300 Read block 4792 len 16384
25851 175/300 Read block 1450 len 20480
25852 176/300 Flush
25853 177/300 Read block 4468 len 4096
25854 178/300 Write block 1029 len 28672 data: 6 5 6 8 9 10 8
25855 179/300 Read block 3048 len 16384
25856 180/300 Read block 665 len 28672
25857 181/300 Read block 4007 len 4096
25858 182/300 Read block 926 len 8192
25859 183/300 Read block 3279 len 36864
25860 184/300 Read block 1341 len 12288
25861 185/300 Write block 3719 len 20480 data: 4 4 2 2 3
25862 186/300 Read block 1460 len 40960
25863 187/300 Read block 1866 len 40960
25864 188/300 Read block 4969 len 28672
25865 189/300 Read block 1253 len 28672
25866 190/300 Read block 3642 len 36864
25867 191/300 Read block 1560 len 16384
25868 192/300 Read block 2210 len 24576
25869 193/300 Read block 2498 len 20480
25870 194/300 Write block 4974 len 16384 data: 3 4 4 4
25871 195/300 Write block 4067 len 40960 data: 6 6 5 5 5 5 6 6 4 4
25872 196/300 Write block 2433 len 12288 data: 5 6 7
25873 197/300 Read block 627 len 8192
25874 198/300 Write block 2989 len 8192 data: 7 7
25875 199/300 Write block 3162 len 32768 data: 4 4 5 5 5 7 9 7
25876 200/300 Write block 2226 len 8192 data: 5 4
25877 201/300 Write block 4054 len 32768 data: 6 5 8 8 8 7 7 7
25878 202/300 Read block 1690 len 32768
25879 203/300 Read block 4453 len 24576
25880 204/300 Read block 4723 len 28672
25881 205/300 Write block 226 len 28672 data: 4 3 4 4 4 4 5
25882 206/300 Write block 1443 len 20480 data: 2 2 3 3 2
25883 207/300 Read block 1581 len 32768
25884 208/300 Read block 4180 len 8192
25885 209/300 Read block 393 len 4096
25886 210/300 Flush
25887 211/300 Write block 3284 len 36864 data: 3 3 4 5 5 6 6 6 6
25888 212/300 Write block 3953 len 32768 data: 6 5 4 4 5 3 3 4
25889 213/300 Write block 3117 len 36864 data: 4 5 6 5 5 5 6 6 6
25890 214/300 Read block 1123 len 28672
25891 215/300 Write block 2522 len 16384 data: 2 3 3 3
25892 216/300 Write block 1913 len 20480 data: 2 3 4 5 5
25893 217/300 Flush
25894 218/300 Write block 4170 len 28672 data: 6 6 6 5 5 3 4
25895 219/300 Read block 3150 len 8192
25896 220/300 Read block 771 len 32768
25897 221/300 Read block 4344 len 32768
25898 222/300 Read block 1367 len 36864
25899 223/300 Write block 1465 len 40960 data: 6 6 6 5 4 6 5 6 7 6
25900 224/300 Read block 3040 len 4096
25901 225/300 Read block 653 len 20480
25902 226/300 Flush
25903 227/300 Read block 2654 len 28672
25904 228/300 Read block 3825 len 28672
25905 229/300 Write block 982 len 32768 data: 4 4 4 5 5 5 7 7
25906 230/300 Write block 2485 len 36864 data: 7 7 7 7 7 7 7 6 6
25907 231/300 Write block 1458 len 12288 data: 5 7 6
25908 232/300 Read block 2185 len 4096
25909 233/300 Read block 1363 len 24576
25910 234/300 Write block 1778 len 8192 data: 5 4
25911 235/300 Write block 135 len 8192 data: 4 4
25912 236/300 Read block 2508 len 20480
25913 237/300 Write block 4995 len 16384 data: 5 4 5 4
25914 238/300 Write block 3738 len 24576 data: 4 5 6 7 8 6
25915 239/300 Write block 845 len 16384 data: 2 2 3 4
25916 240/300 Read block 3114 len 4096
25917 241/300 Write block 1076 len 20480 data: 9 9 7 8 7
25918 242/300 Flush
25919 243/300 Read block 3869 len 32768
25920 244/300 Read block 2362 len 36864
25921 245/300 Flush
25922 246/300 Read block 4904 len 16384
25923 247/300 Flush
25924 248/300 Write block 944 len 16384 data: 4 4 4 4
25925 249/300 Read block 231 len 4096
25926 250/300 Read block 1941 len 12288
25927 251/300 Write block 4197 len 24576 data: 4 5 6 5 5 4
25928 252/300 Flush
25929 253/300 Flush
25930 254/300 Read block 4372 len 8192
25931 255/300 Write block 3061 len 4096 data: 6
25932 256/300 Read block 858 len 12288
25933 257/300 Read block 1206 len 40960
25934 258/300 Read block 1592 len 20480
25935 259/300 Read block 3317 len 36864
25936 260/300 Flush
25937 261/300 Read block 2719 len 32768
25938 262/300 Write block 76 len 4096 data: 10
25939 263/300 Read block 4787 len 36864
25940 264/300 Write block 2029 len 20480 data: 3 3 4 4 4
25941 265/300 Write block 1481 len 12288 data: 8 7 8
25942 266/300 Write block 1134 len 4096 data: 5
25943 267/300 Write block 559 len 20480 data: 10 10 10 10 8
25944 268/300 Read block 3742 len 16384
25945 269/300 Flush
25946 270/300 Write block 76 len 12288 data: 11 9 8
25947 271/300 Write block 4857 len 4096 data: 7
25948 272/300 Write block 3187 len 20480 data: 3 3 3 3 3
25949 273/300 Write block 3638 len 36864 data: 6 5 5 8 10 9 9 8 6
25950 274/300 Flush
25951 275/300 Read block 529 len 16384
25952 276/300 Write block 4627 len 40960 data: 5 6 6 6 6 5 7 7 6 6
25953 277/300 Write block 1395 len 40960 data: 6 7 7 7 8 8 8 8 7 6
25954 278/300 Read block 1229 len 32768
25955 279/300 Write block 4424 len 16384 data: 9 9 8 8
25956 280/300 Write block 4153 len 40960 data: 4 3 4 5 6 5 7 8 7 7
25957 281/300 Flush
25958 282/300 Read block 2746 len 4096
25959 283/300 Read block 618 len 36864
25960 284/300 Write block 3325 len 32768 data: 4 4 5 4 3 4 6 5
25961 285/300 Flush
25962 286/300 Read block 4899 len 28672
25963 287/300 Read block 1104 len 12288
25964 288/300 Write block 3839 len 20480 data: 3 3 3 3 4
25965 289/300 Read block 1827 len 28672
25966 290/300 Write block 4099 len 12288 data: 8 5 5
25967 291/300 Write block 1006 len 40960 data: 4 3 3 3 4 5 5 4 5 5
25968 292/300 Write block 4978 len 16384 data: 4 3 4 4
25969 293/300 Write block 990 len 16384 data: 6 6 7 7
25970 294/300 Write block 3137 len 12288 data: 5 4 5
25971 295/300 Write block 4808 len 4096 data: 7
25972 296/300 Write block 1711 len 40960 data: 4 4 4 5 5 5 6 5 5 4
25973 297/300 Write block 3349 len 36864 data: 3 4 4 7 8 8 8 8 6
25974 298/300 Read block 354 len 8192
25975 299/300 Read block 3766 len 12288
25976 300/300 Write block 2746 len 36864 data: 3 3 3 3 3 2 2 2 3
25977 301/300 Read block 285 len 32768
25978 Main task has work to do, go find it
25979 got dsc Start(0)
25980 start 0
25981 Mar 17 22:22:05.062 INFO request completed, latency_us: 210, response_code: 204, uri: /start/cid/0, method: POST, req_id: a704a143-51aa-4baa-b62d-7f31926501b6, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
25982 Replay: started 0, returned:Ok(())
25983 ----------------------------------------------------------------
25984 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25985 States: Offline Active Active
25986 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
25987 7642 Acked Write 9 Sent Done Done false
25988 7643 Acked Write 1 Sent Done Done false
25989 7644 Acked Write 9 Sent Done Done false
25990 7645 Acked Read 9 Sent Done Done false
25991 7646 Acked Write 2 Sent Done Done false
25992 7647 Acked Write 2 Sent Done Done false
25993 7648 Acked Write 4 Sent Done Done false
25994 7649 Acked Write 1 Sent Done Done false
25995 7650 Acked Read 9 Sent Done Done false
25996 7651 Acked Write 5 Sent Done Done false
25997 7652 Acked Write 3 Sent Done Done false
25998 7653 Acked Write 10 Sent Done Done false
25999 7654 Acked Flush 0 Sent Done Done false
26000 7655 Acked Write 7 Sent Done Done false
26001 7656 Acked Write 2 Sent Done Done false
26002 7657 Acked Read 10 Sent Done Done false
26003 7658 Acked Write 5 Sent Done Done false
26004 7659 Acked Read 5 Sent Done Done false
26005 7660 Acked Read 10 Sent Done Done false
26006 7661 Acked Write 4 Sent Done Done false
26007 7662 Acked Write 6 Sent Done Done false
26008 7663 Acked Read 4 Sent Done Done false
26009 7664 Acked Write 7 Sent Done Done false
26010 7665 Acked Read 3 Sent Done Done false
26011 7666 Acked Read 2 Sent Done Done false
26012 7667 Acked Flush 0 Sent Done Done false
26013 7668 Acked Flush 0 Sent Done Done false
26014 7669 Acked Write 6 Sent Done Done false
26015 7670 Acked Read 10 Sent Done Done false
26016 7671 Acked Read 10 Sent Done Done false
26017 7672 Acked Write 8 Sent Done Done false
26018 7673 Acked Write 1 Sent Done Done false
26019 7674 Acked Read 2 Sent Done Done false
26020 7675 Acked Read 7 Sent Done Done false
26021 7676 Acked Read 7 Sent Done Done false
26022 7677 Acked Flush 0 Sent Done Done false
26023 7678 Acked Read 4 Sent Done Done false
26024 7679 Acked Write 4 Sent Done Done false
26025 7680 Acked Flush 0 Sent Done Done false
26026 7681 Acked Write 9 Sent Done Done false
26027 7682 Acked Read 9 Sent Done Done false
26028 7683 Acked Flush 0 Sent Done Done false
26029 7684 Acked Flush 0 Sent Done Done false
26030 7685 Acked Read 6 Sent Done Done false
26031 7686 Acked Flush 0 Sent Done Done false
26032 7687 Acked Write 2 Sent Done Done false
26033 7688 Acked Write 10 Sent Done Done false
26034 7689 Acked Write 3 Sent Done Done false
26035 7690 Acked Read 2 Sent Done Done false
26036 7691 Acked Read 7 Sent Done Done false
26037 7692 Acked Read 1 Sent Done Done false
26038 7693 Acked Read 1 Sent Done Done false
26039 7694 Acked Read 7 Sent Done Done false
26040 7695 Acked Write 3 Sent Done Done false
26041 7696 Acked Read 3 Sent Done Done false
26042 7697 Acked Read 8 Sent Done Done false
26043 7698 Acked Write 7 Sent Done Done false
26044 7699 Acked Read 4 Sent Done Done false
26045 7700 Acked Read 7 Sent Done Done false
26046 7701 Acked Read 6 Sent Done Done false
26047 7702 Acked Read 8 Sent Done Done false
26048 7703 Acked Read 5 Sent Done Done false
26049 7704 Acked Read 6 Sent Done Done false
26050 7705 Acked Write 6 Sent Done Done false
26051 7706 Acked Read 2 Sent Done Done false
26052 7707 Acked Read 8 Sent Done Done false
26053 7708 Acked Read 9 Sent Done Done false
26054 7709 Acked Read 2 Sent Done Done false
26055 7710 Acked Read 8 Sent Done Done false
26056 7711 Acked Read 5 Sent Done Done false
26057 7712 Acked Write 9 Sent Done Done false
26058 7713 Acked Write 4 Sent Done Done false
26059 7714 Acked Read 9 Sent Done Done false
26060 7715 Acked Read 4 Sent Done Done false
26061 7716 Acked Flush 0 Sent Done Done false
26062 7717 Acked Read 7 Sent Done Done false
26063 7718 Acked Read 5 Sent Done Done false
26064 7719 Acked Read 9 Sent Done Done false
26065 7720 Acked Write 9 Sent Done Done false
26066 7721 Acked Write 2 Sent Done Done false
26067 7722 Acked Read 9 Sent Done Done false
26068 7723 Acked Read 7 Sent Done Done false
26069 7724 Acked Read 6 Sent Done Done false
26070 7725 Acked Read 8 Sent Done Done false
26071 7726 Acked Write 6 Sent Done Done false
26072 7727 Acked Write 3 Sent Done Done false
26073 7728 Acked Flush 0 Sent Done Done false
26074 7729 Acked Flush 0 Sent Done Done false
26075 7730 Acked Write 4 Sent Done Done false
26076 7731 Acked Read 3 Sent Done Done false
26077 7732 Acked Write 5 Sent Done Done false
26078 7733 Acked Read 4 Sent Done Done false
26079 7734 Acked Read 4 Sent Done Done false
26080 7735 Acked Write 1 Sent Done Done false
26081 7736 Acked Read 9 Sent Done Done false
26082 7737 Acked Read 10 Sent Done Done false
26083 7738 Acked Read 8 Sent Done Done false
26084 7739 Acked Read 7 Sent Done Done false
26085 7740 Acked Write 4 Sent Done Done false
26086 7741 Acked Read 2 Sent Done Done false
26087 7742 Acked Write 1 Sent Done Done false
26088 7743 Acked Flush 0 Sent Done Done false
26089 7744 Acked Write 10 Sent Done Done false
26090 7745 Acked Read 3 Sent Done Done false
26091 7746 Acked Read 5 Sent Done Done false
26092 7747 Acked Read 2 Sent Done Done false
26093 7748 Acked Read 4 Sent Done Done false
26094 7749 Acked Write 10 Sent Done Done false
26095 7750 Acked Write 4 Sent Done Done false
26096 7751 Acked Read 2 Sent Done Done false
26097 7752 Acked Read 9 Sent Done Done false
26098 7753 Acked Read 10 Sent Done Done false
26099 7754 Acked Read 7 Sent Done Done false
26100 7755 Acked Write 7 Sent Done Done false
26101 7756 Acked Write 7 Sent Done Done false
26102 7757 Acked Read 4 Sent Done Done false
26103 7758 Acked Write 6 Sent Done Done false
26104 7759 Acked Write 8 Sent Done Done false
26105 7760 Acked Write 9 Sent Done Done false
26106 7761 Acked Read 6 Sent Done Done false
26107 7762 Acked Read 2 Sent Done Done false
26108 7763 Acked Read 3 Sent Done Done false
26109 7764 Acked Flush 0 Sent Done Done false
26110 7765 Acked Read 5 Sent Done Done false
26111 7766 Acked Write 2 Sent Done Done false
26112 7767 Acked Write 10 Sent Done Done false
26113 7768 Acked Write 1 Sent Done Done false
26114 7769 Acked Flush 0 Sent Done Done false
26115 7770 Acked Write 2 Sent Done Done false
26116 7771 Acked Read 5 Sent Done Done false
26117 7772 Acked Read 10 Sent Done Done false
26118 7773 Acked Read 4 Sent Done Done false
26119 7774 Acked Write 9 Sent Done Done false
26120 7775 Acked Read 7 Sent Done Done false
26121 7776 Acked Write 6 Sent Done Done false
26122 7777 Acked Flush 0 Sent Done Done false
26123 7778 Acked Read 7 Sent Done Done false
26124 7779 Acked Read 2 Sent Done Done false
26125 7780 Acked Write 6 Sent Done Done false
26126 7781 Acked Write 1 Sent Done Done false
26127 7782 Acked Write 10 Sent Done Done false
26128 7783 Acked Read 1 Sent Done Done false
26129 7784 Acked Read 7 Sent Done Done false
26130 7785 Acked Write 2 Sent Done Done false
26131 7786 Acked Read 2 Sent Done Done false
26132 7787 Acked Read 2 Sent Done Done false
26133 7788 Acked Read 2 Sent Done Done false
26134 7789 Acked Read 3 Sent Done Done false
26135 7790 Acked Read 6 Sent Done Done false
26136 7791 Acked Read 6 Sent Done Done false
26137 7792 Acked Write 5 Sent Done Done false
26138 7793 Acked Read 5 Sent Done Done false
26139 7794 Acked Read 1 Sent Done Done false
26140 7795 Acked Read 4 Sent Done Done false
26141 7796 Acked Read 3 Sent Done Done false
26142 7797 Acked Flush 0 Sent Done Done false
26143 7798 Acked Read 6 Sent Done Done false
26144 7799 Acked Read 4 Sent Done Done false
26145 7800 Acked Read 10 Sent Done Done false
26146 7801 Acked Flush 0 Sent Done Done false
26147 7802 Acked Read 6 Sent Done Done false
26148 7803 Acked Write 1 Sent Done Done false
26149 7804 Acked Write 8 Sent Done Done false
26150 7805 Acked Write 5 Sent Done Done false
26151 7806 Acked Write 1 Sent Done Done false
26152 7807 Acked Write 9 Sent Done Done false
26153 7808 Acked Write 1 Sent Done Done false
26154 7809 Acked Read 6 Sent Done Done false
26155 7810 Acked Write 1 Sent Done Done false
26156 7811 Acked Read 9 Sent Done Done false
26157 7812 Acked Flush 0 Sent Done Done false
26158 7813 Acked Read 3 Sent Done Done false
26159 7814 Acked Read 5 Sent Done Done false
26160 7815 Acked Read 4 Sent Done Done false
26161 7816 Acked Read 5 Sent Done Done false
26162 7817 Acked Flush 0 Sent Done Done false
26163 7818 Acked Read 1 Sent Done Done false
26164 7819 Acked Write 7 Sent Done Done false
26165 7820 Acked Read 4 Sent Done Done false
26166 7821 Acked Read 7 Sent Done Done false
26167 7822 Acked Read 1 Sent Done Done false
26168 7823 Acked Read 2 Sent Done Done false
26169 7824 Acked Read 9 Sent Done Done false
26170 7825 Acked Read 3 Sent Done Done false
26171 7826 Acked Write 5 Sent Done Done false
26172 7827 Acked Read 10 Sent Done Done false
26173 7828 Acked Read 10 Sent Done Done false
26174 7829 Acked Read 7 Sent Done Done false
26175 7830 Acked Read 7 Sent Done Done false
26176 7831 Acked Read 9 Sent Done Done false
26177 7832 Acked Read 4 Sent Done Done false
26178 7833 Acked Read 6 Sent Done Done false
26179 7834 Acked Read 5 Sent Done Done false
26180 7835 Acked Write 4 Sent Done Done false
26181 7836 Acked Write 10 Sent Done Done false
26182 7837 Acked Write 3 Sent Done Done false
26183 7838 Acked Read 2 Sent Done Done false
26184 7839 Acked Write 2 Sent Done Done false
26185 7840 Acked Write 8 Sent Done Done false
26186 7841 Acked Write 2 Sent Done Done false
26187 7842 Acked Write 8 Sent Done Done false
26188 7843 Acked Read 8 Sent Done Done false
26189 7844 Acked Read 6 Sent Done Done false
26190 7845 Acked Read 7 Sent Done Done false
26191 7846 Acked Write 7 Sent Done Done false
26192 7847 Acked Write 5 Sent Done Done false
26193 7848 Acked Read 8 Sent Done Done false
26194 7849 Acked Read 2 Sent Done Done false
26195 7850 Acked Read 1 Sent Done Done false
26196 7851 Acked Flush 0 Sent Done Done false
26197 7852 Acked Write 9 Sent Done Done false
26198 7853 Acked Write 8 Sent Done Done false
26199 7854 Acked Write 9 Sent Done Done false
26200 7855 Acked Read 7 Sent Done Done false
26201 7856 Acked Write 4 Sent Done Done false
26202 7857 Acked Write 5 Sent Done Done false
26203 7858 Acked Flush 0 Sent Done Done false
26204 7859 Acked Write 7 Sent Done Done false
26205 7860 Acked Read 2 Sent Done Done false
26206 7861 Acked Read 8 Sent Done Done false
26207 7862 Acked Read 8 Sent Done Done false
26208 7863 Acked Read 9 Sent Done Done false
26209 7864 Acked Write 10 Sent Done Done false
26210 7865 Acked Read 1 Sent Done Done false
26211 7866 Acked Read 5 Sent Done Done false
26212 7867 Acked Flush 0 Sent Done Done false
26213 7868 Acked Read 7 Sent Done Done false
26214 7869 Acked Read 7 Sent Done Done false
26215 7870 Acked Write 8 Sent Done Done false
26216 7871 Acked Write 9 Sent Done Done false
26217 7872 Acked Write 3 Sent Done Done false
26218 7873 Acked Read 1 Sent Done Done false
26219 7874 Acked Read 6 Sent Done Done false
26220 7875 Acked Write 2 Sent Done Done false
26221 7876 Acked Write 2 Sent Done Done false
26222 7877 Acked Read 5 Sent Done Done false
26223 7878 Acked Write 4 Sent Done Done false
26224 7879 Acked Write 6 Sent Done Done false
26225 7880 Acked Write 4 Sent Done Done false
26226 7881 Acked Read 1 Sent Done Done false
26227 7882 Acked Write 5 Sent Done Done false
26228 7883 Acked Flush 0 Sent Done Done false
26229 7884 Acked Read 8 Sent Done Done false
26230 7885 Acked Read 9 Sent Done Done false
26231 7886 Acked Flush 0 Sent Done Done false
26232 7887 Acked Read 4 Sent Done Done false
26233 7888 Acked Flush 0 Sent Done Done false
26234 7889 Acked Write 4 Sent Done Done false
26235 7890 Acked Read 1 Sent Done Done false
26236 7891 Acked Read 3 Sent Done Done false
26237 7892 Acked Write 6 Sent Done Done false
26238 7893 Acked Flush 0 Sent Done Done false
26239 7894 Acked Flush 0 Sent Done Done false
26240 7895 Acked Read 2 Sent Done Done false
26241 7896 Acked Write 1 Sent Done Done false
26242 7897 Acked Read 3 Sent Done Done false
26243 7898 Acked Read 10 Sent Done Done false
26244 7899 Acked Read 5 Sent Done Done false
26245 7900 Acked Read 9 Sent Done Done false
26246 7901 Acked Flush 0 Sent Done Done false
26247 7902 Acked Read 8 Sent Done Done false
26248 7903 Acked Write 1 Sent Done Done false
26249 7904 Acked Read 9 Sent Done Done false
26250 7905 Acked Write 5 Sent Done Done false
26251 7906 Acked Write 3 Sent Done Done false
26252 7907 Acked Write 1 Sent Done Done false
26253 7908 Acked Write 5 Sent Done Done false
26254 7909 Acked Read 4 Sent Done Done false
26255 7910 Acked Flush 0 Sent Done Done false
26256 7911 Acked Write 3 Sent Done Done false
26257 7912 Acked Write 1 Sent Done Done false
26258 7913 Acked Write 5 Sent Done Done false
26259 7914 Acked Write 9 Sent Done Done false
26260 7915 Acked Flush 0 Sent Done Done false
26261 7916 Acked Read 4 Sent Done Done false
26262 7917 Acked Write 10 Sent Done Done false
26263 7918 Acked Write 10 Sent Done Done false
26264 7919 Acked Read 8 Sent Done Done false
26265 7920 Acked Write 4 Sent Done Done false
26266 7921 Acked Write 10 Sent Done Done false
26267 7922 Acked Flush 0 Sent Done Done false
26268 7923 Acked Read 1 Sent Done Done false
26269 7924 Acked Read 9 Sent Done Done false
26270 7925 Acked Write 8 Sent Done Done false
26271 7926 Acked Flush 0 Sent Done Done false
26272 7927 Acked Read 7 Sent Done Done false
26273 7928 Acked Read 3 Sent Done Done false
26274 7929 Acked Write 5 Sent Done Done false
26275 7930 Acked Read 7 Sent Done Done false
26276 7931 Acked Write 3 Sent Done Done false
26277 7932 Acked Write 10 Sent Done Done false
26278 7933 Acked Write 4 Sent Done Done false
26279 7934 Acked Write 4 Sent Done Done false
26280 7935 Acked Write 3 Sent Done Done false
26281 7936 Acked Write 1 Sent Done Done false
26282 7937 Acked Write 10 Sent Done Done false
26283 7938 Acked Write 9 Sent Done Done false
26284 7939 Acked Read 2 Sent Done Done false
26285 7940 Acked Read 3 Sent Done Done false
26286 7941 Acked Write 9 Sent Done Done false
26287 7942 Acked Read 8 Sent Done Done false
26288 STATES DS:0 DS:1 DS:2 TOTAL
26289 Sent 301 0 0 301
26290 Done 0 301 301 602
26291 Skipped 0 0 0 0
26292 Error 0 0 0 0
26293 Last Flush: Some(JobId(7641)) Some(JobId(7926)) Some(JobId(7926))
26294 Downstairs last five completed: 7641 7640 7639 7638 7637
26295 Upstairs last five completed: 7942 7941 7940 7939 7938
26296 CLIENT: Up:0 ds:301 act:2
262972025-03-17T22:22:05.565ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
262982025-03-17T22:22:06.150ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
262992025-03-17T22:22:07.558ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
263002025-03-17T22:22:08.153ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
26301 ----------------------------------------------------------------
26302 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26303 States: Offline Active Active
26304 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26305 7642 Acked Write 9 Sent Done Done false
26306 7643 Acked Write 1 Sent Done Done false
26307 7644 Acked Write 9 Sent Done Done false
26308 7645 Acked Read 9 Sent Done Done false
26309 7646 Acked Write 2 Sent Done Done false
26310 7647 Acked Write 2 Sent Done Done false
26311 7648 Acked Write 4 Sent Done Done false
26312 7649 Acked Write 1 Sent Done Done false
26313 7650 Acked Read 9 Sent Done Done false
26314 7651 Acked Write 5 Sent Done Done false
26315 7652 Acked Write 3 Sent Done Done false
26316 7653 Acked Write 10 Sent Done Done false
26317 7654 Acked Flush 0 Sent Done Done false
26318 7655 Acked Write 7 Sent Done Done false
26319 7656 Acked Write 2 Sent Done Done false
26320 7657 Acked Read 10 Sent Done Done false
26321 7658 Acked Write 5 Sent Done Done false
26322 7659 Acked Read 5 Sent Done Done false
26323 7660 Acked Read 10 Sent Done Done false
26324 7661 Acked Write 4 Sent Done Done false
26325 7662 Acked Write 6 Sent Done Done false
26326 7663 Acked Read 4 Sent Done Done false
26327 7664 Acked Write 7 Sent Done Done false
26328 7665 Acked Read 3 Sent Done Done false
26329 7666 Acked Read 2 Sent Done Done false
26330 7667 Acked Flush 0 Sent Done Done false
26331 7668 Acked Flush 0 Sent Done Done false
26332 7669 Acked Write 6 Sent Done Done false
26333 7670 Acked Read 10 Sent Done Done false
26334 7671 Acked Read 10 Sent Done Done false
26335 7672 Acked Write 8 Sent Done Done false
26336 7673 Acked Write 1 Sent Done Done false
26337 7674 Acked Read 2 Sent Done Done false
26338 7675 Acked Read 7 Sent Done Done false
26339 7676 Acked Read 7 Sent Done Done false
26340 7677 Acked Flush 0 Sent Done Done false
26341 7678 Acked Read 4 Sent Done Done false
26342 7679 Acked Write 4 Sent Done Done false
26343 7680 Acked Flush 0 Sent Done Done false
26344 7681 Acked Write 9 Sent Done Done false
26345 7682 Acked Read 9 Sent Done Done false
26346 7683 Acked Flush 0 Sent Done Done false
26347 7684 Acked Flush 0 Sent Done Done false
26348 7685 Acked Read 6 Sent Done Done false
26349 7686 Acked Flush 0 Sent Done Done false
26350 7687 Acked Write 2 Sent Done Done false
26351 7688 Acked Write 10 Sent Done Done false
26352 7689 Acked Write 3 Sent Done Done false
26353 7690 Acked Read 2 Sent Done Done false
26354 7691 Acked Read 7 Sent Done Done false
26355 7692 Acked Read 1 Sent Done Done false
26356 7693 Acked Read 1 Sent Done Done false
26357 7694 Acked Read 7 Sent Done Done false
26358 7695 Acked Write 3 Sent Done Done false
26359 7696 Acked Read 3 Sent Done Done false
26360 7697 Acked Read 8 Sent Done Done false
26361 7698 Acked Write 7 Sent Done Done false
26362 7699 Acked Read 4 Sent Done Done false
26363 7700 Acked Read 7 Sent Done Done false
26364 7701 Acked Read 6 Sent Done Done false
26365 7702 Acked Read 8 Sent Done Done false
26366 7703 Acked Read 5 Sent Done Done false
26367 7704 Acked Read 6 Sent Done Done false
26368 7705 Acked Write 6 Sent Done Done false
26369 7706 Acked Read 2 Sent Done Done false
26370 7707 Acked Read 8 Sent Done Done false
26371 7708 Acked Read 9 Sent Done Done false
26372 7709 Acked Read 2 Sent Done Done false
26373 7710 Acked Read 8 Sent Done Done false
26374 7711 Acked Read 5 Sent Done Done false
26375 7712 Acked Write 9 Sent Done Done false
26376 7713 Acked Write 4 Sent Done Done false
26377 7714 Acked Read 9 Sent Done Done false
26378 7715 Acked Read 4 Sent Done Done false
26379 7716 Acked Flush 0 Sent Done Done false
26380 7717 Acked Read 7 Sent Done Done false
26381 7718 Acked Read 5 Sent Done Done false
26382 7719 Acked Read 9 Sent Done Done false
26383 7720 Acked Write 9 Sent Done Done false
26384 7721 Acked Write 2 Sent Done Done false
26385 7722 Acked Read 9 Sent Done Done false
26386 7723 Acked Read 7 Sent Done Done false
26387 7724 Acked Read 6 Sent Done Done false
26388 7725 Acked Read 8 Sent Done Done false
26389 7726 Acked Write 6 Sent Done Done false
26390 7727 Acked Write 3 Sent Done Done false
26391 7728 Acked Flush 0 Sent Done Done false
26392 7729 Acked Flush 0 Sent Done Done false
26393 7730 Acked Write 4 Sent Done Done false
26394 7731 Acked Read 3 Sent Done Done false
26395 7732 Acked Write 5 Sent Done Done false
26396 7733 Acked Read 4 Sent Done Done false
26397 7734 Acked Read 4 Sent Done Done false
26398 7735 Acked Write 1 Sent Done Done false
26399 7736 Acked Read 9 Sent Done Done false
26400 7737 Acked Read 10 Sent Done Done false
26401 7738 Acked Read 8 Sent Done Done false
26402 7739 Acked Read 7 Sent Done Done false
26403 7740 Acked Write 4 Sent Done Done false
26404 7741 Acked Read 2 Sent Done Done false
26405 7742 Acked Write 1 Sent Done Done false
26406 7743 Acked Flush 0 Sent Done Done false
26407 7744 Acked Write 10 Sent Done Done false
26408 7745 Acked Read 3 Sent Done Done false
26409 7746 Acked Read 5 Sent Done Done false
26410 7747 Acked Read 2 Sent Done Done false
26411 7748 Acked Read 4 Sent Done Done false
26412 7749 Acked Write 10 Sent Done Done false
26413 7750 Acked Write 4 Sent Done Done false
26414 7751 Acked Read 2 Sent Done Done false
26415 7752 Acked Read 9 Sent Done Done false
26416 7753 Acked Read 10 Sent Done Done false
26417 7754 Acked Read 7 Sent Done Done false
26418 7755 Acked Write 7 Sent Done Done false
26419 7756 Acked Write 7 Sent Done Done false
26420 7757 Acked Read 4 Sent Done Done false
26421 7758 Acked Write 6 Sent Done Done false
26422 7759 Acked Write 8 Sent Done Done false
26423 7760 Acked Write 9 Sent Done Done false
26424 7761 Acked Read 6 Sent Done Done false
26425 7762 Acked Read 2 Sent Done Done false
26426 7763 Acked Read 3 Sent Done Done false
26427 7764 Acked Flush 0 Sent Done Done false
26428 7765 Acked Read 5 Sent Done Done false
26429 7766 Acked Write 2 Sent Done Done false
26430 7767 Acked Write 10 Sent Done Done false
26431 7768 Acked Write 1 Sent Done Done false
26432 7769 Acked Flush 0 Sent Done Done false
26433 7770 Acked Write 2 Sent Done Done false
26434 7771 Acked Read 5 Sent Done Done false
26435 7772 Acked Read 10 Sent Done Done false
26436 7773 Acked Read 4 Sent Done Done false
26437 7774 Acked Write 9 Sent Done Done false
26438 7775 Acked Read 7 Sent Done Done false
26439 7776 Acked Write 6 Sent Done Done false
26440 7777 Acked Flush 0 Sent Done Done false
26441 7778 Acked Read 7 Sent Done Done false
26442 7779 Acked Read 2 Sent Done Done false
26443 7780 Acked Write 6 Sent Done Done false
26444 7781 Acked Write 1 Sent Done Done false
26445 7782 Acked Write 10 Sent Done Done false
26446 7783 Acked Read 1 Sent Done Done false
26447 7784 Acked Read 7 Sent Done Done false
26448 7785 Acked Write 2 Sent Done Done false
26449 7786 Acked Read 2 Sent Done Done false
26450 7787 Acked Read 2 Sent Done Done false
26451 7788 Acked Read 2 Sent Done Done false
26452 7789 Acked Read 3 Sent Done Done false
26453 7790 Acked Read 6 Sent Done Done false
26454 7791 Acked Read 6 Sent Done Done false
26455 7792 Acked Write 5 Sent Done Done false
26456 7793 Acked Read 5 Sent Done Done false
26457 7794 Acked Read 1 Sent Done Done false
26458 7795 Acked Read 4 Sent Done Done false
26459 7796 Acked Read 3 Sent Done Done false
26460 7797 Acked Flush 0 Sent Done Done false
26461 7798 Acked Read 6 Sent Done Done false
26462 7799 Acked Read 4 Sent Done Done false
26463 7800 Acked Read 10 Sent Done Done false
26464 7801 Acked Flush 0 Sent Done Done false
26465 7802 Acked Read 6 Sent Done Done false
26466 7803 Acked Write 1 Sent Done Done false
26467 7804 Acked Write 8 Sent Done Done false
26468 7805 Acked Write 5 Sent Done Done false
26469 7806 Acked Write 1 Sent Done Done false
26470 7807 Acked Write 9 Sent Done Done false
26471 7808 Acked Write 1 Sent Done Done false
26472 7809 Acked Read 6 Sent Done Done false
26473 7810 Acked Write 1 Sent Done Done false
26474 7811 Acked Read 9 Sent Done Done false
26475 7812 Acked Flush 0 Sent Done Done false
26476 7813 Acked Read 3 Sent Done Done false
26477 7814 Acked Read 5 Sent Done Done false
26478 7815 Acked Read 4 Sent Done Done false
26479 7816 Acked Read 5 Sent Done Done false
26480 7817 Acked Flush 0 Sent Done Done false
26481 7818 Acked Read 1 Sent Done Done false
26482 7819 Acked Write 7 Sent Done Done false
26483 7820 Acked Read 4 Sent Done Done false
26484 7821 Acked Read 7 Sent Done Done false
26485 7822 Acked Read 1 Sent Done Done false
26486 7823 Acked Read 2 Sent Done Done false
26487 7824 Acked Read 9 Sent Done Done false
26488 7825 Acked Read 3 Sent Done Done false
26489 7826 Acked Write 5 Sent Done Done false
26490 7827 Acked Read 10 Sent Done Done false
26491 7828 Acked Read 10 Sent Done Done false
26492 7829 Acked Read 7 Sent Done Done false
26493 7830 Acked Read 7 Sent Done Done false
26494 7831 Acked Read 9 Sent Done Done false
26495 7832 Acked Read 4 Sent Done Done false
26496 7833 Acked Read 6 Sent Done Done false
26497 7834 Acked Read 5 Sent Done Done false
26498 7835 Acked Write 4 Sent Done Done false
26499 7836 Acked Write 10 Sent Done Done false
26500 7837 Acked Write 3 Sent Done Done false
26501 7838 Acked Read 2 Sent Done Done false
26502 7839 Acked Write 2 Sent Done Done false
26503 7840 Acked Write 8 Sent Done Done false
26504 7841 Acked Write 2 Sent Done Done false
26505 7842 Acked Write 8 Sent Done Done false
26506 7843 Acked Read 8 Sent Done Done false
26507 7844 Acked Read 6 Sent Done Done false
26508 7845 Acked Read 7 Sent Done Done false
26509 7846 Acked Write 7 Sent Done Done false
26510 7847 Acked Write 5 Sent Done Done false
26511 7848 Acked Read 8 Sent Done Done false
26512 7849 Acked Read 2 Sent Done Done false
26513 7850 Acked Read 1 Sent Done Done false
26514 7851 Acked Flush 0 Sent Done Done false
26515 7852 Acked Write 9 Sent Done Done false
26516 7853 Acked Write 8 Sent Done Done false
26517 7854 Acked Write 9 Sent Done Done false
26518 7855 Acked Read 7 Sent Done Done false
26519 7856 Acked Write 4 Sent Done Done false
26520 7857 Acked Write 5 Sent Done Done false
26521 7858 Acked Flush 0 Sent Done Done false
26522 7859 Acked Write 7 Sent Done Done false
26523 7860 Acked Read 2 Sent Done Done false
26524 7861 Acked Read 8 Sent Done Done false
26525 7862 Acked Read 8 Sent Done Done false
26526 7863 Acked Read 9 Sent Done Done false
26527 7864 Acked Write 10 Sent Done Done false
26528 7865 Acked Read 1 Sent Done Done false
26529 7866 Acked Read 5 Sent Done Done false
26530 7867 Acked Flush 0 Sent Done Done false
26531 7868 Acked Read 7 Sent Done Done false
26532 7869 Acked Read 7 Sent Done Done false
26533 7870 Acked Write 8 Sent Done Done false
26534 7871 Acked Write 9 Sent Done Done false
26535 7872 Acked Write 3 Sent Done Done false
26536 7873 Acked Read 1 Sent Done Done false
26537 7874 Acked Read 6 Sent Done Done false
26538 7875 Acked Write 2 Sent Done Done false
26539 7876 Acked Write 2 Sent Done Done false
26540 7877 Acked Read 5 Sent Done Done false
26541 7878 Acked Write 4 Sent Done Done false
26542 7879 Acked Write 6 Sent Done Done false
26543 7880 Acked Write 4 Sent Done Done false
26544 7881 Acked Read 1 Sent Done Done false
26545 7882 Acked Write 5 Sent Done Done false
26546 7883 Acked Flush 0 Sent Done Done false
26547 7884 Acked Read 8 Sent Done Done false
26548 7885 Acked Read 9 Sent Done Done false
26549 7886 Acked Flush 0 Sent Done Done false
26550 7887 Acked Read 4 Sent Done Done false
26551 7888 Acked Flush 0 Sent Done Done false
26552 7889 Acked Write 4 Sent Done Done false
26553 7890 Acked Read 1 Sent Done Done false
26554 7891 Acked Read 3 Sent Done Done false
26555 7892 Acked Write 6 Sent Done Done false
26556 7893 Acked Flush 0 Sent Done Done false
26557 7894 Acked Flush 0 Sent Done Done false
26558 7895 Acked Read 2 Sent Done Done false
26559 7896 Acked Write 1 Sent Done Done false
26560 7897 Acked Read 3 Sent Done Done false
26561 7898 Acked Read 10 Sent Done Done false
26562 7899 Acked Read 5 Sent Done Done false
26563 7900 Acked Read 9 Sent Done Done false
26564 7901 Acked Flush 0 Sent Done Done false
26565 7902 Acked Read 8 Sent Done Done false
26566 7903 Acked Write 1 Sent Done Done false
26567 7904 Acked Read 9 Sent Done Done false
26568 7905 Acked Write 5 Sent Done Done false
26569 7906 Acked Write 3 Sent Done Done false
26570 7907 Acked Write 1 Sent Done Done false
26571 7908 Acked Write 5 Sent Done Done false
26572 7909 Acked Read 4 Sent Done Done false
26573 7910 Acked Flush 0 Sent Done Done false
26574 7911 Acked Write 3 Sent Done Done false
26575 7912 Acked Write 1 Sent Done Done false
26576 7913 Acked Write 5 Sent Done Done false
26577 7914 Acked Write 9 Sent Done Done false
26578 7915 Acked Flush 0 Sent Done Done false
26579 7916 Acked Read 4 Sent Done Done false
26580 7917 Acked Write 10 Sent Done Done false
26581 7918 Acked Write 10 Sent Done Done false
26582 7919 Acked Read 8 Sent Done Done false
26583 7920 Acked Write 4 Sent Done Done false
26584 7921 Acked Write 10 Sent Done Done false
26585 7922 Acked Flush 0 Sent Done Done false
26586 7923 Acked Read 1 Sent Done Done false
26587 7924 Acked Read 9 Sent Done Done false
26588 7925 Acked Write 8 Sent Done Done false
26589 7926 Acked Flush 0 Sent Done Done false
26590 7927 Acked Read 7 Sent Done Done false
26591 7928 Acked Read 3 Sent Done Done false
26592 7929 Acked Write 5 Sent Done Done false
26593 7930 Acked Read 7 Sent Done Done false
26594 7931 Acked Write 3 Sent Done Done false
26595 7932 Acked Write 10 Sent Done Done false
26596 7933 Acked Write 4 Sent Done Done false
26597 7934 Acked Write 4 Sent Done Done false
26598 7935 Acked Write 3 Sent Done Done false
26599 7936 Acked Write 1 Sent Done Done false
26600 7937 Acked Write 10 Sent Done Done false
26601 7938 Acked Write 9 Sent Done Done false
26602 7939 Acked Read 2 Sent Done Done false
26603 7940 Acked Read 3 Sent Done Done false
26604 7941 Acked Write 9 Sent Done Done false
26605 7942 Acked Read 8 Sent Done Done false
26606 STATES DS:0 DS:1 DS:2 TOTAL
26607 Sent 301 0 0 301
26608 Done 0 301 301 602
26609 Skipped 0 0 0 0
26610 Error 0 0 0 0
26611 Last Flush: Some(JobId(7641)) Some(JobId(7926)) Some(JobId(7926))
26612 Downstairs last five completed: 7641 7640 7639 7638 7637
26613 Upstairs last five completed: 7942 7941 7940 7939 7938
26614 CLIENT: Up:0 ds:301 act:2
266152025-03-17T22:22:09.567ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
266162025-03-17T22:22:10.154ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
266172025-03-17T22:22:10.818ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
266182025-03-17T22:22:10.818ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
266192025-03-17T22:22:10.818ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
266202025-03-17T22:22:10.818ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
266212025-03-17T22:22:10.819ZINFOcrucible: client task is exiting = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
266222025-03-17T22:22:11.320ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
266232025-03-17T22:22:12.558ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
26624 [8810] got start action
26625 [8810] I am going to restart
26626 Starting downstairs at port 8810
26627 Make output file at "/tmp/dsc/downstairs-8810.txt"
26628 [8810][0] reports Starting
26629 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1229)
26630 [8810][0] reports Running
26631 ----------------------------------------------------------------
26632 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26633 States: Offline Active Active
26634 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26635 7642 Acked Write 9 Sent Done Done false
26636 7643 Acked Write 1 Sent Done Done false
26637 7644 Acked Write 9 Sent Done Done false
26638 7645 Acked Read 9 Sent Done Done false
26639 7646 Acked Write 2 Sent Done Done false
26640 7647 Acked Write 2 Sent Done Done false
26641 7648 Acked Write 4 Sent Done Done false
26642 7649 Acked Write 1 Sent Done Done false
26643 7650 Acked Read 9 Sent Done Done false
26644 7651 Acked Write 5 Sent Done Done false
26645 7652 Acked Write 3 Sent Done Done false
26646 7653 Acked Write 10 Sent Done Done false
26647 7654 Acked Flush 0 Sent Done Done false
26648 7655 Acked Write 7 Sent Done Done false
26649 7656 Acked Write 2 Sent Done Done false
26650 7657 Acked Read 10 Sent Done Done false
26651 7658 Acked Write 5 Sent Done Done false
26652 7659 Acked Read 5 Sent Done Done false
26653 7660 Acked Read 10 Sent Done Done false
26654 7661 Acked Write 4 Sent Done Done false
26655 7662 Acked Write 6 Sent Done Done false
26656 7663 Acked Read 4 Sent Done Done false
26657 7664 Acked Write 7 Sent Done Done false
26658 7665 Acked Read 3 Sent Done Done false
26659 7666 Acked Read 2 Sent Done Done false
26660 7667 Acked Flush 0 Sent Done Done false
26661 7668 Acked Flush 0 Sent Done Done false
26662 7669 Acked Write 6 Sent Done Done false
26663 7670 Acked Read 10 Sent Done Done false
26664 7671 Acked Read 10 Sent Done Done false
26665 7672 Acked Write 8 Sent Done Done false
26666 7673 Acked Write 1 Sent Done Done false
26667 7674 Acked Read 2 Sent Done Done false
26668 7675 Acked Read 7 Sent Done Done false
26669 7676 Acked Read 7 Sent Done Done false
26670 7677 Acked Flush 0 Sent Done Done false
26671 7678 Acked Read 4 Sent Done Done false
26672 7679 Acked Write 4 Sent Done Done false
26673 7680 Acked Flush 0 Sent Done Done false
26674 7681 Acked Write 9 Sent Done Done false
26675 7682 Acked Read 9 Sent Done Done false
26676 7683 Acked Flush 0 Sent Done Done false
26677 7684 Acked Flush 0 Sent Done Done false
26678 7685 Acked Read 6 Sent Done Done false
26679 7686 Acked Flush 0 Sent Done Done false
26680 7687 Acked Write 2 Sent Done Done false
26681 7688 Acked Write 10 Sent Done Done false
26682 7689 Acked Write 3 Sent Done Done false
26683 7690 Acked Read 2 Sent Done Done false
26684 7691 Acked Read 7 Sent Done Done false
26685 7692 Acked Read 1 Sent Done Done false
26686 7693 Acked Read 1 Sent Done Done false
26687 7694 Acked Read 7 Sent Done Done false
26688 7695 Acked Write 3 Sent Done Done false
26689 7696 Acked Read 3 Sent Done Done false
26690 7697 Acked Read 8 Sent Done Done false
26691 7698 Acked Write 7 Sent Done Done false
26692 7699 Acked Read 4 Sent Done Done false
26693 7700 Acked Read 7 Sent Done Done false
26694 7701 Acked Read 6 Sent Done Done false
26695 7702 Acked Read 8 Sent Done Done false
26696 7703 Acked Read 5 Sent Done Done false
26697 7704 Acked Read 6 Sent Done Done false
26698 7705 Acked Write 6 Sent Done Done false
26699 7706 Acked Read 2 Sent Done Done false
26700 7707 Acked Read 8 Sent Done Done false
26701 7708 Acked Read 9 Sent Done Done false
26702 7709 Acked Read 2 Sent Done Done false
26703 7710 Acked Read 8 Sent Done Done false
26704 7711 Acked Read 5 Sent Done Done false
26705 7712 Acked Write 9 Sent Done Done false
26706 7713 Acked Write 4 Sent Done Done false
26707 7714 Acked Read 9 Sent Done Done false
26708 7715 Acked Read 4 Sent Done Done false
26709 7716 Acked Flush 0 Sent Done Done false
26710 7717 Acked Read 7 Sent Done Done false
26711 7718 Acked Read 5 Sent Done Done false
26712 7719 Acked Read 9 Sent Done Done false
26713 7720 Acked Write 9 Sent Done Done false
26714 7721 Acked Write 2 Sent Done Done false
26715 7722 Acked Read 9 Sent Done Done false
26716 7723 Acked Read 7 Sent Done Done false
26717 7724 Acked Read 6 Sent Done Done false
26718 7725 Acked Read 8 Sent Done Done false
26719 7726 Acked Write 6 Sent Done Done false
26720 7727 Acked Write 3 Sent Done Done false
26721 7728 Acked Flush 0 Sent Done Done false
26722 7729 Acked Flush 0 Sent Done Done false
26723 7730 Acked Write 4 Sent Done Done false
26724 7731 Acked Read 3 Sent Done Done false
26725 7732 Acked Write 5 Sent Done Done false
26726 7733 Acked Read 4 Sent Done Done false
26727 7734 Acked Read 4 Sent Done Done false
26728 7735 Acked Write 1 Sent Done Done false
26729 7736 Acked Read 9 Sent Done Done false
26730 7737 Acked Read 10 Sent Done Done false
26731 7738 Acked Read 8 Sent Done Done false
26732 7739 Acked Read 7 Sent Done Done false
26733 7740 Acked Write 4 Sent Done Done false
26734 7741 Acked Read 2 Sent Done Done false
26735 7742 Acked Write 1 Sent Done Done false
26736 7743 Acked Flush 0 Sent Done Done false
26737 7744 Acked Write 10 Sent Done Done false
26738 7745 Acked Read 3 Sent Done Done false
26739 7746 Acked Read 5 Sent Done Done false
26740 7747 Acked Read 2 Sent Done Done false
26741 7748 Acked Read 4 Sent Done Done false
26742 7749 Acked Write 10 Sent Done Done false
26743 7750 Acked Write 4 Sent Done Done false
26744 7751 Acked Read 2 Sent Done Done false
26745 7752 Acked Read 9 Sent Done Done false
26746 7753 Acked Read 10 Sent Done Done false
26747 7754 Acked Read 7 Sent Done Done false
26748 7755 Acked Write 7 Sent Done Done false
26749 7756 Acked Write 7 Sent Done Done false
26750 7757 Acked Read 4 Sent Done Done false
26751 7758 Acked Write 6 Sent Done Done false
26752 7759 Acked Write 8 Sent Done Done false
26753 7760 Acked Write 9 Sent Done Done false
26754 7761 Acked Read 6 Sent Done Done false
26755 7762 Acked Read 2 Sent Done Done false
26756 7763 Acked Read 3 Sent Done Done false
26757 7764 Acked Flush 0 Sent Done Done false
26758 7765 Acked Read 5 Sent Done Done false
26759 7766 Acked Write 2 Sent Done Done false
26760 7767 Acked Write 10 Sent Done Done false
26761 7768 Acked Write 1 Sent Done Done false
26762 7769 Acked Flush 0 Sent Done Done false
26763 7770 Acked Write 2 Sent Done Done false
26764 7771 Acked Read 5 Sent Done Done false
26765 7772 Acked Read 10 Sent Done Done false
26766 7773 Acked Read 4 Sent Done Done false
26767 7774 Acked Write 9 Sent Done Done false
26768 7775 Acked Read 7 Sent Done Done false
26769 7776 Acked Write 6 Sent Done Done false
26770 7777 Acked Flush 0 Sent Done Done false
26771 7778 Acked Read 7 Sent Done Done false
26772 7779 Acked Read 2 Sent Done Done false
26773 7780 Acked Write 6 Sent Done Done false
26774 7781 Acked Write 1 Sent Done Done false
26775 7782 Acked Write 10 Sent Done Done false
26776 7783 Acked Read 1 Sent Done Done false
26777 7784 Acked Read 7 Sent Done Done false
26778 7785 Acked Write 2 Sent Done Done false
26779 7786 Acked Read 2 Sent Done Done false
26780 7787 Acked Read 2 Sent Done Done false
26781 7788 Acked Read 2 Sent Done Done false
26782 7789 Acked Read 3 Sent Done Done false
26783 7790 Acked Read 6 Sent Done Done false
26784 7791 Acked Read 6 Sent Done Done false
26785 7792 Acked Write 5 Sent Done Done false
26786 7793 Acked Read 5 Sent Done Done false
26787 7794 Acked Read 1 Sent Done Done false
26788 7795 Acked Read 4 Sent Done Done false
26789 7796 Acked Read 3 Sent Done Done false
26790 7797 Acked Flush 0 Sent Done Done false
26791 7798 Acked Read 6 Sent Done Done false
26792 7799 Acked Read 4 Sent Done Done false
26793 7800 Acked Read 10 Sent Done Done false
26794 7801 Acked Flush 0 Sent Done Done false
26795 7802 Acked Read 6 Sent Done Done false
26796 7803 Acked Write 1 Sent Done Done false
26797 7804 Acked Write 8 Sent Done Done false
26798 7805 Acked Write 5 Sent Done Done false
26799 7806 Acked Write 1 Sent Done Done false
26800 7807 Acked Write 9 Sent Done Done false
26801 7808 Acked Write 1 Sent Done Done false
26802 7809 Acked Read 6 Sent Done Done false
26803 7810 Acked Write 1 Sent Done Done false
26804 7811 Acked Read 9 Sent Done Done false
26805 7812 Acked Flush 0 Sent Done Done false
26806 7813 Acked Read 3 Sent Done Done false
26807 7814 Acked Read 5 Sent Done Done false
26808 7815 Acked Read 4 Sent Done Done false
26809 7816 Acked Read 5 Sent Done Done false
26810 7817 Acked Flush 0 Sent Done Done false
26811 7818 Acked Read 1 Sent Done Done false
26812 7819 Acked Write 7 Sent Done Done false
26813 7820 Acked Read 4 Sent Done Done false
26814 7821 Acked Read 7 Sent Done Done false
26815 7822 Acked Read 1 Sent Done Done false
26816 7823 Acked Read 2 Sent Done Done false
26817 7824 Acked Read 9 Sent Done Done false
26818 7825 Acked Read 3 Sent Done Done false
26819 7826 Acked Write 5 Sent Done Done false
26820 7827 Acked Read 10 Sent Done Done false
26821 7828 Acked Read 10 Sent Done Done false
26822 7829 Acked Read 7 Sent Done Done false
26823 7830 Acked Read 7 Sent Done Done false
26824 7831 Acked Read 9 Sent Done Done false
26825 7832 Acked Read 4 Sent Done Done false
26826 7833 Acked Read 6 Sent Done Done false
26827 7834 Acked Read 5 Sent Done Done false
26828 7835 Acked Write 4 Sent Done Done false
26829 7836 Acked Write 10 Sent Done Done false
26830 7837 Acked Write 3 Sent Done Done false
26831 7838 Acked Read 2 Sent Done Done false
26832 7839 Acked Write 2 Sent Done Done false
26833 7840 Acked Write 8 Sent Done Done false
26834 7841 Acked Write 2 Sent Done Done false
26835 7842 Acked Write 8 Sent Done Done false
26836 7843 Acked Read 8 Sent Done Done false
26837 7844 Acked Read 6 Sent Done Done false
26838 7845 Acked Read 7 Sent Done Done false
26839 7846 Acked Write 7 Sent Done Done false
26840 7847 Acked Write 5 Sent Done Done false
26841 7848 Acked Read 8 Sent Done Done false
26842 7849 Acked Read 2 Sent Done Done false
26843 7850 Acked Read 1 Sent Done Done false
26844 7851 Acked Flush 0 Sent Done Done false
26845 7852 Acked Write 9 Sent Done Done false
26846 7853 Acked Write 8 Sent Done Done false
26847 7854 Acked Write 9 Sent Done Done false
26848 7855 Acked Read 7 Sent Done Done false
26849 7856 Acked Write 4 Sent Done Done false
26850 7857 Acked Write 5 Sent Done Done false
26851 7858 Acked Flush 0 Sent Done Done false
26852 7859 Acked Write 7 Sent Done Done false
26853 7860 Acked Read 2 Sent Done Done false
26854 7861 Acked Read 8 Sent Done Done false
26855 7862 Acked Read 8 Sent Done Done false
26856 7863 Acked Read 9 Sent Done Done false
26857 7864 Acked Write 10 Sent Done Done false
26858 7865 Acked Read 1 Sent Done Done false
26859 7866 Acked Read 5 Sent Done Done false
26860 7867 Acked Flush 0 Sent Done Done false
26861 7868 Acked Read 7 Sent Done Done false
26862 7869 Acked Read 7 Sent Done Done false
26863 7870 Acked Write 8 Sent Done Done false
26864 7871 Acked Write 9 Sent Done Done false
26865 7872 Acked Write 3 Sent Done Done false
26866 7873 Acked Read 1 Sent Done Done false
26867 7874 Acked Read 6 Sent Done Done false
26868 7875 Acked Write 2 Sent Done Done false
26869 7876 Acked Write 2 Sent Done Done false
26870 7877 Acked Read 5 Sent Done Done false
26871 7878 Acked Write 4 Sent Done Done false
26872 7879 Acked Write 6 Sent Done Done false
26873 7880 Acked Write 4 Sent Done Done false
26874 7881 Acked Read 1 Sent Done Done false
26875 7882 Acked Write 5 Sent Done Done false
26876 7883 Acked Flush 0 Sent Done Done false
26877 7884 Acked Read 8 Sent Done Done false
26878 7885 Acked Read 9 Sent Done Done false
26879 7886 Acked Flush 0 Sent Done Done false
26880 7887 Acked Read 4 Sent Done Done false
26881 7888 Acked Flush 0 Sent Done Done false
26882 7889 Acked Write 4 Sent Done Done false
26883 7890 Acked Read 1 Sent Done Done false
26884 7891 Acked Read 3 Sent Done Done false
26885 7892 Acked Write 6 Sent Done Done false
26886 7893 Acked Flush 0 Sent Done Done false
26887 7894 Acked Flush 0 Sent Done Done false
26888 7895 Acked Read 2 Sent Done Done false
26889 7896 Acked Write 1 Sent Done Done false
26890 7897 Acked Read 3 Sent Done Done false
26891 7898 Acked Read 10 Sent Done Done false
26892 7899 Acked Read 5 Sent Done Done false
26893 7900 Acked Read 9 Sent Done Done false
26894 7901 Acked Flush 0 Sent Done Done false
26895 7902 Acked Read 8 Sent Done Done false
26896 7903 Acked Write 1 Sent Done Done false
26897 7904 Acked Read 9 Sent Done Done false
26898 7905 Acked Write 5 Sent Done Done false
26899 7906 Acked Write 3 Sent Done Done false
26900 7907 Acked Write 1 Sent Done Done false
26901 7908 Acked Write 5 Sent Done Done false
26902 7909 Acked Read 4 Sent Done Done false
26903 7910 Acked Flush 0 Sent Done Done false
26904 7911 Acked Write 3 Sent Done Done false
26905 7912 Acked Write 1 Sent Done Done false
26906 7913 Acked Write 5 Sent Done Done false
26907 7914 Acked Write 9 Sent Done Done false
26908 7915 Acked Flush 0 Sent Done Done false
26909 7916 Acked Read 4 Sent Done Done false
26910 7917 Acked Write 10 Sent Done Done false
26911 7918 Acked Write 10 Sent Done Done false
26912 7919 Acked Read 8 Sent Done Done false
26913 7920 Acked Write 4 Sent Done Done false
26914 7921 Acked Write 10 Sent Done Done false
26915 7922 Acked Flush 0 Sent Done Done false
26916 7923 Acked Read 1 Sent Done Done false
26917 7924 Acked Read 9 Sent Done Done false
26918 7925 Acked Write 8 Sent Done Done false
26919 7926 Acked Flush 0 Sent Done Done false
26920 7927 Acked Read 7 Sent Done Done false
26921 7928 Acked Read 3 Sent Done Done false
26922 7929 Acked Write 5 Sent Done Done false
26923 7930 Acked Read 7 Sent Done Done false
26924 7931 Acked Write 3 Sent Done Done false
26925 7932 Acked Write 10 Sent Done Done false
26926 7933 Acked Write 4 Sent Done Done false
26927 7934 Acked Write 4 Sent Done Done false
26928 7935 Acked Write 3 Sent Done Done false
26929 7936 Acked Write 1 Sent Done Done false
26930 7937 Acked Write 10 Sent Done Done false
26931 7938 Acked Write 9 Sent Done Done false
26932 7939 Acked Read 2 Sent Done Done false
26933 7940 Acked Read 3 Sent Done Done false
26934 7941 Acked Write 9 Sent Done Done false
26935 7942 Acked Read 8 Sent Done Done false
26936 STATES DS:0 DS:1 DS:2 TOTAL
26937 Sent 301 0 0 301
26938 Done 0 301 301 602
26939 Skipped 0 0 0 0
26940 Error 0 0 0 0
26941 Last Flush: Some(JobId(7641)) Some(JobId(7926)) Some(JobId(7926))
26942 Downstairs last five completed: 7641 7640 7639 7638 7637
26943 Upstairs last five completed: 7942 7941 7940 7939 7938
26944 CLIENT: Up:0 ds:301 act:2
269452025-03-17T22:22:14.162ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
269462025-03-17T22:22:15.164ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
269472025-03-17T22:22:16.166ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
26948 ----------------------------------------------------------------
26949 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26950 States: Offline Active Active
26951 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
26952 7642 Acked Write 9 Sent Done Done false
26953 7643 Acked Write 1 Sent Done Done false
26954 7644 Acked Write 9 Sent Done Done false
26955 7645 Acked Read 9 Sent Done Done false
26956 7646 Acked Write 2 Sent Done Done false
26957 7647 Acked Write 2 Sent Done Done false
26958 7648 Acked Write 4 Sent Done Done false
26959 7649 Acked Write 1 Sent Done Done false
26960 7650 Acked Read 9 Sent Done Done false
26961 7651 Acked Write 5 Sent Done Done false
26962 7652 Acked Write 3 Sent Done Done false
26963 7653 Acked Write 10 Sent Done Done false
26964 7654 Acked Flush 0 Sent Done Done false
26965 7655 Acked Write 7 Sent Done Done false
26966 7656 Acked Write 2 Sent Done Done false
26967 7657 Acked Read 10 Sent Done Done false
26968 7658 Acked Write 5 Sent Done Done false
26969 7659 Acked Read 5 Sent Done Done false
26970 7660 Acked Read 10 Sent Done Done false
26971 7661 Acked Write 4 Sent Done Done false
26972 7662 Acked Write 6 Sent Done Done false
26973 7663 Acked Read 4 Sent Done Done false
26974 7664 Acked Write 7 Sent Done Done false
26975 7665 Acked Read 3 Sent Done Done false
26976 7666 Acked Read 2 Sent Done Done false
26977 7667 Acked Flush 0 Sent Done Done false
26978 7668 Acked Flush 0 Sent Done Done false
26979 7669 Acked Write 6 Sent Done Done false
26980 7670 Acked Read 10 Sent Done Done false
26981 7671 Acked Read 10 Sent Done Done false
26982 7672 Acked Write 8 Sent Done Done false
26983 7673 Acked Write 1 Sent Done Done false
26984 7674 Acked Read 2 Sent Done Done false
26985 7675 Acked Read 7 Sent Done Done false
26986 7676 Acked Read 7 Sent Done Done false
26987 7677 Acked Flush 0 Sent Done Done false
26988 7678 Acked Read 4 Sent Done Done false
26989 7679 Acked Write 4 Sent Done Done false
26990 7680 Acked Flush 0 Sent Done Done false
26991 7681 Acked Write 9 Sent Done Done false
26992 7682 Acked Read 9 Sent Done Done false
26993 7683 Acked Flush 0 Sent Done Done false
26994 7684 Acked Flush 0 Sent Done Done false
26995 7685 Acked Read 6 Sent Done Done false
26996 7686 Acked Flush 0 Sent Done Done false
26997 7687 Acked Write 2 Sent Done Done false
26998 7688 Acked Write 10 Sent Done Done false
26999 7689 Acked Write 3 Sent Done Done false
27000 7690 Acked Read 2 Sent Done Done false
27001 7691 Acked Read 7 Sent Done Done false
27002 7692 Acked Read 1 Sent Done Done false
27003 7693 Acked Read 1 Sent Done Done false
27004 7694 Acked Read 7 Sent Done Done false
27005 7695 Acked Write 3 Sent Done Done false
27006 7696 Acked Read 3 Sent Done Done false
27007 7697 Acked Read 8 Sent Done Done false
27008 7698 Acked Write 7 Sent Done Done false
27009 7699 Acked Read 4 Sent Done Done false
27010 7700 Acked Read 7 Sent Done Done false
27011 7701 Acked Read 6 Sent Done Done false
27012 7702 Acked Read 8 Sent Done Done false
27013 7703 Acked Read 5 Sent Done Done false
27014 7704 Acked Read 6 Sent Done Done false
27015 7705 Acked Write 6 Sent Done Done false
27016 7706 Acked Read 2 Sent Done Done false
27017 7707 Acked Read 8 Sent Done Done false
27018 7708 Acked Read 9 Sent Done Done false
27019 7709 Acked Read 2 Sent Done Done false
27020 7710 Acked Read 8 Sent Done Done false
27021 7711 Acked Read 5 Sent Done Done false
27022 7712 Acked Write 9 Sent Done Done false
27023 7713 Acked Write 4 Sent Done Done false
27024 7714 Acked Read 9 Sent Done Done false
27025 7715 Acked Read 4 Sent Done Done false
27026 7716 Acked Flush 0 Sent Done Done false
27027 7717 Acked Read 7 Sent Done Done false
27028 7718 Acked Read 5 Sent Done Done false
27029 7719 Acked Read 9 Sent Done Done false
27030 7720 Acked Write 9 Sent Done Done false
27031 7721 Acked Write 2 Sent Done Done false
27032 7722 Acked Read 9 Sent Done Done false
27033 7723 Acked Read 7 Sent Done Done false
27034 7724 Acked Read 6 Sent Done Done false
27035 7725 Acked Read 8 Sent Done Done false
27036 7726 Acked Write 6 Sent Done Done false
27037 7727 Acked Write 3 Sent Done Done false
27038 7728 Acked Flush 0 Sent Done Done false
27039 7729 Acked Flush 0 Sent Done Done false
27040 7730 Acked Write 4 Sent Done Done false
27041 7731 Acked Read 3 Sent Done Done false
27042 7732 Acked Write 5 Sent Done Done false
27043 7733 Acked Read 4 Sent Done Done false
27044 7734 Acked Read 4 Sent Done Done false
27045 7735 Acked Write 1 Sent Done Done false
27046 7736 Acked Read 9 Sent Done Done false
27047 7737 Acked Read 10 Sent Done Done false
27048 7738 Acked Read 8 Sent Done Done false
27049 7739 Acked Read 7 Sent Done Done false
27050 7740 Acked Write 4 Sent Done Done false
27051 7741 Acked Read 2 Sent Done Done false
27052 7742 Acked Write 1 Sent Done Done false
27053 7743 Acked Flush 0 Sent Done Done false
27054 7744 Acked Write 10 Sent Done Done false
27055 7745 Acked Read 3 Sent Done Done false
27056 7746 Acked Read 5 Sent Done Done false
27057 7747 Acked Read 2 Sent Done Done false
27058 7748 Acked Read 4 Sent Done Done false
27059 7749 Acked Write 10 Sent Done Done false
27060 7750 Acked Write 4 Sent Done Done false
27061 7751 Acked Read 2 Sent Done Done false
27062 7752 Acked Read 9 Sent Done Done false
27063 7753 Acked Read 10 Sent Done Done false
27064 7754 Acked Read 7 Sent Done Done false
27065 7755 Acked Write 7 Sent Done Done false
27066 7756 Acked Write 7 Sent Done Done false
27067 7757 Acked Read 4 Sent Done Done false
27068 7758 Acked Write 6 Sent Done Done false
27069 7759 Acked Write 8 Sent Done Done false
27070 7760 Acked Write 9 Sent Done Done false
27071 7761 Acked Read 6 Sent Done Done false
27072 7762 Acked Read 2 Sent Done Done false
27073 7763 Acked Read 3 Sent Done Done false
27074 7764 Acked Flush 0 Sent Done Done false
27075 7765 Acked Read 5 Sent Done Done false
27076 7766 Acked Write 2 Sent Done Done false
27077 7767 Acked Write 10 Sent Done Done false
27078 7768 Acked Write 1 Sent Done Done false
27079 7769 Acked Flush 0 Sent Done Done false
27080 7770 Acked Write 2 Sent Done Done false
27081 7771 Acked Read 5 Sent Done Done false
27082 7772 Acked Read 10 Sent Done Done false
27083 7773 Acked Read 4 Sent Done Done false
27084 7774 Acked Write 9 Sent Done Done false
27085 7775 Acked Read 7 Sent Done Done false
27086 7776 Acked Write 6 Sent Done Done false
27087 7777 Acked Flush 0 Sent Done Done false
27088 7778 Acked Read 7 Sent Done Done false
27089 7779 Acked Read 2 Sent Done Done false
27090 7780 Acked Write 6 Sent Done Done false
27091 7781 Acked Write 1 Sent Done Done false
27092 7782 Acked Write 10 Sent Done Done false
27093 7783 Acked Read 1 Sent Done Done false
27094 7784 Acked Read 7 Sent Done Done false
27095 7785 Acked Write 2 Sent Done Done false
27096 7786 Acked Read 2 Sent Done Done false
27097 7787 Acked Read 2 Sent Done Done false
27098 7788 Acked Read 2 Sent Done Done false
27099 7789 Acked Read 3 Sent Done Done false
27100 7790 Acked Read 6 Sent Done Done false
27101 7791 Acked Read 6 Sent Done Done false
27102 7792 Acked Write 5 Sent Done Done false
27103 7793 Acked Read 5 Sent Done Done false
27104 7794 Acked Read 1 Sent Done Done false
27105 7795 Acked Read 4 Sent Done Done false
27106 7796 Acked Read 3 Sent Done Done false
27107 7797 Acked Flush 0 Sent Done Done false
27108 7798 Acked Read 6 Sent Done Done false
27109 7799 Acked Read 4 Sent Done Done false
27110 7800 Acked Read 10 Sent Done Done false
27111 7801 Acked Flush 0 Sent Done Done false
27112 7802 Acked Read 6 Sent Done Done false
27113 7803 Acked Write 1 Sent Done Done false
27114 7804 Acked Write 8 Sent Done Done false
27115 7805 Acked Write 5 Sent Done Done false
27116 7806 Acked Write 1 Sent Done Done false
27117 7807 Acked Write 9 Sent Done Done false
27118 7808 Acked Write 1 Sent Done Done false
27119 7809 Acked Read 6 Sent Done Done false
27120 7810 Acked Write 1 Sent Done Done false
27121 7811 Acked Read 9 Sent Done Done false
27122 7812 Acked Flush 0 Sent Done Done false
27123 7813 Acked Read 3 Sent Done Done false
27124 7814 Acked Read 5 Sent Done Done false
27125 7815 Acked Read 4 Sent Done Done false
27126 7816 Acked Read 5 Sent Done Done false
27127 7817 Acked Flush 0 Sent Done Done false
27128 7818 Acked Read 1 Sent Done Done false
27129 7819 Acked Write 7 Sent Done Done false
27130 7820 Acked Read 4 Sent Done Done false
27131 7821 Acked Read 7 Sent Done Done false
27132 7822 Acked Read 1 Sent Done Done false
27133 7823 Acked Read 2 Sent Done Done false
27134 7824 Acked Read 9 Sent Done Done false
27135 7825 Acked Read 3 Sent Done Done false
27136 7826 Acked Write 5 Sent Done Done false
27137 7827 Acked Read 10 Sent Done Done false
27138 7828 Acked Read 10 Sent Done Done false
27139 7829 Acked Read 7 Sent Done Done false
27140 7830 Acked Read 7 Sent Done Done false
27141 7831 Acked Read 9 Sent Done Done false
27142 7832 Acked Read 4 Sent Done Done false
27143 7833 Acked Read 6 Sent Done Done false
27144 7834 Acked Read 5 Sent Done Done false
27145 7835 Acked Write 4 Sent Done Done false
27146 7836 Acked Write 10 Sent Done Done false
27147 7837 Acked Write 3 Sent Done Done false
27148 7838 Acked Read 2 Sent Done Done false
27149 7839 Acked Write 2 Sent Done Done false
27150 7840 Acked Write 8 Sent Done Done false
27151 7841 Acked Write 2 Sent Done Done false
27152 7842 Acked Write 8 Sent Done Done false
27153 7843 Acked Read 8 Sent Done Done false
27154 7844 Acked Read 6 Sent Done Done false
27155 7845 Acked Read 7 Sent Done Done false
27156 7846 Acked Write 7 Sent Done Done false
27157 7847 Acked Write 5 Sent Done Done false
27158 7848 Acked Read 8 Sent Done Done false
27159 7849 Acked Read 2 Sent Done Done false
27160 7850 Acked Read 1 Sent Done Done false
27161 7851 Acked Flush 0 Sent Done Done false
27162 7852 Acked Write 9 Sent Done Done false
27163 7853 Acked Write 8 Sent Done Done false
27164 7854 Acked Write 9 Sent Done Done false
27165 7855 Acked Read 7 Sent Done Done false
27166 7856 Acked Write 4 Sent Done Done false
27167 7857 Acked Write 5 Sent Done Done false
27168 7858 Acked Flush 0 Sent Done Done false
27169 7859 Acked Write 7 Sent Done Done false
27170 7860 Acked Read 2 Sent Done Done false
27171 7861 Acked Read 8 Sent Done Done false
27172 7862 Acked Read 8 Sent Done Done false
27173 7863 Acked Read 9 Sent Done Done false
27174 7864 Acked Write 10 Sent Done Done false
27175 7865 Acked Read 1 Sent Done Done false
27176 7866 Acked Read 5 Sent Done Done false
27177 7867 Acked Flush 0 Sent Done Done false
27178 7868 Acked Read 7 Sent Done Done false
27179 7869 Acked Read 7 Sent Done Done false
27180 7870 Acked Write 8 Sent Done Done false
27181 7871 Acked Write 9 Sent Done Done false
27182 7872 Acked Write 3 Sent Done Done false
27183 7873 Acked Read 1 Sent Done Done false
27184 7874 Acked Read 6 Sent Done Done false
27185 7875 Acked Write 2 Sent Done Done false
27186 7876 Acked Write 2 Sent Done Done false
27187 7877 Acked Read 5 Sent Done Done false
27188 7878 Acked Write 4 Sent Done Done false
27189 7879 Acked Write 6 Sent Done Done false
27190 7880 Acked Write 4 Sent Done Done false
27191 7881 Acked Read 1 Sent Done Done false
27192 7882 Acked Write 5 Sent Done Done false
27193 7883 Acked Flush 0 Sent Done Done false
27194 7884 Acked Read 8 Sent Done Done false
27195 7885 Acked Read 9 Sent Done Done false
27196 7886 Acked Flush 0 Sent Done Done false
27197 7887 Acked Read 4 Sent Done Done false
27198 7888 Acked Flush 0 Sent Done Done false
27199 7889 Acked Write 4 Sent Done Done false
27200 7890 Acked Read 1 Sent Done Done false
27201 7891 Acked Read 3 Sent Done Done false
27202 7892 Acked Write 6 Sent Done Done false
27203 7893 Acked Flush 0 Sent Done Done false
27204 7894 Acked Flush 0 Sent Done Done false
27205 7895 Acked Read 2 Sent Done Done false
27206 7896 Acked Write 1 Sent Done Done false
27207 7897 Acked Read 3 Sent Done Done false
27208 7898 Acked Read 10 Sent Done Done false
27209 7899 Acked Read 5 Sent Done Done false
27210 7900 Acked Read 9 Sent Done Done false
27211 7901 Acked Flush 0 Sent Done Done false
27212 7902 Acked Read 8 Sent Done Done false
27213 7903 Acked Write 1 Sent Done Done false
27214 7904 Acked Read 9 Sent Done Done false
27215 7905 Acked Write 5 Sent Done Done false
27216 7906 Acked Write 3 Sent Done Done false
27217 7907 Acked Write 1 Sent Done Done false
27218 7908 Acked Write 5 Sent Done Done false
27219 7909 Acked Read 4 Sent Done Done false
27220 7910 Acked Flush 0 Sent Done Done false
27221 7911 Acked Write 3 Sent Done Done false
27222 7912 Acked Write 1 Sent Done Done false
27223 7913 Acked Write 5 Sent Done Done false
27224 7914 Acked Write 9 Sent Done Done false
27225 7915 Acked Flush 0 Sent Done Done false
27226 7916 Acked Read 4 Sent Done Done false
27227 7917 Acked Write 10 Sent Done Done false
27228 7918 Acked Write 10 Sent Done Done false
27229 7919 Acked Read 8 Sent Done Done false
27230 7920 Acked Write 4 Sent Done Done false
27231 7921 Acked Write 10 Sent Done Done false
27232 7922 Acked Flush 0 Sent Done Done false
27233 7923 Acked Read 1 Sent Done Done false
27234 7924 Acked Read 9 Sent Done Done false
27235 7925 Acked Write 8 Sent Done Done false
27236 7926 Acked Flush 0 Sent Done Done false
27237 7927 Acked Read 7 Sent Done Done false
27238 7928 Acked Read 3 Sent Done Done false
27239 7929 Acked Write 5 Sent Done Done false
27240 7930 Acked Read 7 Sent Done Done false
27241 7931 Acked Write 3 Sent Done Done false
27242 7932 Acked Write 10 Sent Done Done false
27243 7933 Acked Write 4 Sent Done Done false
27244 7934 Acked Write 4 Sent Done Done false
27245 7935 Acked Write 3 Sent Done Done false
27246 7936 Acked Write 1 Sent Done Done false
27247 7937 Acked Write 10 Sent Done Done false
27248 7938 Acked Write 9 Sent Done Done false
27249 7939 Acked Read 2 Sent Done Done false
27250 7940 Acked Read 3 Sent Done Done false
27251 7941 Acked Write 9 Sent Done Done false
27252 7942 Acked Read 8 Sent Done Done false
27253 STATES DS:0 DS:1 DS:2 TOTAL
27254 Sent 301 0 0 301
27255 Done 0 301 301 602
27256 Skipped 0 0 0 0
27257 Error 0 0 0 0
27258 Last Flush: Some(JobId(7641)) Some(JobId(7926)) Some(JobId(7926))
27259 Downstairs last five completed: 7641 7640 7639 7638 7637
27260 Upstairs last five completed: 7942 7941 7940 7939 7938
27261 CLIENT: Up:0 ds:301 act:2
272622025-03-17T22:22:17.571ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
272632025-03-17T22:22:18.168ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
272642025-03-17T22:22:19.170ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
272652025-03-17T22:22:20.170ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
272662025-03-17T22:22:20.819ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
272672025-03-17T22:22:20.819ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:47827) = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
272682025-03-17T22:22:20.821ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
272692025-03-17T22:22:20.821ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
272702025-03-17T22:22:20.821ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
272712025-03-17T22:22:20.821ZINFOcrucible: send last flush ID to this DS: Some(JobId(7641)) = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
272722025-03-17T22:22:20.821ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
272732025-03-17T22:22:20.821ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(7641)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
272742025-03-17T22:22:20.821ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(7641)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
27275 ----------------------------------------------------------------
27276 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:16
27277 States: Active Active Active
27278 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27279 7927 Acked Read 7 Done Done Done true
27280 7928 Acked Read 3 Done Done Done true
27281 7929 Acked Write 5 Done Done Done true
27282 7930 Acked Read 7 Done Done Done true
27283 7931 Acked Write 3 Done Done Done true
27284 7932 Acked Write 10 Done Done Done true
27285 7933 Acked Write 4 Done Done Done true
27286 7934 Acked Write 4 Done Done Done true
27287 7935 Acked Write 3 Done Done Done true
27288 7936 Acked Write 1 Done Done Done true
27289 7937 Acked Write 10 Done Done Done true
27290 7938 Acked Write 9 Done Done Done true
27291 7939 Acked Read 2 Done Done Done true
27292 7940 Acked Read 3 Done Done Done true
27293 7941 Acked Write 9 Done Done Done true
27294 7942 Acked Read 8 Done Done Done true
27295 STATES DS:0 DS:1 DS:2 TOTAL
27296 Sent 0 0 0 0
27297 Done 16 16 16 48
27298 Skipped 0 0 0 0
27299 Error 0 0 0 0
27300 Last Flush: Some(JobId(7926)) Some(JobId(7926)) Some(JobId(7926))
27301 Downstairs last five completed: 7926 7925 7924 7923 7922
27302 Upstairs last five completed: 7942 7941 7940 7939 7938
27303 CLIENT: Up:0 ds:16 act:3
27304 ----------------------------------------------------------------
27305 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
27306 Downstairs last five completed: 7943 7942 7941 7940 7939
27307 Upstairs last five completed: 7943 7942 7941 7940 7939
27308 CLIENT: Up:0 ds:0 act:3
27309 Replay: All jobs finished
27310 Main task has work to do, go find it
27311 got dsc Stop(2)
27312 stop 2
27313 Mar 17 22:22:25.073 INFO request completed[8830] Got stop action so:false kr:false
27314 , latency_us: 373, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 8c1a1cee-a08c-44fc-adc0-d0a5d64d1d11, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
273152025-03-17T22:22:25.073ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
273162025-03-17T22:22:25.074ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
273172025-03-17T22:22:25.074ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
273182025-03-17T22:22:25.074ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
27319 Mar 17 22:22:25.074 INFO request completed, latency_us: 199, response_code: 200, uri: /state/cid/2, method: GET, req_id: 42231ba4-061c-4273-9834-ccf1a4aa77e7, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
27320 [8830] Exited with: ExitStatus(unix_wait_status(9))
27321 [8830][2] reports Exit
27322 Mar 17 22:22:29.077 INFO request completed, latency_us: 214, response_code: 200, uri: /state/cid/2, method: GET, req_id: 04d45ed5-9401-4bc6-a512-a46a41ab7082, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
27323 001/300 Write block 2969 len 24576 data: 6 4 4 5 6 4
27324 002/300 Read block 2820 len 36864
27325 003/300 Read block 1097 len 36864
27326 004/300 Read block 3325 len 4096
27327 005/300 Read block 85 len 24576
27328 006/300 Read block 522 len 36864
27329 007/300 Write block 4667 len 16384 data: 5 4 4 3
27330 008/300 Write block 4574 len 16384 data: 5 7 4 4
27331 009/300 Read block 2755 len 20480
27332 010/300 Write block 625 len 40960 data: 6 6 6 7 7 7 7 7 6 6
27333 011/300 Write block 2104 len 12288 data: 4 6 8
27334 012/300 Write block 4556 len 12288 data: 5 3 4
27335 013/300 Read block 1431 len 36864
27336 014/300 Read block 339 len 16384
27337 015/300 Flush
27338 016/300 Write block 2373 len 4096 data: 3
27339 017/300 Read block 3536 len 12288
27340 018/300 Read block 1245 len 32768
27341 019/300 Write block 3324 len 32768 data: 5 5 5 6 5 4 5 7
27342 020/300 Write block 997 len 12288 data: 5 5 3
27343 021/300 Read block 2878 len 12288
27344 022/300 Write block 4637 len 28672 data: 6 6 6 4 5 5 4
27345 023/300 Read block 2879 len 4096
27346 024/300 Flush
27347 025/300 Write block 110 len 4096 data: 4
27348 026/300 Read block 2363 len 4096
27349 027/300 Read block 1957 len 36864
27350 028/300 Write block 4411 len 40960 data: 4 3 3 2 2 4 4 4 5 7
27351 029/300 Write block 2668 len 8192 data: 7 3
27352 030/300 Read block 161 len 16384
27353 031/300 Read block 2820 len 16384
27354 032/300 Flush
27355 033/300 Write block 4331 len 40960 data: 7 7 6 5 6 5 9 9 9 8
27356 034/300 Read block 784 len 32768
27357 035/300 Read block 3781 len 4096
27358 036/300 Flush
27359 037/300 Write block 3160 len 28672 data: 4 3 5 5 6 6 6
27360 038/300 Write block 2843 len 4096 data: 4
27361 039/300 Read block 3946 len 20480
27362 040/300 Read block 3940 len 28672
27363 041/300 Flush
27364 042/300 Read block 2860 len 20480
27365 043/300 Write block 1951 len 20480 data: 4 6 6 4 4
27366 044/300 Write block 4130 len 12288 data: 4 3 3
27367 045/300 Read block 3797 len 40960
27368 046/300 Read block 1038 len 8192
27369 047/300 Read block 2857 len 32768
27370 048/300 Flush
27371 049/300 Write block 4173 len 4096 data: 6
27372 050/300 Read block 1445 len 16384
27373 051/300 Read block 3132 len 32768
27374 052/300 Read block 3958 len 8192
27375 053/300 Write block 1141 len 4096 data: 5
27376 054/300 Read block 335 len 16384
27377 055/300 Read block 4812 len 20480
27378 056/300 Flush
27379 057/300 Read block 787 len 4096
27380 058/300 Read block 137 len 20480
27381 059/300 Read block 4337 len 36864
27382 060/300 Write block 1331 len 28672 data: 5 4 4 5 5 4 3
27383 061/300 Read block 2364 len 4096
27384 062/300 Read block 2213 len 4096
27385 063/300 Write block 3254 len 16384 data: 5 5 4 4
27386 064/300 Write block 2620 len 24576 data: 6 6 5 5 5 5
27387 065/300 Write block 472 len 4096 data: 4
27388 066/300 Read block 2992 len 4096
27389 067/300 Write block 3846 len 28672 data: 3 3 3 3 2 2 2
27390 068/300 Write block 2338 len 40960 data: 6 6 4 4 4 4 4 4 3 3
27391 069/300 Read block 1528 len 24576
27392 070/300 Read block 4673 len 24576
27393 071/300 Write block 4560 len 4096 data: 3
27394 072/300 Read block 2241 len 20480
27395 073/300 Read block 3407 len 40960
27396 074/300 Write block 2201 len 28672 data: 8 8 9 7 6 9 10
27397 075/300 Read block 1647 len 16384
27398 076/300 Read block 101 len 4096
27399 077/300 Read block 3061 len 24576
27400 078/300 Write block 3865 len 32768 data: 5 4 4 4 4 5 6 5
27401 079/300 Write block 2644 len 8192 data: 4 3
27402 080/300 Read block 2621 len 8192
27403 081/300 Read block 745 len 12288
27404 082/300 Write block 549 len 24576 data: 5 5 4 4 6 6
27405 083/300 Flush
27406 084/300 Write block 3974 len 4096 data: 3
27407 085/300 Write block 4293 len 28672 data: 4 4 4 5 7 8 9
27408 086/300 Read block 3299 len 8192
27409 087/300 Write block 2560 len 12288 data: 3 3 5
27410 088/300 Write block 2888 len 24576 data: 7 8 7 5 6 6
27411 089/300 Write block 3426 len 16384 data: 5 5 5 5
27412 090/300 Read block 3794 len 36864
27413 091/300 Write block 4295 len 28672 data: 5 6 8 9 10 9 11
27414 092/300 Write block 2638 len 36864 data: 3 3 2 3 3 4 5 4 4
27415 093/300 Flush
27416 094/300 Read block 4552 len 36864
27417 095/300 Read block 572 len 16384
27418 096/300 Read block 1253 len 24576
27419 097/300 Flush
27420 098/300 Write block 2023 len 24576 data: 5 4 4 3 2 3
27421 099/300 Read block 178 len 28672
27422 100/300 Write block 1645 len 28672 data: 8 8 8 7 5 3 4
27423 101/300 Flush
27424 102/300 Read block 628 len 8192
27425 103/300 Write block 3323 len 20480 data: 5 6 6 6 7
27426 104/300 Read block 1032 len 28672
27427 105/300 Read block 3739 len 8192
27428 106/300 Read block 4453 len 28672
27429 107/300 Write block 2819 len 4096 data: 4
27430 108/300 Read block 4087 len 12288
27431 109/300 Write block 4511 len 24576 data: 5 4 4 5 5 6
27432 110/300 Read block 3126 len 20480
27433 111/300 Read block 1084 len 32768
27434 112/300 Write block 4687 len 4096 data: 8
27435 113/300 Read block 1237 len 24576
27436 114/300 Flush
27437 115/300 Read block 3740 len 8192
27438 116/300 Read block 113 len 20480
27439 117/300 Write block 754 len 20480 data: 5 5 5 5 4
27440 118/300 Write block 2326 len 28672 data: 7 6 6 4 4 5 5
27441 119/300 Write block 1783 len 12288 data: 4 3 4
27442 120/300 Read block 1752 len 8192
27443 121/300 Write block 2653 len 20480 data: 4 3 3 4 5
27444 122/300 Read block 1220 len 24576
27445 123/300 Write block 2116 len 28672 data: 6 8 9 9 8 8 6
27446 124/300 Read block 1692 len 8192
27447 125/300 Write block 1112 len 8192 data: 4 5
27448 126/300 Read block 4343 len 8192
27449 127/300 Flush
27450 128/300 Flush
27451 129/300 Read block 2133 len 12288
27452 130/300 Read block 1607 len 32768
27453 131/300 Flush
27454 132/300 Read block 2875 len 12288
27455 133/300 Read block 3383 len 20480
27456 134/300 Flush
27457 135/300 Write block 1481 len 20480 data: 9 8 9 8 9
27458 136/300 Read block 3718 len 24576
27459 137/300 Read block 3070 len 12288
27460 138/300 Write block 273 len 24576 data: 6 5 5 4 4 4
27461 139/300 Write block 4236 len 8192 data: 5 4
27462 140/300 Read block 199 len 32768
27463 141/300 Write block 2120 len 28672 data: 9 9 7 6 4 3 3
27464 142/300 Read block 4317 len 12288
27465 143/300 Write block 1641 len 40960 data: 8 9 7 7 9 9 9 8 6 4
27466 144/300 Write block 4053 len 40960 data: 6 7 6 9 9 9 8 8 8 8
27467 145/300 Flush
27468 146/300 Read block 3570 len 36864
27469 147/300 Read block 1096 len 4096
27470 148/300 Read block 3981 len 24576
27471 149/300 Flush
27472 150/300 Read block 3224 len 24576
27473 151/300 Write block 2454 len 28672 data: 3 3 3 4 5 4 4
27474 152/300 Read block 4488 len 28672
27475 153/300 Write block 1975 len 28672 data: 4 5 6 7 6 6 7
27476 154/300 Write block 1622 len 4096 data: 5
27477 155/300 Read block 2626 len 24576
27478 156/300 Write block 744 len 4096 data: 6
27479 157/300 Flush
27480 158/300 Read block 2826 len 28672
27481 159/300 Write block 4727 len 8192 data: 5 6
27482 160/300 Write block 4974 len 40960 data: 4 5 5 5 5 4 5 5 4 4
27483 161/300 Read block 696 len 24576
27484 162/300 Read block 1311 len 28672
27485 163/300 Read block 2116 len 16384
27486 164/300 Write block 2128 len 12288 data: 4 4 5
27487 165/300 Write block 4209 len 16384 data: 4 5 5 6
27488 166/300 Read block 3643 len 20480
27489 167/300 Flush
27490 168/300 Read block 3628 len 12288
27491 169/300 Read block 2876 len 40960
27492 170/300 Flush
27493 171/300 Write block 4375 len 16384 data: 6 6 7 5
27494 172/300 Read block 2988 len 36864
27495 173/300 Read block 4394 len 40960
27496 174/300 Read block 4221 len 36864
27497 175/300 Read block 4430 len 16384
27498 176/300 Write block 1370 len 32768 data: 5 5 4 6 6 6 4 4
27499 177/300 Write block 1803 len 16384 data: 5 6 5 4
27500 178/300 Write block 1766 len 28672 data: 3 3 3 4 4 3 4
27501 179/300 Read block 1189 len 8192
27502 180/300 Read block 750 len 12288
27503 181/300 Flush
27504 182/300 Write block 2751 len 40960 data: 3 3 3 4 4 5 7 7 7 7
27505 183/300 Read block 1631 len 16384
27506 184/300 Read block 4903 len 36864
27507 185/300 Read block 913 len 20480
27508 186/300 Write block 1827 len 12288 data: 5 4 5
27509 187/300 Write block 4238 len 8192 data: 4 7
27510 188/300 Write block 2492 len 32768 data: 7 7 7 9 7 8 9 9
27511 189/300 Read block 3539 len 32768
27512 190/300 Write block 3311 len 36864 data: 4 4 4 4 3 2 2 3 4
27513 191/300 Flush
27514 192/300 Write block 4206 len 24576 data: 2 2 5 5 6 6
27515 193/300 Read block 339 len 8192
27516 194/300 Write block 3471 len 20480 data: 4 3 4 3 3
27517 195/300 Read block 1923 len 12288
27518 196/300 Read block 3971 len 20480
27519 197/300 Write block 1388 len 24576 data: 4 5 6 6 7 7
27520 198/300 Write block 4607 len 20480 data: 9 10 9 9 7
27521 199/300 Write block 3865 len 28672 data: 6 5 5 5 5 6 7
27522 200/300 Read block 4376 len 24576
27523 201/300 Write block 4366 len 36864 data: 6 8 8 7 7 8 8 7 6
27524 202/300 Read block 1967 len 12288
27525 203/300 Write block 592 len 28672 data: 10 8 10 10 9 7 7
27526 204/300 Write block 4317 len 28672 data: 4 4 4 3 3 3 2
27527 205/300 Read block 352 len 12288
27528 206/300 Read block 2487 len 16384
27529 207/300 Read block 4039 len 16384
27530 208/300 Flush
27531 209/300 Read block 3829 len 28672
27532 210/300 Flush
27533 211/300 Read block 2873 len 36864
27534 212/300 Write block 168 len 16384 data: 7 7 8 8
27535 213/300 Write block 4240 len 40960 data: 6 7 6 6 6 6 5 6 5 3
27536 214/300 Write block 2610 len 8192 data: 9 7
27537 215/300 Write block 1926 len 20480 data: 8 7 8 6 5
27538 216/300 Read block 4983 len 16384
27539 217/300 Read block 1647 len 36864
27540 218/300 Read block 2317 len 40960
27541 219/300 Write block 1277 len 20480 data: 6 6 5 4 4
27542 220/300 Write block 3772 len 16384 data: 4 4 4 4
27543 221/300 Read block 4564 len 8192
27544 222/300 Write block 4285 len 4096 data: 5
27545 223/300 Read block 4121 len 36864
27546 224/300 Write block 2631 len 32768 data: 4 3 3 2 2 2 3 4
27547 225/300 Write block 1173 len 24576 data: 5 2 3 6 6 7
27548 226/300 Write block 3469 len 24576 data: 2 3 5 4 5 4
27549 227/300 Read block 2541 len 4096
27550 228/300 Write block 2245 len 36864 data: 5 6 6 6 5 4 3 3 4
27551 229/300 Flush
27552 230/300 Read block 2419 len 28672
27553 231/300 Read block 4899 len 28672
27554 232/300 Flush
27555 233/300 Write block 2147 len 40960 data: 6 6 6 5 4 3 3 4 5 3
27556 234/300 Flush
27557 235/300 Flush
27558 236/300 Write block 2219 len 12288 data: 9 8 7
27559 237/300 Write block 4389 len 32768 data: 4 3 3 3 4 4 3 3
27560 238/300 Write block 1854 len 8192 data: 5 5
27561 239/300 Write block 2864 len 28672 data: 7 7 6 4 6 6 6
27562 240/300 Read block 35 len 20480
27563 241/300 Write block 76 len 4096 data: 12
27564 242/300 Write block 3281 len 20480 data: 5 4 5 4 4
27565 243/300 Write block 4691 len 12288 data: 5 5 5
27566 244/300 Read block 3152 len 40960
27567 245/300 Flush
27568 246/300 Read block 1275 len 4096
27569 247/300 Read block 2796 len 12288
27570 248/300 Read block 1133 len 36864
27571 249/300 Read block 4990 len 4096
27572 250/300 Read block 3942 len 4096
27573 251/300 Flush
27574 252/300 Read block 4715 len 32768
27575 253/300 Read block 805 len 16384
27576 254/300 Write block 3167 len 36864 data: 8 10 8 7 6 5 7 7 6
27577 255/300 Write block 4166 len 36864 data: 6 5 6 5 7 7 7 7 6
27578 256/300 Write block 4990 len 12288 data: 4 5 5
27579 257/300 Read block 4779 len 36864
27580 258/300 Write block 1643 len 36864 data: 8 8 10 10 10 9 7 5 5
27581 259/300 Read block 1643 len 8192
27582 260/300 Read block 739 len 16384
27583 261/300 Read block 741 len 36864
27584 262/300 Write block 4338 len 36864 data: 10 10 9 5 7 6 6 5 5
27585 263/300 Read block 1362 len 4096
27586 264/300 Write block 1926 len 20480 data: 9 8 9 7 6
27587 265/300 Read block 187 len 8192
27588 266/300 Read block 426 len 8192
27589 267/300 Read block 613 len 32768
27590 268/300 Read block 3467 len 16384
27591 269/300 Write block 2816 len 12288 data: 5 5 6
27592 270/300 Read block 2192 len 32768
27593 271/300 Read block 4989 len 4096
27594 272/300 Flush
27595 273/300 Read block 2131 len 16384
27596 274/300 Read block 498 len 20480
27597 275/300 Read block 3721 len 16384
27598 276/300 Read block 2095 len 20480
27599 277/300 Write block 288 len 40960 data: 3 2 2 2 4 4 4 4 3 5
27600 278/300 Write block 1141 len 36864 data: 6 5 4 4 4 4 4 4 3
27601 279/300 Write block 138 len 12288 data: 5 5 6
27602 280/300 Write block 4679 len 12288 data: 4 5 6
27603 281/300 Write block 1771 len 36864 data: 4 5 4 4 4 3 5 6 5
27604 282/300 Read block 3689 len 24576
27605 283/300 Read block 4971 len 20480
27606 284/300 Write block 617 len 24576 data: 6 4 4 5 5 5
27607 285/300 Write block 1225 len 32768 data: 8 7 6 7 3 3 3 2
27608 286/300 Flush
27609 287/300 Write block 1526 len 4096 data: 10
27610 288/300 Write block 2407 len 20480 data: 5 5 5 5 5
27611 289/300 Read block 1873 len 8192
27612 290/300 Write block 318 len 20480 data: 8 8 4 4 4
27613 291/300 Write block 3947 len 24576 data: 5 5 5 5 4 5
27614 292/300 Flush
27615 293/300 Write block 4032 len 32768 data: 5 4 4 3 4 4 4 5
27616 294/300 Write block 1624 len 24576 data: 5 4 3 3 3 3
27617 295/300 Read block 82 len 28672
27618 296/300 Write block 2755 len 32768 data: 5 6 8 8 8 8 6 6
27619 297/300 Read block 4504 len 16384
27620 298/300 Read block 4175 len 36864
27621 299/300 Read block 4053 len 36864
27622 300/300 Write block 375 len 20480 data: 7 6 6 5 5
27623 301/300 Write block 2640 len 24576 data: 3 4 4 5 6 5
27624 Main task has work to do, go find it
27625 got dsc Start(2)
27626 start 2
27627 Mar 17 22:22:29.318 INFO request completed, latency_us: 270, response_code: 204, uri: /start/cid/2, method: POST, req_id: 2603f3e8-33bc-4525-a097-d8e7c98e8c74, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
27628 Replay: started 2, returned:Ok(())
27629 ----------------------------------------------------------------
27630 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27631 States: Active Active Offline
27632 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27633 7944 Acked Write 6 Done Done Sent false
27634 7945 Acked Read 9 Done Done Sent false
27635 7946 Acked Read 9 Done Done Sent false
27636 7947 Acked Read 1 Done Done Sent false
27637 7948 Acked Read 6 Done Done Sent false
27638 7949 Acked Read 9 Done Done Sent false
27639 7950 Acked Write 4 Done Done Sent false
27640 7951 Acked Write 4 Done Done Sent false
27641 7952 Acked Read 5 Done Done Sent false
27642 7953 Acked Write 10 Done Done Sent false
27643 7954 Acked Write 3 Done Done Sent false
27644 7955 Acked Write 3 Done Done Sent false
27645 7956 Acked Read 9 Done Done Sent false
27646 7957 Acked Read 4 Done Done Sent false
27647 7958 Acked Flush 0 Done Done Sent false
27648 7959 Acked Write 1 Done Done Sent false
27649 7960 Acked Read 3 Done Done Sent false
27650 7961 Acked Read 8 Done Done Sent false
27651 7962 Acked Write 8 Done Done Sent false
27652 7963 Acked Write 3 Done Done Sent false
27653 7964 Acked Read 3 Done Done Sent false
27654 7965 Acked Write 7 Done Done Sent false
27655 7966 Acked Read 1 Done Done Sent false
27656 7967 Acked Flush 0 Done Done Sent false
27657 7968 Acked Write 1 Done Done Sent false
27658 7969 Acked Read 1 Done Done Sent false
27659 7970 Acked Read 9 Done Done Sent false
27660 7971 Acked Write 10 Done Done Sent false
27661 7972 Acked Write 2 Done Done Sent false
27662 7973 Acked Read 4 Done Done Sent false
27663 7974 Acked Read 4 Done Done Sent false
27664 7975 Acked Flush 0 Done Done Sent false
27665 7976 Acked Write 10 Done Done Sent false
27666 7977 Acked Read 8 Done Done Sent false
27667 7978 Acked Read 1 Done Done Sent false
27668 7979 Acked Flush 0 Done Done Sent false
27669 7980 Acked Write 7 Done Done Sent false
27670 7981 Acked Write 1 Done Done Sent false
27671 7982 Acked Read 5 Done Done Sent false
27672 7983 Acked Read 7 Done Done Sent false
27673 7984 Acked Flush 0 Done Done Sent false
27674 7985 Acked Read 5 Done Done Sent false
27675 7986 Acked Write 5 Done Done Sent false
27676 7987 Acked Write 3 Done Done Sent false
27677 7988 Acked Read 10 Done Done Sent false
27678 7989 Acked Read 2 Done Done Sent false
27679 7990 Acked Read 8 Done Done Sent false
27680 7991 Acked Flush 0 Done Done Sent false
27681 7992 Acked Write 1 Done Done Sent false
27682 7993 Acked Read 4 Done Done Sent false
27683 7994 Acked Read 8 Done Done Sent false
27684 7995 Acked Read 2 Done Done Sent false
27685 7996 Acked Write 1 Done Done Sent false
27686 7997 Acked Read 4 Done Done Sent false
27687 7998 Acked Read 5 Done Done Sent false
27688 7999 Acked Flush 0 Done Done Sent false
27689 8000 Acked Read 1 Done Done Sent false
27690 8001 Acked Read 5 Done Done Sent false
27691 8002 Acked Read 9 Done Done Sent false
27692 8003 Acked Write 7 Done Done Sent false
27693 8004 Acked Read 1 Done Done Sent false
27694 8005 Acked Read 1 Done Done Sent false
27695 8006 Acked Write 4 Done Done Sent false
27696 8007 Acked Write 6 Done Done Sent false
27697 8008 Acked Write 1 Done Done Sent false
27698 8009 Acked Read 1 Done Done Sent false
27699 8010 Acked Write 7 Done Done Sent false
27700 8011 Acked Write 10 Done Done Sent false
27701 8012 Acked Read 6 Done Done Sent false
27702 8013 Acked Read 6 Done Done Sent false
27703 8014 Acked Write 1 Done Done Sent false
27704 8015 Acked Read 5 Done Done Sent false
27705 8016 Acked Read 10 Done Done Sent false
27706 8017 Acked Write 7 Done Done Sent false
27707 8018 Acked Read 4 Done Done Sent false
27708 8019 Acked Read 1 Done Done Sent false
27709 8020 Acked Read 6 Done Done Sent false
27710 8021 Acked Write 8 Done Done Sent false
27711 8022 Acked Write 2 Done Done Sent false
27712 8023 Acked Read 2 Done Done Sent false
27713 8024 Acked Read 3 Done Done Sent false
27714 8025 Acked Write 6 Done Done Sent false
27715 8026 Acked Flush 0 Done Done Sent false
27716 8027 Acked Write 1 Done Done Sent false
27717 8028 Acked Write 7 Done Done Sent false
27718 8029 Acked Read 2 Done Done Sent false
27719 8030 Acked Write 3 Done Done Sent false
27720 8031 Acked Write 6 Done Done Sent false
27721 8032 Acked Write 4 Done Done Sent false
27722 8033 Acked Read 9 Done Done Sent false
27723 8034 Acked Write 7 Done Done Sent false
27724 8035 Acked Write 9 Done Done Sent false
27725 8036 Acked Flush 0 Done Done Sent false
27726 8037 Acked Read 9 Done Done Sent false
27727 8038 Acked Read 4 Done Done Sent false
27728 8039 Acked Read 6 Done Done Sent false
27729 8040 Acked Flush 0 Done Done Sent false
27730 8041 Acked Write 6 Done Done Sent false
27731 8042 Acked Read 7 Done Done Sent false
27732 8043 Acked Write 7 Done Done Sent false
27733 8044 Acked Flush 0 Done Done Sent false
27734 8045 Acked Read 2 Done Done Sent false
27735 8046 Acked Write 5 Done Done Sent false
27736 8047 Acked Read 7 Done Done Sent false
27737 8048 Acked Read 2 Done Done Sent false
27738 8049 Acked Read 7 Done Done Sent false
27739 8050 Acked Write 1 Done Done Sent false
27740 8051 Acked Read 3 Done Done Sent false
27741 8052 Acked Write 6 Done Done Sent false
27742 8053 Acked Read 5 Done Done Sent false
27743 8054 Acked Read 8 Done Done Sent false
27744 8055 Acked Write 1 Done Done Sent false
27745 8056 Acked Read 6 Done Done Sent false
27746 8057 Acked Flush 0 Done Done Sent false
27747 8058 Acked Read 2 Done Done Sent false
27748 8059 Acked Read 5 Done Done Sent false
27749 8060 Acked Write 5 Done Done Sent false
27750 8061 Acked Write 7 Done Done Sent false
27751 8062 Acked Write 3 Done Done Sent false
27752 8063 Acked Read 2 Done Done Sent false
27753 8064 Acked Write 5 Done Done Sent false
27754 8065 Acked Read 6 Done Done Sent false
27755 8066 Acked Write 7 Done Done Sent false
27756 8067 Acked Read 2 Done Done Sent false
27757 8068 Acked Write 2 Done Done Sent false
27758 8069 Acked Read 2 Done Done Sent false
27759 8070 Acked Flush 0 Done Done Sent false
27760 8071 Acked Flush 0 Done Done Sent false
27761 8072 Acked Read 3 Done Done Sent false
27762 8073 Acked Read 8 Done Done Sent false
27763 8074 Acked Flush 0 Done Done Sent false
27764 8075 Acked Read 3 Done Done Sent false
27765 8076 Acked Read 5 Done Done Sent false
27766 8077 Acked Flush 0 Done Done Sent false
27767 8078 Acked Write 5 Done Done Sent false
27768 8079 Acked Read 6 Done Done Sent false
27769 8080 Acked Read 3 Done Done Sent false
27770 8081 Acked Write 6 Done Done Sent false
27771 8082 Acked Write 2 Done Done Sent false
27772 8083 Acked Read 8 Done Done Sent false
27773 8084 Acked Write 7 Done Done Sent false
27774 8085 Acked Read 3 Done Done Sent false
27775 8086 Acked Write 10 Done Done Sent false
27776 8087 Acked Write 10 Done Done Sent false
27777 8088 Acked Flush 0 Done Done Sent false
27778 8089 Acked Read 9 Done Done Sent false
27779 8090 Acked Read 1 Done Done Sent false
27780 8091 Acked Read 6 Done Done Sent false
27781 8092 Acked Flush 0 Done Done Sent false
27782 8093 Acked Read 6 Done Done Sent false
27783 8094 Acked Write 7 Done Done Sent false
27784 8095 Acked Read 7 Done Done Sent false
27785 8096 Acked Write 7 Done Done Sent false
27786 8097 Acked Write 1 Done Done Sent false
27787 8098 Acked Read 6 Done Done Sent false
27788 8099 Acked Write 1 Done Done Sent false
27789 8100 Acked Flush 0 Done Done Sent false
27790 8101 Acked Read 7 Done Done Sent false
27791 8102 Acked Write 2 Done Done Sent false
27792 8103 Acked Write 10 Done Done Sent false
27793 8104 Acked Read 6 Done Done Sent false
27794 8105 Acked Read 7 Done Done Sent false
27795 8106 Acked Read 4 Done Done Sent false
27796 8107 Acked Write 3 Done Done Sent false
27797 8108 Acked Write 4 Done Done Sent false
27798 8109 Acked Read 5 Done Done Sent false
27799 8110 Acked Flush 0 Done Done Sent false
27800 8111 Acked Read 3 Done Done Sent false
27801 8112 Acked Read 10 Done Done Sent false
27802 8113 Acked Flush 0 Done Done Sent false
27803 8114 Acked Write 4 Done Done Sent false
27804 8115 Acked Read 9 Done Done Sent false
27805 8116 Acked Read 10 Done Done Sent false
27806 8117 Acked Read 9 Done Done Sent false
27807 8118 Acked Read 4 Done Done Sent false
27808 8119 Acked Write 8 Done Done Sent false
27809 8120 Acked Write 4 Done Done Sent false
27810 8121 Acked Write 7 Done Done Sent false
27811 8122 Acked Read 2 Done Done Sent false
27812 8123 Acked Read 3 Done Done Sent false
27813 8124 Acked Flush 0 Done Done Sent false
27814 8125 Acked Write 10 Done Done Sent false
27815 8126 Acked Read 4 Done Done Sent false
27816 8127 Acked Read 9 Done Done Sent false
27817 8128 Acked Read 5 Done Done Sent false
27818 8129 Acked Write 3 Done Done Sent false
27819 8130 Acked Write 2 Done Done Sent false
27820 8131 Acked Write 8 Done Done Sent false
27821 8132 Acked Read 8 Done Done Sent false
27822 8133 Acked Write 9 Done Done Sent false
27823 8134 Acked Flush 0 Done Done Sent false
27824 8135 Acked Write 6 Done Done Sent false
27825 8136 Acked Read 2 Done Done Sent false
27826 8137 Acked Write 5 Done Done Sent false
27827 8138 Acked Read 3 Done Done Sent false
27828 8139 Acked Read 5 Done Done Sent false
27829 8140 Acked Write 6 Done Done Sent false
27830 8141 Acked Write 5 Done Done Sent false
27831 8142 Acked Write 7 Done Done Sent false
27832 8143 Acked Read 6 Done Done Sent false
27833 8144 Acked Write 9 Done Done Sent false
27834 8145 Acked Read 3 Done Done Sent false
27835 8146 Acked Write 7 Done Done Sent false
27836 8147 Acked Write 7 Done Done Sent false
27837 8148 Acked Read 3 Done Done Sent false
27838 8149 Acked Read 4 Done Done Sent false
27839 8150 Acked Read 4 Done Done Sent false
27840 8151 Acked Flush 0 Done Done Sent false
27841 8152 Acked Read 7 Done Done Sent false
27842 8153 Acked Flush 0 Done Done Sent false
27843 8154 Acked Read 9 Done Done Sent false
27844 8155 Acked Write 4 Done Done Sent false
27845 8156 Acked Write 10 Done Done Sent false
27846 8157 Acked Write 2 Done Done Sent false
27847 8158 Acked Write 5 Done Done Sent false
27848 8159 Acked Read 4 Done Done Sent false
27849 8160 Acked Read 9 Done Done Sent false
27850 8161 Acked Read 10 Done Done Sent false
27851 8162 Acked Write 5 Done Done Sent false
27852 8163 Acked Write 4 Done Done Sent false
27853 8164 Acked Read 2 Done Done Sent false
27854 8165 Acked Write 1 Done Done Sent false
27855 8166 Acked Read 9 Done Done Sent false
27856 8167 Acked Write 8 Done Done Sent false
27857 8168 Acked Write 6 Done Done Sent false
27858 8169 Acked Write 6 Done Done Sent false
27859 8170 Acked Read 1 Done Done Sent false
27860 8171 Acked Write 9 Done Done Sent false
27861 8172 Acked Flush 0 Done Done Sent false
27862 8173 Acked Read 7 Done Done Sent false
27863 8174 Acked Read 7 Done Done Sent false
27864 8175 Acked Flush 0 Done Done Sent false
27865 8176 Acked Write 10 Done Done Sent false
27866 8177 Acked Flush 0 Done Done Sent false
27867 8178 Acked Flush 0 Done Done Sent false
27868 8179 Acked Write 3 Done Done Sent false
27869 8180 Acked Write 8 Done Done Sent false
27870 8181 Acked Write 2 Done Done Sent false
27871 8182 Acked Write 7 Done Done Sent false
27872 8183 Acked Read 5 Done Done Sent false
27873 8184 Acked Write 1 Done Done Sent false
27874 8185 Acked Write 5 Done Done Sent false
27875 8186 Acked Write 3 Done Done Sent false
27876 8187 Acked Read 10 Done Done Sent false
27877 8188 Acked Flush 0 Done Done Sent false
27878 8189 Acked Read 1 Done Done Sent false
27879 8190 Acked Read 3 Done Done Sent false
27880 8191 Acked Read 9 Done Done Sent false
27881 8192 Acked Read 1 Done Done Sent false
27882 8193 Acked Read 1 Done Done Sent false
27883 8194 Acked Flush 0 Done Done Sent false
27884 8195 Acked Read 8 Done Done Sent false
27885 8196 Acked Read 4 Done Done Sent false
27886 8197 Acked Write 9 Done Done Sent false
27887 8198 Acked Write 9 Done Done Sent false
27888 8199 Acked Write 3 Done Done Sent false
27889 8200 Acked Read 9 Done Done Sent false
27890 8201 Acked Write 9 Done Done Sent false
27891 8202 Acked Read 2 Done Done Sent false
27892 8203 Acked Read 4 Done Done Sent false
27893 8204 Acked Read 9 Done Done Sent false
27894 8205 Acked Write 9 Done Done Sent false
27895 8206 Acked Read 1 Done Done Sent false
27896 8207 Acked Write 5 Done Done Sent false
27897 8208 Acked Read 2 Done Done Sent false
27898 8209 Acked Read 2 Done Done Sent false
27899 8210 Acked Read 8 Done Done Sent false
27900 8211 Acked Read 4 Done Done Sent false
27901 8212 Acked Write 3 Done Done Sent false
27902 8213 Acked Read 8 Done Done Sent false
27903 8214 Acked Read 1 Done Done Sent false
27904 8215 Acked Flush 0 Done Done Sent false
27905 8216 Acked Read 4 Done Done Sent false
27906 8217 Acked Read 5 Done Done Sent false
27907 8218 Acked Read 4 Done Done Sent false
27908 8219 Acked Read 5 Done Done Sent false
27909 8220 Acked Write 10 Done Done Sent false
27910 8221 Acked Write 9 Done Done Sent false
27911 8222 Acked Write 3 Done Done Sent false
27912 8223 Acked Write 3 Done Done Sent false
27913 8224 Acked Write 9 Done Done Sent false
27914 8225 Acked Read 6 Done Done Sent false
27915 8226 Acked Read 5 Done Done Sent false
27916 8227 Acked Write 6 Done Done Sent false
27917 8228 Acked Write 8 Done Done Sent false
27918 8229 Acked Flush 0 Done Done Sent false
27919 8230 Acked Write 1 Done Done Sent false
27920 8231 Acked Write 5 Done Done Sent false
27921 8232 Acked Read 2 Done Done Sent false
27922 8233 Acked Write 5 Done Done Sent false
27923 8234 Acked Write 6 Done Done Sent false
27924 8235 Acked Flush 0 Done Done Sent false
27925 8236 Acked Write 8 Done Done Sent false
27926 8237 Acked Write 6 Done Done Sent false
27927 8238 Acked Read 7 Done Done Sent false
27928 8239 Acked Write 8 Done Done Sent false
27929 8240 Acked Read 4 Done Done Sent false
27930 8241 Acked Read 9 Done Done Sent false
27931 8242 Acked Read 9 Done Done Sent false
27932 8243 Acked Write 5 Done Done Sent false
27933 8244 Acked Write 6 Done Done Sent false
27934 STATES DS:0 DS:1 DS:2 TOTAL
27935 Sent 0 0 301 301
27936 Done 301 301 0 602
27937 Skipped 0 0 0 0
27938 Error 0 0 0 0
27939 Last Flush: Some(JobId(8235)) Some(JobId(8235)) Some(JobId(7943))
27940 Downstairs last five completed: 7943 7942 7941 7940 7939
27941 Upstairs last five completed: 8244 8243 8242 8241 8240
27942 CLIENT: Up:0 ds:301 act:2
279432025-03-17T22:22:30.184ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
279442025-03-17T22:22:31.323ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
279452025-03-17T22:22:32.561ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
279462025-03-17T22:22:33.189ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
27947 ----------------------------------------------------------------
27948 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
27949 States: Active Active Offline
27950 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
27951 7944 Acked Write 6 Done Done Sent false
27952 7945 Acked Read 9 Done Done Sent false
27953 7946 Acked Read 9 Done Done Sent false
27954 7947 Acked Read 1 Done Done Sent false
27955 7948 Acked Read 6 Done Done Sent false
27956 7949 Acked Read 9 Done Done Sent false
27957 7950 Acked Write 4 Done Done Sent false
27958 7951 Acked Write 4 Done Done Sent false
27959 7952 Acked Read 5 Done Done Sent false
27960 7953 Acked Write 10 Done Done Sent false
27961 7954 Acked Write 3 Done Done Sent false
27962 7955 Acked Write 3 Done Done Sent false
27963 7956 Acked Read 9 Done Done Sent false
27964 7957 Acked Read 4 Done Done Sent false
27965 7958 Acked Flush 0 Done Done Sent false
27966 7959 Acked Write 1 Done Done Sent false
27967 7960 Acked Read 3 Done Done Sent false
27968 7961 Acked Read 8 Done Done Sent false
27969 7962 Acked Write 8 Done Done Sent false
27970 7963 Acked Write 3 Done Done Sent false
27971 7964 Acked Read 3 Done Done Sent false
27972 7965 Acked Write 7 Done Done Sent false
27973 7966 Acked Read 1 Done Done Sent false
27974 7967 Acked Flush 0 Done Done Sent false
27975 7968 Acked Write 1 Done Done Sent false
27976 7969 Acked Read 1 Done Done Sent false
27977 7970 Acked Read 9 Done Done Sent false
27978 7971 Acked Write 10 Done Done Sent false
27979 7972 Acked Write 2 Done Done Sent false
27980 7973 Acked Read 4 Done Done Sent false
27981 7974 Acked Read 4 Done Done Sent false
27982 7975 Acked Flush 0 Done Done Sent false
27983 7976 Acked Write 10 Done Done Sent false
27984 7977 Acked Read 8 Done Done Sent false
27985 7978 Acked Read 1 Done Done Sent false
27986 7979 Acked Flush 0 Done Done Sent false
27987 7980 Acked Write 7 Done Done Sent false
27988 7981 Acked Write 1 Done Done Sent false
27989 7982 Acked Read 5 Done Done Sent false
27990 7983 Acked Read 7 Done Done Sent false
27991 7984 Acked Flush 0 Done Done Sent false
27992 7985 Acked Read 5 Done Done Sent false
27993 7986 Acked Write 5 Done Done Sent false
27994 7987 Acked Write 3 Done Done Sent false
27995 7988 Acked Read 10 Done Done Sent false
27996 7989 Acked Read 2 Done Done Sent false
27997 7990 Acked Read 8 Done Done Sent false
27998 7991 Acked Flush 0 Done Done Sent false
27999 7992 Acked Write 1 Done Done Sent false
28000 7993 Acked Read 4 Done Done Sent false
28001 7994 Acked Read 8 Done Done Sent false
28002 7995 Acked Read 2 Done Done Sent false
28003 7996 Acked Write 1 Done Done Sent false
28004 7997 Acked Read 4 Done Done Sent false
28005 7998 Acked Read 5 Done Done Sent false
28006 7999 Acked Flush 0 Done Done Sent false
28007 8000 Acked Read 1 Done Done Sent false
28008 8001 Acked Read 5 Done Done Sent false
28009 8002 Acked Read 9 Done Done Sent false
28010 8003 Acked Write 7 Done Done Sent false
28011 8004 Acked Read 1 Done Done Sent false
28012 8005 Acked Read 1 Done Done Sent false
28013 8006 Acked Write 4 Done Done Sent false
28014 8007 Acked Write 6 Done Done Sent false
28015 8008 Acked Write 1 Done Done Sent false
28016 8009 Acked Read 1 Done Done Sent false
28017 8010 Acked Write 7 Done Done Sent false
28018 8011 Acked Write 10 Done Done Sent false
28019 8012 Acked Read 6 Done Done Sent false
28020 8013 Acked Read 6 Done Done Sent false
28021 8014 Acked Write 1 Done Done Sent false
28022 8015 Acked Read 5 Done Done Sent false
28023 8016 Acked Read 10 Done Done Sent false
28024 8017 Acked Write 7 Done Done Sent false
28025 8018 Acked Read 4 Done Done Sent false
28026 8019 Acked Read 1 Done Done Sent false
28027 8020 Acked Read 6 Done Done Sent false
28028 8021 Acked Write 8 Done Done Sent false
28029 8022 Acked Write 2 Done Done Sent false
28030 8023 Acked Read 2 Done Done Sent false
28031 8024 Acked Read 3 Done Done Sent false
28032 8025 Acked Write 6 Done Done Sent false
28033 8026 Acked Flush 0 Done Done Sent false
28034 8027 Acked Write 1 Done Done Sent false
28035 8028 Acked Write 7 Done Done Sent false
28036 8029 Acked Read 2 Done Done Sent false
28037 8030 Acked Write 3 Done Done Sent false
28038 8031 Acked Write 6 Done Done Sent false
28039 8032 Acked Write 4 Done Done Sent false
28040 8033 Acked Read 9 Done Done Sent false
28041 8034 Acked Write 7 Done Done Sent false
28042 8035 Acked Write 9 Done Done Sent false
28043 8036 Acked Flush 0 Done Done Sent false
28044 8037 Acked Read 9 Done Done Sent false
28045 8038 Acked Read 4 Done Done Sent false
28046 8039 Acked Read 6 Done Done Sent false
28047 8040 Acked Flush 0 Done Done Sent false
28048 8041 Acked Write 6 Done Done Sent false
28049 8042 Acked Read 7 Done Done Sent false
28050 8043 Acked Write 7 Done Done Sent false
28051 8044 Acked Flush 0 Done Done Sent false
28052 8045 Acked Read 2 Done Done Sent false
28053 8046 Acked Write 5 Done Done Sent false
28054 8047 Acked Read 7 Done Done Sent false
28055 8048 Acked Read 2 Done Done Sent false
28056 8049 Acked Read 7 Done Done Sent false
28057 8050 Acked Write 1 Done Done Sent false
28058 8051 Acked Read 3 Done Done Sent false
28059 8052 Acked Write 6 Done Done Sent false
28060 8053 Acked Read 5 Done Done Sent false
28061 8054 Acked Read 8 Done Done Sent false
28062 8055 Acked Write 1 Done Done Sent false
28063 8056 Acked Read 6 Done Done Sent false
28064 8057 Acked Flush 0 Done Done Sent false
28065 8058 Acked Read 2 Done Done Sent false
28066 8059 Acked Read 5 Done Done Sent false
28067 8060 Acked Write 5 Done Done Sent false
28068 8061 Acked Write 7 Done Done Sent false
28069 8062 Acked Write 3 Done Done Sent false
28070 8063 Acked Read 2 Done Done Sent false
28071 8064 Acked Write 5 Done Done Sent false
28072 8065 Acked Read 6 Done Done Sent false
28073 8066 Acked Write 7 Done Done Sent false
28074 8067 Acked Read 2 Done Done Sent false
28075 8068 Acked Write 2 Done Done Sent false
28076 8069 Acked Read 2 Done Done Sent false
28077 8070 Acked Flush 0 Done Done Sent false
28078 8071 Acked Flush 0 Done Done Sent false
28079 8072 Acked Read 3 Done Done Sent false
28080 8073 Acked Read 8 Done Done Sent false
28081 8074 Acked Flush 0 Done Done Sent false
28082 8075 Acked Read 3 Done Done Sent false
28083 8076 Acked Read 5 Done Done Sent false
28084 8077 Acked Flush 0 Done Done Sent false
28085 8078 Acked Write 5 Done Done Sent false
28086 8079 Acked Read 6 Done Done Sent false
28087 8080 Acked Read 3 Done Done Sent false
28088 8081 Acked Write 6 Done Done Sent false
28089 8082 Acked Write 2 Done Done Sent false
28090 8083 Acked Read 8 Done Done Sent false
28091 8084 Acked Write 7 Done Done Sent false
28092 8085 Acked Read 3 Done Done Sent false
28093 8086 Acked Write 10 Done Done Sent false
28094 8087 Acked Write 10 Done Done Sent false
28095 8088 Acked Flush 0 Done Done Sent false
28096 8089 Acked Read 9 Done Done Sent false
28097 8090 Acked Read 1 Done Done Sent false
28098 8091 Acked Read 6 Done Done Sent false
28099 8092 Acked Flush 0 Done Done Sent false
28100 8093 Acked Read 6 Done Done Sent false
28101 8094 Acked Write 7 Done Done Sent false
28102 8095 Acked Read 7 Done Done Sent false
28103 8096 Acked Write 7 Done Done Sent false
28104 8097 Acked Write 1 Done Done Sent false
28105 8098 Acked Read 6 Done Done Sent false
28106 8099 Acked Write 1 Done Done Sent false
28107 8100 Acked Flush 0 Done Done Sent false
28108 8101 Acked Read 7 Done Done Sent false
28109 8102 Acked Write 2 Done Done Sent false
28110 8103 Acked Write 10 Done Done Sent false
28111 8104 Acked Read 6 Done Done Sent false
28112 8105 Acked Read 7 Done Done Sent false
28113 8106 Acked Read 4 Done Done Sent false
28114 8107 Acked Write 3 Done Done Sent false
28115 8108 Acked Write 4 Done Done Sent false
28116 8109 Acked Read 5 Done Done Sent false
28117 8110 Acked Flush 0 Done Done Sent false
28118 8111 Acked Read 3 Done Done Sent false
28119 8112 Acked Read 10 Done Done Sent false
28120 8113 Acked Flush 0 Done Done Sent false
28121 8114 Acked Write 4 Done Done Sent false
28122 8115 Acked Read 9 Done Done Sent false
28123 8116 Acked Read 10 Done Done Sent false
28124 8117 Acked Read 9 Done Done Sent false
28125 8118 Acked Read 4 Done Done Sent false
28126 8119 Acked Write 8 Done Done Sent false
28127 8120 Acked Write 4 Done Done Sent false
28128 8121 Acked Write 7 Done Done Sent false
28129 8122 Acked Read 2 Done Done Sent false
28130 8123 Acked Read 3 Done Done Sent false
28131 8124 Acked Flush 0 Done Done Sent false
28132 8125 Acked Write 10 Done Done Sent false
28133 8126 Acked Read 4 Done Done Sent false
28134 8127 Acked Read 9 Done Done Sent false
28135 8128 Acked Read 5 Done Done Sent false
28136 8129 Acked Write 3 Done Done Sent false
28137 8130 Acked Write 2 Done Done Sent false
28138 8131 Acked Write 8 Done Done Sent false
28139 8132 Acked Read 8 Done Done Sent false
28140 8133 Acked Write 9 Done Done Sent false
28141 8134 Acked Flush 0 Done Done Sent false
28142 8135 Acked Write 6 Done Done Sent false
28143 8136 Acked Read 2 Done Done Sent false
28144 8137 Acked Write 5 Done Done Sent false
28145 8138 Acked Read 3 Done Done Sent false
28146 8139 Acked Read 5 Done Done Sent false
28147 8140 Acked Write 6 Done Done Sent false
28148 8141 Acked Write 5 Done Done Sent false
28149 8142 Acked Write 7 Done Done Sent false
28150 8143 Acked Read 6 Done Done Sent false
28151 8144 Acked Write 9 Done Done Sent false
28152 8145 Acked Read 3 Done Done Sent false
28153 8146 Acked Write 7 Done Done Sent false
28154 8147 Acked Write 7 Done Done Sent false
28155 8148 Acked Read 3 Done Done Sent false
28156 8149 Acked Read 4 Done Done Sent false
28157 8150 Acked Read 4 Done Done Sent false
28158 8151 Acked Flush 0 Done Done Sent false
28159 8152 Acked Read 7 Done Done Sent false
28160 8153 Acked Flush 0 Done Done Sent false
28161 8154 Acked Read 9 Done Done Sent false
28162 8155 Acked Write 4 Done Done Sent false
28163 8156 Acked Write 10 Done Done Sent false
28164 8157 Acked Write 2 Done Done Sent false
28165 8158 Acked Write 5 Done Done Sent false
28166 8159 Acked Read 4 Done Done Sent false
28167 8160 Acked Read 9 Done Done Sent false
28168 8161 Acked Read 10 Done Done Sent false
28169 8162 Acked Write 5 Done Done Sent false
28170 8163 Acked Write 4 Done Done Sent false
28171 8164 Acked Read 2 Done Done Sent false
28172 8165 Acked Write 1 Done Done Sent false
28173 8166 Acked Read 9 Done Done Sent false
28174 8167 Acked Write 8 Done Done Sent false
28175 8168 Acked Write 6 Done Done Sent false
28176 8169 Acked Write 6 Done Done Sent false
28177 8170 Acked Read 1 Done Done Sent false
28178 8171 Acked Write 9 Done Done Sent false
28179 8172 Acked Flush 0 Done Done Sent false
28180 8173 Acked Read 7 Done Done Sent false
28181 8174 Acked Read 7 Done Done Sent false
28182 8175 Acked Flush 0 Done Done Sent false
28183 8176 Acked Write 10 Done Done Sent false
28184 8177 Acked Flush 0 Done Done Sent false
28185 8178 Acked Flush 0 Done Done Sent false
28186 8179 Acked Write 3 Done Done Sent false
28187 8180 Acked Write 8 Done Done Sent false
28188 8181 Acked Write 2 Done Done Sent false
28189 8182 Acked Write 7 Done Done Sent false
28190 8183 Acked Read 5 Done Done Sent false
28191 8184 Acked Write 1 Done Done Sent false
28192 8185 Acked Write 5 Done Done Sent false
28193 8186 Acked Write 3 Done Done Sent false
28194 8187 Acked Read 10 Done Done Sent false
28195 8188 Acked Flush 0 Done Done Sent false
28196 8189 Acked Read 1 Done Done Sent false
28197 8190 Acked Read 3 Done Done Sent false
28198 8191 Acked Read 9 Done Done Sent false
28199 8192 Acked Read 1 Done Done Sent false
28200 8193 Acked Read 1 Done Done Sent false
28201 8194 Acked Flush 0 Done Done Sent false
28202 8195 Acked Read 8 Done Done Sent false
28203 8196 Acked Read 4 Done Done Sent false
28204 8197 Acked Write 9 Done Done Sent false
28205 8198 Acked Write 9 Done Done Sent false
28206 8199 Acked Write 3 Done Done Sent false
28207 8200 Acked Read 9 Done Done Sent false
28208 8201 Acked Write 9 Done Done Sent false
28209 8202 Acked Read 2 Done Done Sent false
28210 8203 Acked Read 4 Done Done Sent false
28211 8204 Acked Read 9 Done Done Sent false
28212 8205 Acked Write 9 Done Done Sent false
28213 8206 Acked Read 1 Done Done Sent false
28214 8207 Acked Write 5 Done Done Sent false
28215 8208 Acked Read 2 Done Done Sent false
28216 8209 Acked Read 2 Done Done Sent false
28217 8210 Acked Read 8 Done Done Sent false
28218 8211 Acked Read 4 Done Done Sent false
28219 8212 Acked Write 3 Done Done Sent false
28220 8213 Acked Read 8 Done Done Sent false
28221 8214 Acked Read 1 Done Done Sent false
28222 8215 Acked Flush 0 Done Done Sent false
28223 8216 Acked Read 4 Done Done Sent false
28224 8217 Acked Read 5 Done Done Sent false
28225 8218 Acked Read 4 Done Done Sent false
28226 8219 Acked Read 5 Done Done Sent false
28227 8220 Acked Write 10 Done Done Sent false
28228 8221 Acked Write 9 Done Done Sent false
28229 8222 Acked Write 3 Done Done Sent false
28230 8223 Acked Write 3 Done Done Sent false
28231 8224 Acked Write 9 Done Done Sent false
28232 8225 Acked Read 6 Done Done Sent false
28233 8226 Acked Read 5 Done Done Sent false
28234 8227 Acked Write 6 Done Done Sent false
28235 8228 Acked Write 8 Done Done Sent false
28236 8229 Acked Flush 0 Done Done Sent false
28237 8230 Acked Write 1 Done Done Sent false
28238 8231 Acked Write 5 Done Done Sent false
28239 8232 Acked Read 2 Done Done Sent false
28240 8233 Acked Write 5 Done Done Sent false
28241 8234 Acked Write 6 Done Done Sent false
28242 8235 Acked Flush 0 Done Done Sent false
28243 8236 Acked Write 8 Done Done Sent false
28244 8237 Acked Write 6 Done Done Sent false
28245 8238 Acked Read 7 Done Done Sent false
28246 8239 Acked Write 8 Done Done Sent false
28247 8240 Acked Read 4 Done Done Sent false
28248 8241 Acked Read 9 Done Done Sent false
28249 8242 Acked Read 9 Done Done Sent false
28250 8243 Acked Write 5 Done Done Sent false
28251 8244 Acked Write 6 Done Done Sent false
28252 STATES DS:0 DS:1 DS:2 TOTAL
28253 Sent 0 0 301 301
28254 Done 301 301 0 602
28255 Skipped 0 0 0 0
28256 Error 0 0 0 0
28257 Last Flush: Some(JobId(8235)) Some(JobId(8235)) Some(JobId(7943))
28258 Downstairs last five completed: 7943 7942 7941 7940 7939
28259 Upstairs last five completed: 8244 8243 8242 8241 8240
28260 CLIENT: Up:0 ds:301 act:2
282612025-03-17T22:22:34.191ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
282622025-03-17T22:22:35.075ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
282632025-03-17T22:22:35.075ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
282642025-03-17T22:22:35.075ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
282652025-03-17T22:22:35.075ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
282662025-03-17T22:22:35.075ZINFOcrucible: client task is exiting = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
28267 [8830] got start action
28268 [8830] I am going to restart
28269 Starting downstairs at port 8830
28270 Make output file at "/tmp/dsc/downstairs-8830.txt"
28271 [8830][2] reports Starting
28272 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1230)
28273 [8830][2] reports Running
282742025-03-17T22:22:35.576ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
282752025-03-17T22:22:36.324ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
28276 ----------------------------------------------------------------
28277 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
28278 States: Active Active Offline
28279 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28280 7944 Acked Write 6 Done Done Sent false
28281 7945 Acked Read 9 Done Done Sent false
28282 7946 Acked Read 9 Done Done Sent false
28283 7947 Acked Read 1 Done Done Sent false
28284 7948 Acked Read 6 Done Done Sent false
28285 7949 Acked Read 9 Done Done Sent false
28286 7950 Acked Write 4 Done Done Sent false
28287 7951 Acked Write 4 Done Done Sent false
28288 7952 Acked Read 5 Done Done Sent false
28289 7953 Acked Write 10 Done Done Sent false
28290 7954 Acked Write 3 Done Done Sent false
28291 7955 Acked Write 3 Done Done Sent false
28292 7956 Acked Read 9 Done Done Sent false
28293 7957 Acked Read 4 Done Done Sent false
28294 7958 Acked Flush 0 Done Done Sent false
28295 7959 Acked Write 1 Done Done Sent false
28296 7960 Acked Read 3 Done Done Sent false
28297 7961 Acked Read 8 Done Done Sent false
28298 7962 Acked Write 8 Done Done Sent false
28299 7963 Acked Write 3 Done Done Sent false
28300 7964 Acked Read 3 Done Done Sent false
28301 7965 Acked Write 7 Done Done Sent false
28302 7966 Acked Read 1 Done Done Sent false
28303 7967 Acked Flush 0 Done Done Sent false
28304 7968 Acked Write 1 Done Done Sent false
28305 7969 Acked Read 1 Done Done Sent false
28306 7970 Acked Read 9 Done Done Sent false
28307 7971 Acked Write 10 Done Done Sent false
28308 7972 Acked Write 2 Done Done Sent false
28309 7973 Acked Read 4 Done Done Sent false
28310 7974 Acked Read 4 Done Done Sent false
28311 7975 Acked Flush 0 Done Done Sent false
28312 7976 Acked Write 10 Done Done Sent false
28313 7977 Acked Read 8 Done Done Sent false
28314 7978 Acked Read 1 Done Done Sent false
28315 7979 Acked Flush 0 Done Done Sent false
28316 7980 Acked Write 7 Done Done Sent false
28317 7981 Acked Write 1 Done Done Sent false
28318 7982 Acked Read 5 Done Done Sent false
28319 7983 Acked Read 7 Done Done Sent false
28320 7984 Acked Flush 0 Done Done Sent false
28321 7985 Acked Read 5 Done Done Sent false
28322 7986 Acked Write 5 Done Done Sent false
28323 7987 Acked Write 3 Done Done Sent false
28324 7988 Acked Read 10 Done Done Sent false
28325 7989 Acked Read 2 Done Done Sent false
28326 7990 Acked Read 8 Done Done Sent false
28327 7991 Acked Flush 0 Done Done Sent false
28328 7992 Acked Write 1 Done Done Sent false
28329 7993 Acked Read 4 Done Done Sent false
28330 7994 Acked Read 8 Done Done Sent false
28331 7995 Acked Read 2 Done Done Sent false
28332 7996 Acked Write 1 Done Done Sent false
28333 7997 Acked Read 4 Done Done Sent false
28334 7998 Acked Read 5 Done Done Sent false
28335 7999 Acked Flush 0 Done Done Sent false
28336 8000 Acked Read 1 Done Done Sent false
28337 8001 Acked Read 5 Done Done Sent false
28338 8002 Acked Read 9 Done Done Sent false
28339 8003 Acked Write 7 Done Done Sent false
28340 8004 Acked Read 1 Done Done Sent false
28341 8005 Acked Read 1 Done Done Sent false
28342 8006 Acked Write 4 Done Done Sent false
28343 8007 Acked Write 6 Done Done Sent false
28344 8008 Acked Write 1 Done Done Sent false
28345 8009 Acked Read 1 Done Done Sent false
28346 8010 Acked Write 7 Done Done Sent false
28347 8011 Acked Write 10 Done Done Sent false
28348 8012 Acked Read 6 Done Done Sent false
28349 8013 Acked Read 6 Done Done Sent false
28350 8014 Acked Write 1 Done Done Sent false
28351 8015 Acked Read 5 Done Done Sent false
28352 8016 Acked Read 10 Done Done Sent false
28353 8017 Acked Write 7 Done Done Sent false
28354 8018 Acked Read 4 Done Done Sent false
28355 8019 Acked Read 1 Done Done Sent false
28356 8020 Acked Read 6 Done Done Sent false
28357 8021 Acked Write 8 Done Done Sent false
28358 8022 Acked Write 2 Done Done Sent false
28359 8023 Acked Read 2 Done Done Sent false
28360 8024 Acked Read 3 Done Done Sent false
28361 8025 Acked Write 6 Done Done Sent false
28362 8026 Acked Flush 0 Done Done Sent false
28363 8027 Acked Write 1 Done Done Sent false
28364 8028 Acked Write 7 Done Done Sent false
28365 8029 Acked Read 2 Done Done Sent false
28366 8030 Acked Write 3 Done Done Sent false
28367 8031 Acked Write 6 Done Done Sent false
28368 8032 Acked Write 4 Done Done Sent false
28369 8033 Acked Read 9 Done Done Sent false
28370 8034 Acked Write 7 Done Done Sent false
28371 8035 Acked Write 9 Done Done Sent false
28372 8036 Acked Flush 0 Done Done Sent false
28373 8037 Acked Read 9 Done Done Sent false
28374 8038 Acked Read 4 Done Done Sent false
28375 8039 Acked Read 6 Done Done Sent false
28376 8040 Acked Flush 0 Done Done Sent false
28377 8041 Acked Write 6 Done Done Sent false
28378 8042 Acked Read 7 Done Done Sent false
28379 8043 Acked Write 7 Done Done Sent false
28380 8044 Acked Flush 0 Done Done Sent false
28381 8045 Acked Read 2 Done Done Sent false
28382 8046 Acked Write 5 Done Done Sent false
28383 8047 Acked Read 7 Done Done Sent false
28384 8048 Acked Read 2 Done Done Sent false
28385 8049 Acked Read 7 Done Done Sent false
28386 8050 Acked Write 1 Done Done Sent false
28387 8051 Acked Read 3 Done Done Sent false
28388 8052 Acked Write 6 Done Done Sent false
28389 8053 Acked Read 5 Done Done Sent false
28390 8054 Acked Read 8 Done Done Sent false
28391 8055 Acked Write 1 Done Done Sent false
28392 8056 Acked Read 6 Done Done Sent false
28393 8057 Acked Flush 0 Done Done Sent false
28394 8058 Acked Read 2 Done Done Sent false
28395 8059 Acked Read 5 Done Done Sent false
28396 8060 Acked Write 5 Done Done Sent false
28397 8061 Acked Write 7 Done Done Sent false
28398 8062 Acked Write 3 Done Done Sent false
28399 8063 Acked Read 2 Done Done Sent false
28400 8064 Acked Write 5 Done Done Sent false
28401 8065 Acked Read 6 Done Done Sent false
28402 8066 Acked Write 7 Done Done Sent false
28403 8067 Acked Read 2 Done Done Sent false
28404 8068 Acked Write 2 Done Done Sent false
28405 8069 Acked Read 2 Done Done Sent false
28406 8070 Acked Flush 0 Done Done Sent false
28407 8071 Acked Flush 0 Done Done Sent false
28408 8072 Acked Read 3 Done Done Sent false
28409 8073 Acked Read 8 Done Done Sent false
28410 8074 Acked Flush 0 Done Done Sent false
28411 8075 Acked Read 3 Done Done Sent false
28412 8076 Acked Read 5 Done Done Sent false
28413 8077 Acked Flush 0 Done Done Sent false
28414 8078 Acked Write 5 Done Done Sent false
28415 8079 Acked Read 6 Done Done Sent false
28416 8080 Acked Read 3 Done Done Sent false
28417 8081 Acked Write 6 Done Done Sent false
28418 8082 Acked Write 2 Done Done Sent false
28419 8083 Acked Read 8 Done Done Sent false
28420 8084 Acked Write 7 Done Done Sent false
28421 8085 Acked Read 3 Done Done Sent false
28422 8086 Acked Write 10 Done Done Sent false
28423 8087 Acked Write 10 Done Done Sent false
28424 8088 Acked Flush 0 Done Done Sent false
28425 8089 Acked Read 9 Done Done Sent false
28426 8090 Acked Read 1 Done Done Sent false
28427 8091 Acked Read 6 Done Done Sent false
28428 8092 Acked Flush 0 Done Done Sent false
28429 8093 Acked Read 6 Done Done Sent false
28430 8094 Acked Write 7 Done Done Sent false
28431 8095 Acked Read 7 Done Done Sent false
28432 8096 Acked Write 7 Done Done Sent false
28433 8097 Acked Write 1 Done Done Sent false
28434 8098 Acked Read 6 Done Done Sent false
28435 8099 Acked Write 1 Done Done Sent false
28436 8100 Acked Flush 0 Done Done Sent false
28437 8101 Acked Read 7 Done Done Sent false
28438 8102 Acked Write 2 Done Done Sent false
28439 8103 Acked Write 10 Done Done Sent false
28440 8104 Acked Read 6 Done Done Sent false
28441 8105 Acked Read 7 Done Done Sent false
28442 8106 Acked Read 4 Done Done Sent false
28443 8107 Acked Write 3 Done Done Sent false
28444 8108 Acked Write 4 Done Done Sent false
28445 8109 Acked Read 5 Done Done Sent false
28446 8110 Acked Flush 0 Done Done Sent false
28447 8111 Acked Read 3 Done Done Sent false
28448 8112 Acked Read 10 Done Done Sent false
28449 8113 Acked Flush 0 Done Done Sent false
28450 8114 Acked Write 4 Done Done Sent false
28451 8115 Acked Read 9 Done Done Sent false
28452 8116 Acked Read 10 Done Done Sent false
28453 8117 Acked Read 9 Done Done Sent false
28454 8118 Acked Read 4 Done Done Sent false
28455 8119 Acked Write 8 Done Done Sent false
28456 8120 Acked Write 4 Done Done Sent false
28457 8121 Acked Write 7 Done Done Sent false
28458 8122 Acked Read 2 Done Done Sent false
28459 8123 Acked Read 3 Done Done Sent false
28460 8124 Acked Flush 0 Done Done Sent false
28461 8125 Acked Write 10 Done Done Sent false
28462 8126 Acked Read 4 Done Done Sent false
28463 8127 Acked Read 9 Done Done Sent false
28464 8128 Acked Read 5 Done Done Sent false
28465 8129 Acked Write 3 Done Done Sent false
28466 8130 Acked Write 2 Done Done Sent false
28467 8131 Acked Write 8 Done Done Sent false
28468 8132 Acked Read 8 Done Done Sent false
28469 8133 Acked Write 9 Done Done Sent false
28470 8134 Acked Flush 0 Done Done Sent false
28471 8135 Acked Write 6 Done Done Sent false
28472 8136 Acked Read 2 Done Done Sent false
28473 8137 Acked Write 5 Done Done Sent false
28474 8138 Acked Read 3 Done Done Sent false
28475 8139 Acked Read 5 Done Done Sent false
28476 8140 Acked Write 6 Done Done Sent false
28477 8141 Acked Write 5 Done Done Sent false
28478 8142 Acked Write 7 Done Done Sent false
28479 8143 Acked Read 6 Done Done Sent false
28480 8144 Acked Write 9 Done Done Sent false
28481 8145 Acked Read 3 Done Done Sent false
28482 8146 Acked Write 7 Done Done Sent false
28483 8147 Acked Write 7 Done Done Sent false
28484 8148 Acked Read 3 Done Done Sent false
28485 8149 Acked Read 4 Done Done Sent false
28486 8150 Acked Read 4 Done Done Sent false
28487 8151 Acked Flush 0 Done Done Sent false
28488 8152 Acked Read 7 Done Done Sent false
28489 8153 Acked Flush 0 Done Done Sent false
28490 8154 Acked Read 9 Done Done Sent false
28491 8155 Acked Write 4 Done Done Sent false
28492 8156 Acked Write 10 Done Done Sent false
28493 8157 Acked Write 2 Done Done Sent false
28494 8158 Acked Write 5 Done Done Sent false
28495 8159 Acked Read 4 Done Done Sent false
28496 8160 Acked Read 9 Done Done Sent false
28497 8161 Acked Read 10 Done Done Sent false
28498 8162 Acked Write 5 Done Done Sent false
28499 8163 Acked Write 4 Done Done Sent false
28500 8164 Acked Read 2 Done Done Sent false
28501 8165 Acked Write 1 Done Done Sent false
28502 8166 Acked Read 9 Done Done Sent false
28503 8167 Acked Write 8 Done Done Sent false
28504 8168 Acked Write 6 Done Done Sent false
28505 8169 Acked Write 6 Done Done Sent false
28506 8170 Acked Read 1 Done Done Sent false
28507 8171 Acked Write 9 Done Done Sent false
28508 8172 Acked Flush 0 Done Done Sent false
28509 8173 Acked Read 7 Done Done Sent false
28510 8174 Acked Read 7 Done Done Sent false
28511 8175 Acked Flush 0 Done Done Sent false
28512 8176 Acked Write 10 Done Done Sent false
28513 8177 Acked Flush 0 Done Done Sent false
28514 8178 Acked Flush 0 Done Done Sent false
28515 8179 Acked Write 3 Done Done Sent false
28516 8180 Acked Write 8 Done Done Sent false
28517 8181 Acked Write 2 Done Done Sent false
28518 8182 Acked Write 7 Done Done Sent false
28519 8183 Acked Read 5 Done Done Sent false
28520 8184 Acked Write 1 Done Done Sent false
28521 8185 Acked Write 5 Done Done Sent false
28522 8186 Acked Write 3 Done Done Sent false
28523 8187 Acked Read 10 Done Done Sent false
28524 8188 Acked Flush 0 Done Done Sent false
28525 8189 Acked Read 1 Done Done Sent false
28526 8190 Acked Read 3 Done Done Sent false
28527 8191 Acked Read 9 Done Done Sent false
28528 8192 Acked Read 1 Done Done Sent false
28529 8193 Acked Read 1 Done Done Sent false
28530 8194 Acked Flush 0 Done Done Sent false
28531 8195 Acked Read 8 Done Done Sent false
28532 8196 Acked Read 4 Done Done Sent false
28533 8197 Acked Write 9 Done Done Sent false
28534 8198 Acked Write 9 Done Done Sent false
28535 8199 Acked Write 3 Done Done Sent false
28536 8200 Acked Read 9 Done Done Sent false
28537 8201 Acked Write 9 Done Done Sent false
28538 8202 Acked Read 2 Done Done Sent false
28539 8203 Acked Read 4 Done Done Sent false
28540 8204 Acked Read 9 Done Done Sent false
28541 8205 Acked Write 9 Done Done Sent false
28542 8206 Acked Read 1 Done Done Sent false
28543 8207 Acked Write 5 Done Done Sent false
28544 8208 Acked Read 2 Done Done Sent false
28545 8209 Acked Read 2 Done Done Sent false
28546 8210 Acked Read 8 Done Done Sent false
28547 8211 Acked Read 4 Done Done Sent false
28548 8212 Acked Write 3 Done Done Sent false
28549 8213 Acked Read 8 Done Done Sent false
28550 8214 Acked Read 1 Done Done Sent false
28551 8215 Acked Flush 0 Done Done Sent false
28552 8216 Acked Read 4 Done Done Sent false
28553 8217 Acked Read 5 Done Done Sent false
28554 8218 Acked Read 4 Done Done Sent false
28555 8219 Acked Read 5 Done Done Sent false
28556 8220 Acked Write 10 Done Done Sent false
28557 8221 Acked Write 9 Done Done Sent false
28558 8222 Acked Write 3 Done Done Sent false
28559 8223 Acked Write 3 Done Done Sent false
28560 8224 Acked Write 9 Done Done Sent false
28561 8225 Acked Read 6 Done Done Sent false
28562 8226 Acked Read 5 Done Done Sent false
28563 8227 Acked Write 6 Done Done Sent false
28564 8228 Acked Write 8 Done Done Sent false
28565 8229 Acked Flush 0 Done Done Sent false
28566 8230 Acked Write 1 Done Done Sent false
28567 8231 Acked Write 5 Done Done Sent false
28568 8232 Acked Read 2 Done Done Sent false
28569 8233 Acked Write 5 Done Done Sent false
28570 8234 Acked Write 6 Done Done Sent false
28571 8235 Acked Flush 0 Done Done Sent false
28572 8236 Acked Write 8 Done Done Sent false
28573 8237 Acked Write 6 Done Done Sent false
28574 8238 Acked Read 7 Done Done Sent false
28575 8239 Acked Write 8 Done Done Sent false
28576 8240 Acked Read 4 Done Done Sent false
28577 8241 Acked Read 9 Done Done Sent false
28578 8242 Acked Read 9 Done Done Sent false
28579 8243 Acked Write 5 Done Done Sent false
28580 8244 Acked Write 6 Done Done Sent false
28581 STATES DS:0 DS:1 DS:2 TOTAL
28582 Sent 0 0 301 301
28583 Done 301 301 0 602
28584 Skipped 0 0 0 0
28585 Error 0 0 0 0
28586 Last Flush: Some(JobId(8235)) Some(JobId(8235)) Some(JobId(7943))
28587 Downstairs last five completed: 7943 7942 7941 7940 7939
28588 Upstairs last five completed: 8244 8243 8242 8241 8240
28589 CLIENT: Up:0 ds:301 act:2
285902025-03-17T22:22:38.198ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
285912025-03-17T22:22:39.199ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
285922025-03-17T22:22:40.199ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
28593 {"----------------------------------------------------------------
28594 msg" Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
28595 :States:" Activeflush check fired despite having jobs; resetting it Active" Offline
28596 ,"v"JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28597 :0,"name":"crucible 7944 Acked Write 6" Done Done," Sentlevel" false
28598 : 7945 Acked Read 940 Done Done Sent false
28599 7946 Acked Read 9 Done Done Sent false
28600 7947 Acked Read 1 Done Done Sent false
28601 7948 Acked Read 6 Done Done Sent false
28602 7949 Acked Read 9 Done Done Sent false
28603 7950 Acked Write 4 Done Done Sent false
28604 7951 Acked Write 4 Done Done Sent false
28605 7952 Acked Read 5 Done Done Sent false
28606 7953 Acked Write 10 Done Done Sent false
28607 7954 Acked Write 3 Done Done Sent false
28608 7955 Acked Write 3 Done Done Sent false
28609 7956 Acked Read 9, Done" Donetime Sent" false:
28610 " 7957 Acked Read 42025-03-17T22:22:41.325570187Z Done" Done Sent false
28611 ,"hostname" 7958 Acked Flush 0: Done" Done Sent false
28612 w-01JPK2S810SFAA9787XEG8GFX5" 7959 Acked Write 1, Done" Donepid Sent": false
28613 1206 7960 Acked Read 3 Done Done Sent false
28614 ," 7961 Acked Read 8session_id": Done" Done Sent false
28615 0f8310d7-099e-4ead-8484-91ddc162d307" 7962 Acked Write 8 Done} Done
28616 Sent false
28617 7963 Acked Write 3 Done Done Sent false
28618 7964 Acked Read 3 Done Done Sent false
28619 7965 Acked Write 7 Done Done Sent false
28620 7966 Acked Read 1 Done Done Sent false
28621 7967 Acked Flush 0 Done Done Sent false
28622 7968 Acked Write 1 Done Done Sent false
28623 7969 Acked Read 1 Done Done Sent false
28624 7970 Acked Read 9 Done Done Sent false
28625 7971 Acked Write 10 Done Done Sent false
28626 7972 Acked Write 2 Done Done Sent false
28627 7973 Acked Read 4 Done Done Sent false
28628 7974 Acked Read 4 Done Done Sent false
28629 7975 Acked Flush 0 Done Done Sent false
28630 7976 Acked Write 10 Done Done Sent false
28631 7977 Acked Read 8 Done Done Sent false
28632 7978 Acked Read 1 Done Done Sent false
28633 7979 Acked Flush 0 Done Done Sent false
28634 7980 Acked Write 7 Done Done Sent false
28635 7981 Acked Write 1 Done Done Sent false
28636 7982 Acked Read 5 Done Done Sent false
28637 7983 Acked Read 7 Done Done Sent false
28638 7984 Acked Flush 0 Done Done Sent false
28639 7985 Acked Read 5 Done Done Sent false
28640 7986 Acked Write 5 Done Done Sent false
28641 7987 Acked Write 3 Done Done Sent false
28642 7988 Acked Read 10 Done Done Sent false
28643 7989 Acked Read 2 Done Done Sent false
28644 7990 Acked Read 8 Done Done Sent false
28645 7991 Acked Flush 0 Done Done Sent false
28646 7992 Acked Write 1 Done Done Sent false
28647 7993 Acked Read 4 Done Done Sent false
28648 7994 Acked Read 8 Done Done Sent false
28649 7995 Acked Read 2 Done Done Sent false
28650 7996 Acked Write 1 Done Done Sent false
28651 7997 Acked Read 4 Done Done Sent false
28652 7998 Acked Read 5 Done Done Sent false
28653 7999 Acked Flush 0 Done Done Sent false
28654 8000 Acked Read 1 Done Done Sent false
28655 8001 Acked Read 5 Done Done Sent false
28656 8002 Acked Read 9 Done Done Sent false
28657 8003 Acked Write 7 Done Done Sent false
28658 8004 Acked Read 1 Done Done Sent false
28659 8005 Acked Read 1 Done Done Sent false
28660 8006 Acked Write 4 Done Done Sent false
28661 8007 Acked Write 6 Done Done Sent false
28662 8008 Acked Write 1 Done Done Sent false
28663 8009 Acked Read 1 Done Done Sent false
28664 8010 Acked Write 7 Done Done Sent false
28665 8011 Acked Write 10 Done Done Sent false
28666 8012 Acked Read 6 Done Done Sent false
28667 8013 Acked Read 6 Done Done Sent false
28668 8014 Acked Write 1 Done Done Sent false
28669 8015 Acked Read 5 Done Done Sent false
28670 8016 Acked Read 10 Done Done Sent false
28671 8017 Acked Write 7 Done Done Sent false
28672 8018 Acked Read 4 Done Done Sent false
28673 8019 Acked Read 1 Done Done Sent false
28674 8020 Acked Read 6 Done Done Sent false
28675 8021 Acked Write 8 Done Done Sent false
28676 8022 Acked Write 2 Done Done Sent false
28677 8023 Acked Read 2 Done Done Sent false
28678 8024 Acked Read 3 Done Done Sent false
28679 8025 Acked Write 6 Done Done Sent false
28680 8026 Acked Flush 0 Done Done Sent false
28681 8027 Acked Write 1 Done Done Sent false
28682 8028 Acked Write 7 Done Done Sent false
28683 8029 Acked Read 2 Done Done Sent false
28684 8030 Acked Write 3 Done Done Sent false
28685 8031 Acked Write 6 Done Done Sent false
28686 8032 Acked Write 4 Done Done Sent false
28687 8033 Acked Read 9 Done Done Sent false
28688 8034 Acked Write 7 Done Done Sent false
28689 8035 Acked Write 9 Done Done Sent false
28690 8036 Acked Flush 0 Done Done Sent false
28691 8037 Acked Read 9 Done Done Sent false
28692 8038 Acked Read 4 Done Done Sent false
28693 8039 Acked Read 6 Done Done Sent false
28694 8040 Acked Flush 0 Done Done Sent false
28695 8041 Acked Write 6 Done Done Sent false
28696 8042 Acked Read 7 Done Done Sent false
28697 8043 Acked Write 7 Done Done Sent false
28698 8044 Acked Flush 0 Done Done Sent false
28699 8045 Acked Read 2 Done Done Sent false
28700 8046 Acked Write 5 Done Done Sent false
28701 8047 Acked Read 7 Done Done Sent false
28702 8048 Acked Read 2 Done Done Sent false
28703 8049 Acked Read 7 Done Done Sent false
28704 8050 Acked Write 1 Done Done Sent false
28705 8051 Acked Read 3 Done Done Sent false
28706 8052 Acked Write 6 Done Done Sent false
28707 8053 Acked Read 5 Done Done Sent false
28708 8054 Acked Read 8 Done Done Sent false
28709 8055 Acked Write 1 Done Done Sent false
28710 8056 Acked Read 6 Done Done Sent false
28711 8057 Acked Flush 0 Done Done Sent false
28712 8058 Acked Read 2 Done Done Sent false
28713 8059 Acked Read 5 Done Done Sent false
28714 8060 Acked Write 5 Done Done Sent false
28715 8061 Acked Write 7 Done Done Sent false
28716 8062 Acked Write 3 Done Done Sent false
28717 8063 Acked Read 2 Done Done Sent false
28718 8064 Acked Write 5 Done Done Sent false
28719 8065 Acked Read 6 Done Done Sent false
28720 8066 Acked Write 7 Done Done Sent false
28721 8067 Acked Read 2 Done Done Sent false
28722 8068 Acked Write 2 Done Done Sent false
28723 8069 Acked Read 2 Done Done Sent false
28724 8070 Acked Flush 0 Done Done Sent false
28725 8071 Acked Flush 0 Done Done Sent false
28726 8072 Acked Read 3 Done Done Sent false
28727 8073 Acked Read 8 Done Done Sent false
28728 8074 Acked Flush 0 Done Done Sent false
28729 8075 Acked Read 3 Done Done Sent false
28730 8076 Acked Read 5 Done Done Sent false
28731 8077 Acked Flush 0 Done Done Sent false
28732 8078 Acked Write 5 Done Done Sent false
28733 8079 Acked Read 6 Done Done Sent false
28734 8080 Acked Read 3 Done Done Sent false
28735 8081 Acked Write 6 Done Done Sent false
28736 8082 Acked Write 2 Done Done Sent false
28737 8083 Acked Read 8 Done Done Sent false
28738 8084 Acked Write 7 Done Done Sent false
28739 8085 Acked Read 3 Done Done Sent false
28740 8086 Acked Write 10 Done Done Sent false
28741 8087 Acked Write 10 Done Done Sent false
28742 8088 Acked Flush 0 Done Done Sent false
28743 8089 Acked Read 9 Done Done Sent false
28744 8090 Acked Read 1 Done Done Sent false
28745 8091 Acked Read 6 Done Done Sent false
28746 8092 Acked Flush 0 Done Done Sent false
28747 8093 Acked Read 6 Done Done Sent false
28748 8094 Acked Write 7 Done Done Sent false
28749 8095 Acked Read 7 Done Done Sent false
28750 8096 Acked Write 7 Done Done Sent false
28751 8097 Acked Write 1 Done Done Sent false
28752 8098 Acked Read 6 Done Done Sent false
28753 8099 Acked Write 1 Done Done Sent false
28754 8100 Acked Flush 0 Done Done Sent false
28755 8101 Acked Read 7 Done Done Sent false
28756 8102 Acked Write 2 Done Done Sent false
28757 8103 Acked Write 10 Done Done Sent false
28758 8104 Acked Read 6 Done Done Sent false
28759 8105 Acked Read 7 Done Done Sent false
28760 8106 Acked Read 4 Done Done Sent false
28761 8107 Acked Write 3 Done Done Sent false
28762 8108 Acked Write 4 Done Done Sent false
28763 8109 Acked Read 5 Done Done Sent false
28764 8110 Acked Flush 0 Done Done Sent false
28765 8111 Acked Read 3 Done Done Sent false
28766 8112 Acked Read 10 Done Done Sent false
28767 8113 Acked Flush 0 Done Done Sent false
28768 8114 Acked Write 4 Done Done Sent false
28769 8115 Acked Read 9 Done Done Sent false
28770 8116 Acked Read 10 Done Done Sent false
28771 8117 Acked Read 9 Done Done Sent false
28772 8118 Acked Read 4 Done Done Sent false
28773 8119 Acked Write 8 Done Done Sent false
28774 8120 Acked Write 4 Done Done Sent false
28775 8121 Acked Write 7 Done Done Sent false
28776 8122 Acked Read 2 Done Done Sent false
28777 8123 Acked Read 3 Done Done Sent false
28778 8124 Acked Flush 0 Done Done Sent false
28779 8125 Acked Write 10 Done Done Sent false
28780 8126 Acked Read 4 Done Done Sent false
28781 8127 Acked Read 9 Done Done Sent false
28782 8128 Acked Read 5 Done Done Sent false
28783 8129 Acked Write 3 Done Done Sent false
28784 8130 Acked Write 2 Done Done Sent false
28785 8131 Acked Write 8 Done Done Sent false
28786 8132 Acked Read 8 Done Done Sent false
28787 8133 Acked Write 9 Done Done Sent false
28788 8134 Acked Flush 0 Done Done Sent false
28789 8135 Acked Write 6 Done Done Sent false
28790 8136 Acked Read 2 Done Done Sent false
28791 8137 Acked Write 5 Done Done Sent false
28792 8138 Acked Read 3 Done Done Sent false
28793 8139 Acked Read 5 Done Done Sent false
28794 8140 Acked Write 6 Done Done Sent false
28795 8141 Acked Write 5 Done Done Sent false
28796 8142 Acked Write 7 Done Done Sent false
28797 8143 Acked Read 6 Done Done Sent false
28798 8144 Acked Write 9 Done Done Sent false
28799 8145 Acked Read 3 Done Done Sent false
28800 8146 Acked Write 7 Done Done Sent false
28801 8147 Acked Write 7 Done Done Sent false
28802 8148 Acked Read 3 Done Done Sent false
28803 8149 Acked Read 4 Done Done Sent false
28804 8150 Acked Read 4 Done Done Sent false
28805 8151 Acked Flush 0 Done Done Sent false
28806 8152 Acked Read 7 Done Done Sent false
28807 8153 Acked Flush 0 Done Done Sent false
28808 8154 Acked Read 9 Done Done Sent false
28809 8155 Acked Write 4 Done Done Sent false
28810 8156 Acked Write 10 Done Done Sent false
28811 8157 Acked Write 2 Done Done Sent false
28812 8158 Acked Write 5 Done Done Sent false
28813 8159 Acked Read 4 Done Done Sent false
28814 8160 Acked Read 9 Done Done Sent false
28815 8161 Acked Read 10 Done Done Sent false
28816 8162 Acked Write 5 Done Done Sent false
28817 8163 Acked Write 4 Done Done Sent false
28818 8164 Acked Read 2 Done Done Sent false
28819 8165 Acked Write 1 Done Done Sent false
28820 8166 Acked Read 9 Done Done Sent false
28821 8167 Acked Write 8 Done Done Sent false
28822 8168 Acked Write 6 Done Done Sent false
28823 8169 Acked Write 6 Done Done Sent false
28824 8170 Acked Read 1 Done Done Sent false
28825 8171 Acked Write 9 Done Done Sent false
28826 8172 Acked Flush 0 Done Done Sent false
28827 8173 Acked Read 7 Done Done Sent false
28828 8174 Acked Read 7 Done Done Sent false
28829 8175 Acked Flush 0 Done Done Sent false
28830 8176 Acked Write 10 Done Done Sent false
28831 8177 Acked Flush 0 Done Done Sent false
28832 8178 Acked Flush 0 Done Done Sent false
28833 8179 Acked Write 3 Done Done Sent false
28834 8180 Acked Write 8 Done Done Sent false
28835 8181 Acked Write 2 Done Done Sent false
28836 8182 Acked Write 7 Done Done Sent false
28837 8183 Acked Read 5 Done Done Sent false
28838 8184 Acked Write 1 Done Done Sent false
28839 8185 Acked Write 5 Done Done Sent false
28840 8186 Acked Write 3 Done Done Sent false
28841 8187 Acked Read 10 Done Done Sent false
28842 8188 Acked Flush 0 Done Done Sent false
28843 8189 Acked Read 1 Done Done Sent false
28844 8190 Acked Read 3 Done Done Sent false
28845 8191 Acked Read 9 Done Done Sent false
28846 8192 Acked Read 1 Done Done Sent false
28847 8193 Acked Read 1 Done Done Sent false
28848 8194 Acked Flush 0 Done Done Sent false
28849 8195 Acked Read 8 Done Done Sent false
28850 8196 Acked Read 4 Done Done Sent false
28851 8197 Acked Write 9 Done Done Sent false
28852 8198 Acked Write 9 Done Done Sent false
28853 8199 Acked Write 3 Done Done Sent false
28854 8200 Acked Read 9 Done Done Sent false
28855 8201 Acked Write 9 Done Done Sent false
28856 8202 Acked Read 2 Done Done Sent false
28857 8203 Acked Read 4 Done Done Sent false
28858 8204 Acked Read 9 Done Done Sent false
28859 8205 Acked Write 9 Done Done Sent false
28860 8206 Acked Read 1 Done Done Sent false
28861 8207 Acked Write 5 Done Done Sent false
28862 8208 Acked Read 2 Done Done Sent false
28863 8209 Acked Read 2 Done Done Sent false
28864 8210 Acked Read 8 Done Done Sent false
28865 8211 Acked Read 4 Done Done Sent false
28866 8212 Acked Write 3 Done Done Sent false
28867 8213 Acked Read 8 Done Done Sent false
28868 8214 Acked Read 1 Done Done Sent false
28869 8215 Acked Flush 0 Done Done Sent false
28870 8216 Acked Read 4 Done Done Sent false
28871 8217 Acked Read 5 Done Done Sent false
28872 8218 Acked Read 4 Done Done Sent false
28873 8219 Acked Read 5 Done Done Sent false
28874 8220 Acked Write 10 Done Done Sent false
28875 8221 Acked Write 9 Done Done Sent false
28876 8222 Acked Write 3 Done Done Sent false
28877 8223 Acked Write 3 Done Done Sent false
28878 8224 Acked Write 9 Done Done Sent false
28879 8225 Acked Read 6 Done Done Sent false
28880 8226 Acked Read 5 Done Done Sent false
28881 8227 Acked Write 6 Done Done Sent false
28882 8228 Acked Write 8 Done Done Sent false
28883 8229 Acked Flush 0 Done Done Sent false
28884 8230 Acked Write 1 Done Done Sent false
28885 8231 Acked Write 5 Done Done Sent false
28886 8232 Acked Read 2 Done Done Sent false
28887 8233 Acked Write 5 Done Done Sent false
28888 8234 Acked Write 6 Done Done Sent false
28889 8235 Acked Flush 0 Done Done Sent false
28890 8236 Acked Write 8 Done Done Sent false
28891 8237 Acked Write 6 Done Done Sent false
28892 8238 Acked Read 7 Done Done Sent false
28893 8239 Acked Write 8 Done Done Sent false
28894 8240 Acked Read 4 Done Done Sent false
28895 8241 Acked Read 9 Done Done Sent false
28896 8242 Acked Read 9 Done Done Sent false
28897 8243 Acked Write 5 Done Done Sent false
28898 8244 Acked Write 6 Done Done Sent false
28899 STATES DS:0 DS:1 DS:2 TOTAL
28900 Sent 0 0 301 301
28901 Done 301 301 0 602
28902 Skipped 0 0 0 0
28903 Error 0 0 0 0
28904 Last Flush: Some(JobId(8235)) Some(JobId(8235)) Some(JobId(7943))
28905 Downstairs last five completed: 7943 7942 7941 7940 7939
28906 Upstairs last five completed: 8244 8243 8242 8241 8240
28907 CLIENT: Up:0 ds:301 act:2
289082025-03-17T22:22:42.565ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
289092025-03-17T22:22:43.205ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
289102025-03-17T22:22:44.207ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
289112025-03-17T22:22:45.077ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
289122025-03-17T22:22:45.077ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:51493) = io task client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
289132025-03-17T22:22:45.078ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
289142025-03-17T22:22:45.079ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
289152025-03-17T22:22:45.079ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
289162025-03-17T22:22:45.079ZINFOcrucible: send last flush ID to this DS: Some(JobId(7943)) = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
289172025-03-17T22:22:45.079ZINFOcrucible: Transition from Offline to Active = downstairs client = 2 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
289182025-03-17T22:22:45.079ZINFOcrucible: [2] client re-new 301 jobs since flush Some(JobId(7943)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
289192025-03-17T22:22:45.079ZINFOcrucible: [2] Replayed 301 jobs since flush: Some(JobId(7943)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
28920 ----------------------------------------------------------------
28921 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:9
28922 States: Active Active Active
28923 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
28924 8236 Acked Write 8 Done Done Done true
28925 8237 Acked Write 6 Done Done Done true
28926 8238 Acked Read 7 Done Done Done true
28927 8239 Acked Write 8 Done Done Done true
28928 8240 Acked Read 4 Done Done Done true
28929 8241 Acked Read 9 Done Done Done true
28930 8242 Acked Read 9 Done Done Done true
28931 8243 Acked Write 5 Done Done Done true
28932 8244 Acked Write 6 Done Done Done true
28933 STATES DS:0 DS:1 DS:2 TOTAL
28934 Sent 0 0 0 0
28935 Done 9 9 9 27
28936 Skipped 0 0 0 0
28937 Error 0 0 0 0
28938 Last Flush: Some(JobId(8235)) Some(JobId(8235)) Some(JobId(8235))
28939 Downstairs last five completed: 8235 8234 8233 8232 8231
28940 Upstairs last five completed: 8244 8243 8242 8241 8240
28941 CLIENT: Up:0 ds:9 act:3
28942 ----------------------------------------------------------------
28943 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
28944 Downstairs last five completed: 8245 8244 8243 8242 8241
28945 Upstairs last five completed: 8245 8244 8243 8242 8241
28946 CLIENT: Up:0 ds:0 act:3
28947 Replay: All jobs finished
28948 Main task has work to do, go find it
28949 got dsc Stop(1)
28950 stop 1
28951 Mar 17 22:22:49.330 INFO request completed, [8820] Got stop action so:false kr:false
28952 latency_us: 368, response_code: 204, uri: /stop/cid/1, method: POST, req_id: eea8e7e8-0355-4562-bbce-c5714e4d6dbf, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
289532025-03-17T22:22:49.331ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
289542025-03-17T22:22:49.331ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
289552025-03-17T22:22:49.331ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
289562025-03-17T22:22:49.331ZINFOcrucible: client task is exiting = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
28957 Mar 17 22:22:49.331 INFO request completed, latency_us: 308, response_code: 200, uri: /state/cid/1, method: GET, req_id: ce25c5a8-36a9-49b0-9eff-091c9bbe6418, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
28958 [8820] Exited with: ExitStatus(unix_wait_status(9))
28959 [8820][1] reports Exit
28960 Mar 17 22:22:53.333 INFO request completed, latency_us: 332, response_code: 200, uri: /state/cid/1, method: GET, req_id: ae46a121-0983-4624-8bfe-482d392ca503, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
28961 001/300 Read block 863 len 20480
28962 002/300 Read block 1083 len 36864
28963 003/300 Write block 1887 len 24576 data: 7 6 5 5 4 3
28964 004/300 Write block 859 len 20480 data: 6 6 6 5 4
28965 005/300 Read block 3053 len 12288
28966 006/300 Write block 2351 len 16384 data: 3 4 4 3
28967 007/300 Write block 498 len 16384 data: 8 5 5 5
28968 008/300 Read block 2672 len 16384
28969 009/300 Read block 4973 len 8192
28970 010/300 Read block 1800 len 40960
28971 011/300 Flush
28972 012/300 Read block 727 len 4096
28973 013/300 Read block 3607 len 28672
28974 014/300 Flush
28975 015/300 Flush
28976 016/300 Read block 1688 len 12288
28977 017/300 Write block 553 len 20480 data: 7 7 7 8 10
28978 018/300 Read block 796 len 4096
28979 019/300 Read block 2941 len 24576
28980 020/300 Read block 99 len 16384
28981 021/300 Write block 672 len 4096 data: 3
28982 022/300 Flush
28983 023/300 Write block 188 len 32768 data: 7 6 5 4 4 4 4 4
28984 024/300 Write block 1844 len 20480 data: 5 6 4 5 5
28985 025/300 Read block 871 len 12288
28986 026/300 Read block 4731 len 4096
28987 027/300 Write block 3493 len 24576 data: 7 7 8 7 6 5
28988 028/300 Read block 2720 len 8192
28989 029/300 Write block 4543 len 8192 data: 6 5
28990 030/300 Read block 2226 len 8192
28991 031/300 Write block 840 len 36864 data: 5 5 3 2 2 3 3 4 5
28992 032/300 Write block 3418 len 12288 data: 8 6 6
28993 033/300 Read block 683 len 20480
28994 034/300 Write block 3412 len 4096 data: 3
28995 035/300 Write block 2720 len 12288 data: 4 4 2
28996 036/300 Flush
28997 037/300 Read block 2738 len 8192
28998 038/300 Read block 694 len 32768
28999 039/300 Read block 477 len 28672
29000 040/300 Read block 4321 len 20480
29001 041/300 Flush
29002 042/300 Write block 2008 len 4096 data: 6
29003 043/300 Read block 1804 len 16384
29004 044/300 Read block 2192 len 8192
29005 045/300 Write block 1401 len 28672 data: 9 9 8 7 6 7 6
29006 046/300 Read block 103 len 40960
29007 047/300 Read block 2639 len 4096
29008 048/300 Read block 2786 len 40960
29009 049/300 Write block 3366 len 20480 data: 5 5 5 5 5
29010 050/300 Write block 2683 len 32768 data: 2 2 3 5 5 5 5 5
29011 051/300 Flush
29012 052/300 Read block 4484 len 24576
29013 053/300 Read block 2999 len 12288
29014 054/300 Read block 257 len 16384
29015 055/300 Read block 3188 len 8192
29016 056/300 Read block 3769 len 8192
29017 057/300 Read block 3894 len 16384
29018 058/300 Read block 1320 len 24576
29019 059/300 Read block 2185 len 36864
29020 060/300 Read block 4264 len 28672
29021 061/300 Write block 3200 len 20480 data: 4 4 5 5 5
29022 062/300 Write block 3479 len 16384 data: 2 2 2 2
29023 063/300 Write block 441 len 4096 data: 3
29024 064/300 Write block 4626 len 32768 data: 6 6 7 7 7 7 6 8
29025 065/300 Read block 1356 len 12288
29026 066/300 Read block 2127 len 24576
29027 067/300 Write block 3733 len 8192 data: 6 6
29028 068/300 Read block 14 len 36864
29029 069/300 Write block 453 len 20480 data: 3 3 2 2 3
29030 070/300 Flush
29031 071/300 Read block 2379 len 4096
29032 072/300 Write block 455 len 4096 data: 3
29033 073/300 Write block 2954 len 36864 data: 5 5 9 9 9 10 10 8 7
29034 074/300 Read block 653 len 24576
29035 075/300 Write block 4194 len 32768 data: 3 3 3 5 6 7 6 6
29036 076/300 Read block 2842 len 24576
29037 077/300 Write block 4226 len 32768 data: 2 3 5 5 5 5 5 6
29038 078/300 Read block 399 len 32768
29039 079/300 Read block 1761 len 32768
29040 080/300 Read block 1202 len 36864
29041 081/300 Read block 338 len 12288
29042 082/300 Read block 538 len 4096
29043 083/300 Write block 2852 len 24576 data: 7 5 4 5 4 4
29044 084/300 Read block 1052 len 32768
29045 085/300 Read block 3339 len 24576
29046 086/300 Write block 1313 len 40960 data: 7 5 4 3 3 3 3 4 5 4
29047 087/300 Flush
29048 088/300 Read block 467 len 4096
29049 089/300 Write block 917 len 24576 data: 4 3 3 3 3 4
29050 090/300 Write block 348 len 20480 data: 5 4 4 4 6
29051 091/300 Write block 3885 len 40960 data: 5 5 8 7 6 6 5 6 6 8
29052 092/300 Read block 4413 len 8192
29053 093/300 Read block 459 len 24576
29054 094/300 Write block 1287 len 8192 data: 6 7
29055 095/300 Read block 1358 len 20480
29056 096/300 Read block 3120 len 4096
29057 097/300 Read block 1060 len 16384
29058 098/300 Read block 2564 len 16384
29059 099/300 Read block 4331 len 8192
29060 100/300 Write block 1898 len 8192 data: 6 4
29061 101/300 Read block 3799 len 32768
29062 102/300 Read block 4660 len 4096
29063 103/300 Write block 3644 len 40960 data: 10 9 7 7 6 7 7 6 6 8
29064 104/300 Write block 1114 len 16384 data: 5 4 4 4
29065 105/300 Write block 2344 len 8192 data: 5 5
29066 106/300 Read block 4007 len 20480
29067 107/300 Read block 1808 len 8192
29068 108/300 Read block 1711 len 24576
29069 109/300 Flush
29070 110/300 Flush
29071 111/300 Read block 21 len 24576
29072 112/300 Write block 1094 len 20480 data: 6 6 6 6 7
29073 113/300 Write block 2919 len 16384 data: 6 6 7 7
29074 114/300 Write block 4731 len 16384 data: 6 7 6 6
29075 115/300 Flush
29076 116/300 Write block 1453 len 24576 data: 5 5 4 5 5 6
29077 117/300 Read block 1688 len 12288
29078 118/300 Read block 1489 len 8192
29079 119/300 Write block 3546 len 24576 data: 4 4 6 5 5 6
29080 120/300 Write block 2466 len 16384 data: 3 3 2 4
29081 121/300 Read block 1839 len 24576
29082 122/300 Read block 3735 len 8192
29083 123/300 Write block 4702 len 40960 data: 5 5 4 5 5 4 4 4 5 5
29084 124/300 Flush
29085 125/300 Read block 959 len 32768
29086 126/300 Write block 210 len 4096 data: 3
29087 127/300 Read block 3266 len 12288
29088 128/300 Write block 960 len 12288 data: 6 5 4
29089 129/300 Write block 2927 len 24576 data: 6 7 8 11 9 8
29090 130/300 Read block 3676 len 36864
29091 131/300 Read block 1900 len 28672
29092 132/300 Read block 3633 len 8192
29093 133/300 Read block 4265 len 20480
29094 134/300 Write block 4217 len 8192 data: 6 5
29095 135/300 Read block 3886 len 36864
29096 136/300 Write block 2684 len 20480 data: 3 4 6 6 6
29097 137/300 Write block 4585 len 40960 data: 2 2 2 2 2 2 2 2 3 3
29098 138/300 Write block 4218 len 36864 data: 6 4 4 2 3 3 2 2 3
29099 139/300 Write block 4565 len 12288 data: 3 3 3
29100 140/300 Read block 1405 len 28672
29101 141/300 Read block 4711 len 12288
29102 142/300 Flush
29103 143/300 Read block 2717 len 12288
29104 144/300 Flush
29105 145/300 Write block 4261 len 24576 data: 7 6 5 4 5 7
29106 146/300 Write block 502 len 36864 data: 8 7 8 8 8 7 6 6 5
29107 147/300 Write block 4733 len 8192 data: 7 7
29108 148/300 Read block 1791 len 16384
29109 149/300 Read block 2628 len 8192
29110 150/300 Flush
29111 151/300 Flush
29112 152/300 Write block 4162 len 28672 data: 8 7 6 4 7 6 7
29113 153/300 Read block 3786 len 20480
29114 154/300 Read block 3138 len 28672
29115 155/300 Read block 2972 len 12288
29116 156/300 Read block 1323 len 16384
29117 157/300 Read block 2123 len 40960
29118 158/300 Write block 1927 len 8192 data: 9 10
29119 159/300 Read block 1984 len 40960
29120 160/300 Read block 136 len 40960
29121 161/300 Read block 3618 len 40960
29122 162/300 Write block 1554 len 28672 data: 3 3 4 4 6 5 5
29123 163/300 Read block 1164 len 36864
29124 164/300 Write block 3855 len 12288 data: 3 3 3
29125 165/300 Write block 4407 len 24576 data: 7 7 8 6 5 4
29126 166/300 Flush
29127 167/300 Write block 3351 len 12288 data: 5 8 9
29128 168/300 Read block 404 len 12288
29129 169/300 Write block 791 len 12288 data: 4 4 4
29130 170/300 Read block 3550 len 36864
29131 171/300 Write block 642 len 8192 data: 5 5
29132 172/300 Write block 2394 len 36864 data: 5 5 4 3 3 2 2 2 2
29133 173/300 Write block 3006 len 8192 data: 8 8
29134 174/300 Read block 3492 len 24576
29135 175/300 Write block 3336 len 16384 data: 6 4 5 4
29136 176/300 Read block 2139 len 32768
29137 177/300 Write block 3505 len 36864 data: 3 3 5 5 5 6 4 6 6
29138 178/300 Read block 3322 len 16384
29139 179/300 Read block 2637 len 36864
29140 180/300 Flush
29141 181/300 Read block 3017 len 8192
29142 182/300 Read block 3440 len 24576
29143 183/300 Flush
29144 184/300 Read block 1956 len 40960
29145 185/300 Write block 1236 len 32768 data: 6 6 6 5 4 3 3 4
29146 186/300 Read block 4868 len 4096
29147 187/300 Flush
29148 188/300 Read block 68 len 12288
29149 189/300 Write block 3548 len 12288 data: 7 6 6
29150 190/300 Read block 4597 len 24576
29151 191/300 Write block 4863 len 32768 data: 11 11 11 10 10 6 5 7
29152 192/300 Write block 2619 len 28672 data: 6 7 7 6 6 6 6
29153 193/300 Write block 2290 len 4096 data: 3
29154 194/300 Write block 4331 len 28672 data: 8 8 7 6 7 6 10
29155 195/300 Write block 4644 len 16384 data: 5 4 3 5
29156 196/300 Write block 4728 len 28672 data: 7 6 7 7 8 8 8
29157 197/300 Write block 3136 len 12288 data: 7 6 5
29158 198/300 Read block 178 len 8192
29159 199/300 Read block 4972 len 16384
29160 200/300 Write block 4500 len 40960 data: 6 5 6 6 4 4 4 4 5 5
29161 201/300 Read block 313 len 36864
29162 202/300 Write block 2356 len 4096 data: 4
29163 203/300 Read block 4545 len 16384
29164 204/300 Write block 786 len 28672 data: 5 5 4 3 3 5 5
29165 205/300 Read block 2875 len 20480
29166 206/300 Write block 4060 len 24576 data: 9 9 9 7 7 5
29167 207/300 Write block 2739 len 24576 data: 3 4 4 4 4 4
29168 208/300 Write block 2713 len 32768 data: 2 3 3 4 4 4 4 5
29169 209/300 Read block 940 len 28672
29170 210/300 Read block 4735 len 32768
29171 211/300 Read block 4964 len 36864
29172 212/300 Read block 1830 len 32768
29173 213/300 Read block 1846 len 32768
29174 214/300 Read block 1056 len 16384
29175 215/300 Read block 2944 len 16384
29176 216/300 Read block 4685 len 32768
29177 217/300 Read block 406 len 16384
29178 218/300 Write block 3608 len 16384 data: 4 4 5 6
29179 219/300 Flush
29180 220/300 Read block 1134 len 32768
29181 221/300 Flush
29182 222/300 Write block 1225 len 16384 data: 9 8 7 8
29183 223/300 Read block 975 len 24576
29184 224/300 Read block 2382 len 40960
29185 225/300 Read block 2521 len 40960
29186 226/300 Read block 4959 len 24576
29187 227/300 Read block 1258 len 40960
29188 228/300 Write block 3806 len 16384 data: 6 5 4 3
29189 229/300 Read block 4609 len 4096
29190 230/300 Write block 1291 len 32768 data: 6 5 5 4 6 6 6 7
29191 231/300 Read block 1716 len 40960
29192 232/300 Read block 2680 len 20480
29193 233/300 Read block 4610 len 8192
29194 234/300 Write block 1765 len 36864 data: 3 4 4 4 5 5 5 6 5
29195 235/300 Read block 2677 len 32768
29196 236/300 Write block 2683 len 16384 data: 3 4 5 7
29197 237/300 Read block 3499 len 40960
29198 238/300 Read block 497 len 36864
29199 239/300 Read block 2227 len 8192
29200 240/300 Read block 2760 len 24576
29201 241/300 Write block 1138 len 40960 data: 5 5 6 7 6 5 5 5 5 5
29202 242/300 Write block 463 len 20480 data: 5 7 8 7 6
29203 243/300 Flush
29204 244/300 Write block 4337 len 4096 data: 11
29205 245/300 Read block 2985 len 28672
29206 246/300 Read block 930 len 36864
29207 247/300 Flush
29208 248/300 Read block 3729 len 20480
29209 249/300 Read block 672 len 8192
29210 250/300 Write block 2859 len 28672 data: 3 2 3 5 7 8 8
29211 251/300 Write block 1807 len 40960 data: 6 6 5 4 3 3 3 3 4 4
29212 252/300 Read block 4402 len 36864
29213 253/300 Read block 3345 len 20480
29214 254/300 Write block 586 len 24576 data: 4 5 6 8 9 8
29215 255/300 Read block 1980 len 24576
29216 256/300 Read block 2160 len 28672
29217 257/300 Write block 3045 len 32768 data: 5 5 6 5 4 5 4 5
29218 258/300 Read block 4447 len 28672
29219 259/300 Write block 310 len 40960 data: 5 5 4 5 6 6 8 8 9 9
29220 260/300 Write block 1018 len 40960 data: 5 4 5 4 3 3 3 3 4 5
29221 261/300 Write block 2412 len 40960 data: 5 5 4 5 5 7 5 7 6 7
29222 262/300 Read block 4692 len 20480
29223 263/300 Flush
29224 264/300 Write block 4687 len 12288 data: 9 9 8
29225 265/300 Read block 4521 len 20480
29226 266/300 Write block 4060 len 28672 data: 10 10 10 8 8 6 5
29227 267/300 Read block 942 len 24576
29228 268/300 Write block 2474 len 40960 data: 5 5 8 7 7 7 5 4 5 6
29229 269/300 Write block 4669 len 36864 data: 5 4 3 3 4 4 4 5 3
29230 270/300 Read block 2245 len 24576
29231 271/300 Write block 1204 len 8192 data: 4 3
29232 272/300 Write block 2852 len 24576 data: 8 6 5 6 5 5
29233 273/300 Write block 3415 len 40960 data: 4 6 7 9 7 7 5 7 6 6
29234 274/300 Read block 2705 len 24576
29235 275/300 Flush
29236 276/300 Write block 1826 len 4096 data: 5
29237 277/300 Read block 2271 len 32768
29238 278/300 Read block 538 len 24576
29239 279/300 Write block 2784 len 8192 data: 8 8
29240 280/300 Write block 1827 len 4096 data: 6
29241 281/300 Flush
29242 282/300 Write block 713 len 40960 data: 3 5 6 4 3 3 4 5 7 6
29243 283/300 Flush
29244 284/300 Read block 87 len 20480
29245 285/300 Write block 3521 len 4096 data: 2
29246 286/300 Write block 936 len 24576 data: 4 4 4 3 4 4
29247 287/300 Write block 887 len 20480 data: 5 7 7 7 6
29248 288/300 Write block 2148 len 8192 data: 7 7
29249 289/300 Read block 867 len 12288
29250 290/300 Read block 1369 len 28672
29251 291/300 Read block 2627 len 28672
29252 292/300 Read block 3455 len 40960
29253 293/300 Write block 2872 len 16384 data: 6 6 5 5
29254 294/300 Flush
29255 295/300 Read block 3694 len 28672
29256 296/300 Read block 738 len 20480
29257 297/300 Read block 3035 len 28672
29258 298/300 Write block 4329 len 24576 data: 5 7 9 9 8 7
29259 299/300 Write block 3066 len 40960 data: 6 6 6 5 5 6 5 6 5 4
29260 300/300 Read block 4949 len 16384
29261 301/300 Write block 4616 len 12288 data: 6 6 4
29262 Main task has work to do, go find it
29263 got dsc Start(1)
29264 start 1
29265 Mar 17 22:22:53.584 INFO request completed, latency_us: 247, response_code: 204, uri: /start/cid/1, method: POST, req_id: bafdbcbf-ad69-422a-9bc4-fccd5f48e570, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
29266 Replay: started 1, returned:Ok(())
29267 ----------------------------------------------------------------
29268 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29269 States: Active Offline Active
29270 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29271 8246 Acked Read 5 Done Sent Done false
29272 8247 Acked Read 9 Done Sent Done false
29273 8248 Acked Write 6 Done Sent Done false
29274 8249 Acked Write 5 Done Sent Done false
29275 8250 Acked Read 3 Done Sent Done false
29276 8251 Acked Write 4 Done Sent Done false
29277 8252 Acked Write 4 Done Sent Done false
29278 8253 Acked Read 4 Done Sent Done false
29279 8254 Acked Read 2 Done Sent Done false
29280 8255 Acked Read 10 Done Sent Done false
29281 8256 Acked Flush 0 Done Sent Done false
29282 8257 Acked Read 1 Done Sent Done false
29283 8258 Acked Read 7 Done Sent Done false
29284 8259 Acked Flush 0 Done Sent Done false
29285 8260 Acked Flush 0 Done Sent Done false
29286 8261 Acked Read 3 Done Sent Done false
29287 8262 Acked Write 5 Done Sent Done false
29288 8263 Acked Read 1 Done Sent Done false
29289 8264 Acked Read 6 Done Sent Done false
29290 8265 Acked Read 4 Done Sent Done false
29291 8266 Acked Write 1 Done Sent Done false
29292 8267 Acked Flush 0 Done Sent Done false
29293 8268 Acked Write 8 Done Sent Done false
29294 8269 Acked Write 5 Done Sent Done false
29295 8270 Acked Read 3 Done Sent Done false
29296 8271 Acked Read 1 Done Sent Done false
29297 8272 Acked Write 6 Done Sent Done false
29298 8273 Acked Read 2 Done Sent Done false
29299 8274 Acked Write 2 Done Sent Done false
29300 8275 Acked Read 2 Done Sent Done false
29301 8276 Acked Write 9 Done Sent Done false
29302 8277 Acked Write 3 Done Sent Done false
29303 8278 Acked Read 5 Done Sent Done false
29304 8279 Acked Write 1 Done Sent Done false
29305 8280 Acked Write 3 Done Sent Done false
29306 8281 Acked Flush 0 Done Sent Done false
29307 8282 Acked Read 2 Done Sent Done false
29308 8283 Acked Read 8 Done Sent Done false
29309 8284 Acked Read 7 Done Sent Done false
29310 8285 Acked Read 5 Done Sent Done false
29311 8286 Acked Flush 0 Done Sent Done false
29312 8287 Acked Write 1 Done Sent Done false
29313 8288 Acked Read 4 Done Sent Done false
29314 8289 Acked Read 2 Done Sent Done false
29315 8290 Acked Write 7 Done Sent Done false
29316 8291 Acked Read 10 Done Sent Done false
29317 8292 Acked Read 1 Done Sent Done false
29318 8293 Acked Read 10 Done Sent Done false
29319 8294 Acked Write 5 Done Sent Done false
29320 8295 Acked Write 8 Done Sent Done false
29321 8296 Acked Flush 0 Done Sent Done false
29322 8297 Acked Read 6 Done Sent Done false
29323 8298 Acked Read 3 Done Sent Done false
29324 8299 Acked Read 4 Done Sent Done false
29325 8300 Acked Read 2 Done Sent Done false
29326 8301 Acked Read 2 Done Sent Done false
29327 8302 Acked Read 4 Done Sent Done false
29328 8303 Acked Read 6 Done Sent Done false
29329 8304 Acked Read 9 Done Sent Done false
29330 8305 Acked Read 7 Done Sent Done false
29331 8306 Acked Write 5 Done Sent Done false
29332 8307 Acked Write 4 Done Sent Done false
29333 8308 Acked Write 1 Done Sent Done false
29334 8309 Acked Write 8 Done Sent Done false
29335 8310 Acked Read 3 Done Sent Done false
29336 8311 Acked Read 6 Done Sent Done false
29337 8312 Acked Write 2 Done Sent Done false
29338 8313 Acked Read 9 Done Sent Done false
29339 8314 Acked Write 5 Done Sent Done false
29340 8315 Acked Flush 0 Done Sent Done false
29341 8316 Acked Read 1 Done Sent Done false
29342 8317 Acked Write 1 Done Sent Done false
29343 8318 Acked Write 9 Done Sent Done false
29344 8319 Acked Read 6 Done Sent Done false
29345 8320 Acked Write 8 Done Sent Done false
29346 8321 Acked Read 6 Done Sent Done false
29347 8322 Acked Write 8 Done Sent Done false
29348 8323 Acked Read 8 Done Sent Done false
29349 8324 Acked Read 8 Done Sent Done false
29350 8325 Acked Read 9 Done Sent Done false
29351 8326 Acked Read 3 Done Sent Done false
29352 8327 Acked Read 1 Done Sent Done false
29353 8328 Acked Write 6 Done Sent Done false
29354 8329 Acked Read 8 Done Sent Done false
29355 8330 Acked Read 6 Done Sent Done false
29356 8331 Acked Write 10 Done Sent Done false
29357 8332 Acked Flush 0 Done Sent Done false
29358 8333 Acked Read 1 Done Sent Done false
29359 8334 Acked Write 6 Done Sent Done false
29360 8335 Acked Write 5 Done Sent Done false
29361 8336 Acked Write 10 Done Sent Done false
29362 8337 Acked Read 2 Done Sent Done false
29363 8338 Acked Read 6 Done Sent Done false
29364 8339 Acked Write 2 Done Sent Done false
29365 8340 Acked Read 5 Done Sent Done false
29366 8341 Acked Read 1 Done Sent Done false
29367 8342 Acked Read 4 Done Sent Done false
29368 8343 Acked Read 4 Done Sent Done false
29369 8344 Acked Read 2 Done Sent Done false
29370 8345 Acked Write 2 Done Sent Done false
29371 8346 Acked Read 8 Done Sent Done false
29372 8347 Acked Read 1 Done Sent Done false
29373 8348 Acked Write 10 Done Sent Done false
29374 8349 Acked Write 4 Done Sent Done false
29375 8350 Acked Write 2 Done Sent Done false
29376 8351 Acked Read 5 Done Sent Done false
29377 8352 Acked Read 2 Done Sent Done false
29378 8353 Acked Read 6 Done Sent Done false
29379 8354 Acked Flush 0 Done Sent Done false
29380 8355 Acked Flush 0 Done Sent Done false
29381 8356 Acked Read 6 Done Sent Done false
29382 8357 Acked Write 5 Done Sent Done false
29383 8358 Acked Write 4 Done Sent Done false
29384 8359 Acked Write 4 Done Sent Done false
29385 8360 Acked Flush 0 Done Sent Done false
29386 8361 Acked Write 6 Done Sent Done false
29387 8362 Acked Read 3 Done Sent Done false
29388 8363 Acked Read 2 Done Sent Done false
29389 8364 Acked Write 6 Done Sent Done false
29390 8365 Acked Write 4 Done Sent Done false
29391 8366 Acked Read 6 Done Sent Done false
29392 8367 Acked Read 2 Done Sent Done false
29393 8368 Acked Write 10 Done Sent Done false
29394 8369 Acked Flush 0 Done Sent Done false
29395 8370 Acked Read 8 Done Sent Done false
29396 8371 Acked Write 1 Done Sent Done false
29397 8372 Acked Read 3 Done Sent Done false
29398 8373 Acked Write 3 Done Sent Done false
29399 8374 Acked Write 6 Done Sent Done false
29400 8375 Acked Read 9 Done Sent Done false
29401 8376 Acked Read 7 Done Sent Done false
29402 8377 Acked Read 2 Done Sent Done false
29403 8378 Acked Read 5 Done Sent Done false
29404 8379 Acked Write 2 Done Sent Done false
29405 8380 Acked Read 9 Done Sent Done false
29406 8381 Acked Write 5 Done Sent Done false
29407 8382 Acked Write 10 Done Sent Done false
29408 8383 Acked Write 9 Done Sent Done false
29409 8384 Acked Write 3 Done Sent Done false
29410 8385 Acked Read 7 Done Sent Done false
29411 8386 Acked Read 3 Done Sent Done false
29412 8387 Acked Flush 0 Done Sent Done false
29413 8388 Acked Read 3 Done Sent Done false
29414 8389 Acked Flush 0 Done Sent Done false
29415 8390 Acked Write 6 Done Sent Done false
29416 8391 Acked Write 9 Done Sent Done false
29417 8392 Acked Write 2 Done Sent Done false
29418 8393 Acked Read 4 Done Sent Done false
29419 8394 Acked Read 2 Done Sent Done false
29420 8395 Acked Flush 0 Done Sent Done false
29421 8396 Acked Flush 0 Done Sent Done false
29422 8397 Acked Write 7 Done Sent Done false
29423 8398 Acked Read 5 Done Sent Done false
29424 8399 Acked Read 7 Done Sent Done false
29425 8400 Acked Read 3 Done Sent Done false
29426 8401 Acked Read 4 Done Sent Done false
29427 8402 Acked Read 10 Done Sent Done false
29428 8403 Acked Write 2 Done Sent Done false
29429 8404 Acked Read 10 Done Sent Done false
29430 8405 Acked Read 10 Done Sent Done false
29431 8406 Acked Read 10 Done Sent Done false
29432 8407 Acked Write 7 Done Sent Done false
29433 8408 Acked Read 9 Done Sent Done false
29434 8409 Acked Write 3 Done Sent Done false
29435 8410 Acked Write 6 Done Sent Done false
29436 8411 Acked Flush 0 Done Sent Done false
29437 8412 Acked Write 3 Done Sent Done false
29438 8413 Acked Read 3 Done Sent Done false
29439 8414 Acked Write 3 Done Sent Done false
29440 8415 Acked Read 9 Done Sent Done false
29441 8416 Acked Write 2 Done Sent Done false
29442 8417 Acked Write 9 Done Sent Done false
29443 8418 Acked Write 2 Done Sent Done false
29444 8419 Acked Read 6 Done Sent Done false
29445 8420 Acked Write 4 Done Sent Done false
29446 8421 Acked Read 8 Done Sent Done false
29447 8422 Acked Write 9 Done Sent Done false
29448 8423 Acked Read 4 Done Sent Done false
29449 8424 Acked Read 9 Done Sent Done false
29450 8425 Acked Flush 0 Done Sent Done false
29451 8426 Acked Read 2 Done Sent Done false
29452 8427 Acked Read 6 Done Sent Done false
29453 8428 Acked Flush 0 Done Sent Done false
29454 8429 Acked Read 10 Done Sent Done false
29455 8430 Acked Write 8 Done Sent Done false
29456 8431 Acked Read 1 Done Sent Done false
29457 8432 Acked Flush 0 Done Sent Done false
29458 8433 Acked Read 3 Done Sent Done false
29459 8434 Acked Write 3 Done Sent Done false
29460 8435 Acked Read 6 Done Sent Done false
29461 8436 Acked Write 8 Done Sent Done false
29462 8437 Acked Write 7 Done Sent Done false
29463 8438 Acked Write 1 Done Sent Done false
29464 8439 Acked Write 7 Done Sent Done false
29465 8440 Acked Write 4 Done Sent Done false
29466 8441 Acked Write 7 Done Sent Done false
29467 8442 Acked Write 3 Done Sent Done false
29468 8443 Acked Read 2 Done Sent Done false
29469 8444 Acked Read 4 Done Sent Done false
29470 8445 Acked Write 10 Done Sent Done false
29471 8446 Acked Read 9 Done Sent Done false
29472 8447 Acked Write 1 Done Sent Done false
29473 8448 Acked Read 4 Done Sent Done false
29474 8449 Acked Write 7 Done Sent Done false
29475 8450 Acked Read 5 Done Sent Done false
29476 8451 Acked Write 6 Done Sent Done false
29477 8452 Acked Write 6 Done Sent Done false
29478 8453 Acked Write 8 Done Sent Done false
29479 8454 Acked Read 7 Done Sent Done false
29480 8455 Acked Read 8 Done Sent Done false
29481 8456 Acked Read 9 Done Sent Done false
29482 8457 Acked Read 8 Done Sent Done false
29483 8458 Acked Read 8 Done Sent Done false
29484 8459 Acked Read 4 Done Sent Done false
29485 8460 Acked Read 4 Done Sent Done false
29486 8461 Acked Read 8 Done Sent Done false
29487 8462 Acked Read 4 Done Sent Done false
29488 8463 Acked Write 4 Done Sent Done false
29489 8464 Acked Flush 0 Done Sent Done false
29490 8465 Acked Read 8 Done Sent Done false
29491 8466 Acked Flush 0 Done Sent Done false
29492 8467 Acked Write 4 Done Sent Done false
29493 8468 Acked Read 6 Done Sent Done false
29494 8469 Acked Read 10 Done Sent Done false
29495 8470 Acked Read 10 Done Sent Done false
29496 8471 Acked Read 6 Done Sent Done false
29497 8472 Acked Read 10 Done Sent Done false
29498 8473 Acked Write 4 Done Sent Done false
29499 8474 Acked Read 1 Done Sent Done false
29500 8475 Acked Write 8 Done Sent Done false
29501 8476 Acked Read 10 Done Sent Done false
29502 8477 Acked Read 5 Done Sent Done false
29503 8478 Acked Read 2 Done Sent Done false
29504 8479 Acked Write 9 Done Sent Done false
29505 8480 Acked Read 8 Done Sent Done false
29506 8481 Acked Write 4 Done Sent Done false
29507 8482 Acked Read 10 Done Sent Done false
29508 8483 Acked Read 9 Done Sent Done false
29509 8484 Acked Read 2 Done Sent Done false
29510 8485 Acked Read 6 Done Sent Done false
29511 8486 Acked Write 10 Done Sent Done false
29512 8487 Acked Write 5 Done Sent Done false
29513 8488 Acked Flush 0 Done Sent Done false
29514 8489 Acked Write 1 Done Sent Done false
29515 8490 Acked Read 7 Done Sent Done false
29516 8491 Acked Read 9 Done Sent Done false
29517 8492 Acked Flush 0 Done Sent Done false
29518 8493 Acked Read 5 Done Sent Done false
29519 8494 Acked Read 2 Done Sent Done false
29520 8495 Acked Write 7 Done Sent Done false
29521 8496 Acked Write 10 Done Sent Done false
29522 8497 Acked Read 9 Done Sent Done false
29523 8498 Acked Read 5 Done Sent Done false
29524 8499 Acked Write 6 Done Sent Done false
29525 8500 Acked Read 6 Done Sent Done false
29526 8501 Acked Read 7 Done Sent Done false
29527 8502 Acked Write 8 Done Sent Done false
29528 8503 Acked Read 7 Done Sent Done false
29529 8504 Acked Write 10 Done Sent Done false
29530 8505 Acked Write 10 Done Sent Done false
29531 8506 Acked Write 10 Done Sent Done false
29532 8507 Acked Read 5 Done Sent Done false
29533 8508 Acked Flush 0 Done Sent Done false
29534 8509 Acked Write 3 Done Sent Done false
29535 8510 Acked Read 5 Done Sent Done false
29536 8511 Acked Write 7 Done Sent Done false
29537 8512 Acked Read 6 Done Sent Done false
29538 8513 Acked Write 10 Done Sent Done false
29539 8514 Acked Write 9 Done Sent Done false
29540 8515 Acked Read 6 Done Sent Done false
29541 8516 Acked Write 2 Done Sent Done false
29542 8517 Acked Write 6 Done Sent Done false
29543 8518 Acked Write 10 Done Sent Done false
29544 8519 Acked Read 6 Done Sent Done false
29545 8520 Acked Flush 0 Done Sent Done false
29546 8521 Acked Write 1 Done Sent Done false
29547 8522 Acked Read 8 Done Sent Done false
29548 8523 Acked Read 6 Done Sent Done false
29549 8524 Acked Write 2 Done Sent Done false
29550 8525 Acked Write 1 Done Sent Done false
29551 8526 Acked Flush 0 Done Sent Done false
29552 8527 Acked Write 10 Done Sent Done false
29553 8528 Acked Flush 0 Done Sent Done false
29554 8529 Acked Read 5 Done Sent Done false
29555 8530 Acked Write 1 Done Sent Done false
29556 8531 Acked Write 6 Done Sent Done false
29557 8532 Acked Write 5 Done Sent Done false
29558 8533 Acked Write 2 Done Sent Done false
29559 8534 Acked Read 3 Done Sent Done false
29560 8535 Acked Read 7 Done Sent Done false
29561 8536 Acked Read 7 Done Sent Done false
29562 8537 Acked Read 10 Done Sent Done false
29563 8538 Acked Write 4 Done Sent Done false
29564 8539 Acked Flush 0 Done Sent Done false
29565 8540 Acked Read 7 Done Sent Done false
29566 8541 Acked Read 5 Done Sent Done false
29567 8542 Acked Read 7 Done Sent Done false
29568 8543 Acked Write 6 Done Sent Done false
29569 8544 Acked Write 10 Done Sent Done false
29570 8545 Acked Read 4 Done Sent Done false
29571 8546 Acked Write 3 Done Sent Done false
29572 STATES DS:0 DS:1 DS:2 TOTAL
29573 Sent 0 301 0 301
29574 Done 301 0 301 602
29575 Skipped 0 0 0 0
29576 Error 0 0 0 0
29577 Last Flush: Some(JobId(8539)) Some(JobId(8245)) Some(JobId(8539))
29578 Downstairs last five completed: 8245 8244 8243 8242 8241
29579 Upstairs last five completed: 8546 8545 8544 8543 8542
29580 CLIENT: Up:0 ds:301 act:2
295812025-03-17T22:22:54.222ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
29582 [8820] got start action
29583 [8820] I am going to restart
29584 Starting downstairs at port 8820
29585 Make output file at "/tmp/dsc/downstairs-8820.txt"
29586 [8820][1] reports Starting
29587 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1231)
29588 [8820][1] reports Running
295892025-03-17T22:22:55.582ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
295902025-03-17T22:22:56.328ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
295912025-03-17T22:22:57.226ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
29592 ----------------------------------------------------------------
29593 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
29594 States: Active Offline Active
29595 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
29596 8246 Acked Read 5 Done Sent Done false
29597 8247 Acked Read 9 Done Sent Done false
29598 8248 Acked Write 6 Done Sent Done false
29599 8249 Acked Write 5 Done Sent Done false
29600 8250 Acked Read 3 Done Sent Done false
29601 8251 Acked Write 4 Done Sent Done false
29602 8252 Acked Write 4 Done Sent Done false
29603 8253 Acked Read 4 Done Sent Done false
29604 8254 Acked Read 2 Done Sent Done false
29605 8255 Acked Read 10 Done Sent Done false
29606 8256 Acked Flush 0 Done Sent Done false
29607 8257 Acked Read 1 Done Sent Done false
29608 8258 Acked Read 7 Done Sent Done false
29609 8259 Acked Flush 0 Done Sent Done false
29610 8260 Acked Flush 0 Done Sent Done false
29611 8261 Acked Read 3 Done Sent Done false
29612 8262 Acked Write 5 Done Sent Done false
29613 8263 Acked Read 1 Done Sent Done false
29614 8264 Acked Read 6 Done Sent Done false
29615 8265 Acked Read 4 Done Sent Done false
29616 8266 Acked Write 1 Done Sent Done false
29617 8267 Acked Flush 0 Done Sent Done false
29618 8268 Acked Write 8 Done Sent Done false
29619 8269 Acked Write 5 Done Sent Done false
29620 8270 Acked Read 3 Done Sent Done false
29621 8271 Acked Read 1 Done Sent Done false
29622 8272 Acked Write 6 Done Sent Done false
29623 8273 Acked Read 2 Done Sent Done false
29624 8274 Acked Write 2 Done Sent Done false
29625 8275 Acked Read 2 Done Sent Done false
29626 8276 Acked Write 9 Done Sent Done false
29627 8277 Acked Write 3 Done Sent Done false
29628 8278 Acked Read 5 Done Sent Done false
29629 8279 Acked Write 1 Done Sent Done false
29630 8280 Acked Write 3 Done Sent Done false
29631 8281 Acked Flush 0 Done Sent Done false
29632 8282 Acked Read 2 Done Sent Done false
29633 8283 Acked Read 8 Done Sent Done false
29634 8284 Acked Read 7 Done Sent Done false
29635 8285 Acked Read 5 Done Sent Done false
29636 8286 Acked Flush 0 Done Sent Done false
29637 8287 Acked Write 1 Done Sent Done false
29638 8288 Acked Read 4 Done Sent Done false
29639 8289 Acked Read 2 Done Sent Done false
29640 8290 Acked Write 7 Done Sent Done false
29641 8291 Acked Read 10 Done Sent Done false
29642 8292 Acked Read 1 Done Sent Done false
29643 8293 Acked Read 10 Done Sent Done false
29644 8294 Acked Write 5 Done Sent Done false
29645 8295 Acked Write 8 Done Sent Done false
29646 8296 Acked Flush 0 Done Sent Done false
29647 8297 Acked Read 6 Done Sent Done false
29648 8298 Acked Read 3 Done Sent Done false
29649 8299 Acked Read 4 Done Sent Done false
29650 8300 Acked Read 2 Done Sent Done false
29651 8301 Acked Read 2 Done Sent Done false
29652 8302 Acked Read 4 Done Sent Done false
29653 8303 Acked Read 6 Done Sent Done false
29654 8304 Acked Read 9 Done Sent Done false
29655 8305 Acked Read 7 Done Sent Done false
29656 8306 Acked Write 5 Done Sent Done false
29657 8307 Acked Write 4 Done Sent Done false
29658 8308 Acked Write 1 Done Sent Done false
29659 8309 Acked Write 8 Done Sent Done false
29660 8310 Acked Read 3 Done Sent Done false
29661 8311 Acked Read 6 Done Sent Done false
29662 8312 Acked Write 2 Done Sent Done false
29663 8313 Acked Read 9 Done Sent Done false
29664 8314 Acked Write 5 Done Sent Done false
29665 8315 Acked Flush 0 Done Sent Done false
29666 8316 Acked Read 1 Done Sent Done false
29667 8317 Acked Write 1 Done Sent Done false
29668 8318 Acked Write 9 Done Sent Done false
29669 8319 Acked Read 6 Done Sent Done false
29670 8320 Acked Write 8 Done Sent Done false
29671 8321 Acked Read 6 Done Sent Done false
29672 8322 Acked Write 8 Done Sent Done false
29673 8323 Acked Read 8 Done Sent Done false
29674 8324 Acked Read 8 Done Sent Done false
29675 8325 Acked Read 9 Done Sent Done false
29676 8326 Acked Read 3 Done Sent Done false
29677 8327 Acked Read 1 Done Sent Done false
29678 8328 Acked Write 6 Done Sent Done false
29679 8329 Acked Read 8 Done Sent Done false
29680 8330 Acked Read 6 Done Sent Done false
29681 8331 Acked Write 10 Done Sent Done false
29682 8332 Acked Flush 0 Done Sent Done false
29683 8333 Acked Read 1 Done Sent Done false
29684 8334 Acked Write 6 Done Sent Done false
29685 8335 Acked Write 5 Done Sent Done false
29686 8336 Acked Write 10 Done Sent Done false
29687 8337 Acked Read 2 Done Sent Done false
29688 8338 Acked Read 6 Done Sent Done false
29689 8339 Acked Write 2 Done Sent Done false
29690 8340 Acked Read 5 Done Sent Done false
29691 8341 Acked Read 1 Done Sent Done false
29692 8342 Acked Read 4 Done Sent Done false
29693 8343 Acked Read 4 Done Sent Done false
29694 8344 Acked Read 2 Done Sent Done false
29695 8345 Acked Write 2 Done Sent Done false
29696 8346 Acked Read 8 Done Sent Done false
29697 8347 Acked Read 1 Done Sent Done false
29698 8348 Acked Write 10 Done Sent Done false
29699 8349 Acked Write 4 Done Sent Done false
29700 8350 Acked Write 2 Done Sent Done false
29701 8351 Acked Read 5 Done Sent Done false
29702 8352 Acked Read 2 Done Sent Done false
29703 8353 Acked Read 6 Done Sent Done false
29704 8354 Acked Flush 0 Done Sent Done false
29705 8355 Acked Flush 0 Done Sent Done false
29706 8356 Acked Read 6 Done Sent Done false
29707 8357 Acked Write 5 Done Sent Done false
29708 8358 Acked Write 4 Done Sent Done false
29709 8359 Acked Write 4 Done Sent Done false
29710 8360 Acked Flush 0 Done Sent Done false
29711 8361 Acked Write 6 Done Sent Done false
29712 8362 Acked Read 3 Done Sent Done false
29713 8363 Acked Read 2 Done Sent Done false
29714 8364 Acked Write 6 Done Sent Done false
29715 8365 Acked Write 4 Done Sent Done false
29716 8366 Acked Read 6 Done Sent Done false
29717 8367 Acked Read 2 Done Sent Done false
29718 8368 Acked Write 10 Done Sent Done false
29719 8369 Acked Flush 0 Done Sent Done false
29720 8370 Acked Read 8 Done Sent Done false
29721 8371 Acked Write 1 Done Sent Done false
29722 8372 Acked Read 3 Done Sent Done false
29723 8373 Acked Write 3 Done Sent Done false
29724 8374 Acked Write 6 Done Sent Done false
29725 8375 Acked Read 9 Done Sent Done false
29726 8376 Acked Read 7 Done Sent Done false
29727 8377 Acked Read 2 Done Sent Done false
29728 8378 Acked Read 5 Done Sent Done false
29729 8379 Acked Write 2 Done Sent Done false
29730 8380 Acked Read 9 Done Sent Done false
29731 8381 Acked Write 5 Done Sent Done false
29732 8382 Acked Write 10 Done Sent Done false
29733 8383 Acked Write 9 Done Sent Done false
29734 8384 Acked Write 3 Done Sent Done false
29735 8385 Acked Read 7 Done Sent Done false
29736 8386 Acked Read 3 Done Sent Done false
29737 8387 Acked Flush 0 Done Sent Done false
29738 8388 Acked Read 3 Done Sent Done false
29739 8389 Acked Flush 0 Done Sent Done false
29740 8390 Acked Write 6 Done Sent Done false
29741 8391 Acked Write 9 Done Sent Done false
29742 8392 Acked Write 2 Done Sent Done false
29743 8393 Acked Read 4 Done Sent Done false
29744 8394 Acked Read 2 Done Sent Done false
29745 8395 Acked Flush 0 Done Sent Done false
29746 8396 Acked Flush 0 Done Sent Done false
29747 8397 Acked Write 7 Done Sent Done false
29748 8398 Acked Read 5 Done Sent Done false
29749 8399 Acked Read 7 Done Sent Done false
29750 8400 Acked Read 3 Done Sent Done false
29751 8401 Acked Read 4 Done Sent Done false
29752 8402 Acked Read 10 Done Sent Done false
29753 8403 Acked Write 2 Done Sent Done false
29754 8404 Acked Read 10 Done Sent Done false
29755 8405 Acked Read 10 Done Sent Done false
29756 8406 Acked Read 10 Done Sent Done false
29757 8407 Acked Write 7 Done Sent Done false
29758 8408 Acked Read 9 Done Sent Done false
29759 8409 Acked Write 3 Done Sent Done false
29760 8410 Acked Write 6 Done Sent Done false
29761 8411 Acked Flush 0 Done Sent Done false
29762 8412 Acked Write 3 Done Sent Done false
29763 8413 Acked Read 3 Done Sent Done false
29764 8414 Acked Write 3 Done Sent Done false
29765 8415 Acked Read 9 Done Sent Done false
29766 8416 Acked Write 2 Done Sent Done false
29767 8417 Acked Write 9 Done Sent Done false
29768 8418 Acked Write 2 Done Sent Done false
29769 8419 Acked Read 6 Done Sent Done false
29770 8420 Acked Write 4 Done Sent Done false
29771 8421 Acked Read 8 Done Sent Done false
29772 8422 Acked Write 9 Done Sent Done false
29773 8423 Acked Read 4 Done Sent Done false
29774 8424 Acked Read 9 Done Sent Done false
29775 8425 Acked Flush 0 Done Sent Done false
29776 8426 Acked Read 2 Done Sent Done false
29777 8427 Acked Read 6 Done Sent Done false
29778 8428 Acked Flush 0 Done Sent Done false
29779 8429 Acked Read 10 Done Sent Done false
29780 8430 Acked Write 8 Done Sent Done false
29781 8431 Acked Read 1 Done Sent Done false
29782 8432 Acked Flush 0 Done Sent Done false
29783 8433 Acked Read 3 Done Sent Done false
29784 8434 Acked Write 3 Done Sent Done false
29785 8435 Acked Read 6 Done Sent Done false
29786 8436 Acked Write 8 Done Sent Done false
29787 8437 Acked Write 7 Done Sent Done false
29788 8438 Acked Write 1 Done Sent Done false
29789 8439 Acked Write 7 Done Sent Done false
29790 8440 Acked Write 4 Done Sent Done false
29791 8441 Acked Write 7 Done Sent Done false
29792 8442 Acked Write 3 Done Sent Done false
29793 8443 Acked Read 2 Done Sent Done false
29794 8444 Acked Read 4 Done Sent Done false
29795 8445 Acked Write 10 Done Sent Done false
29796 8446 Acked Read 9 Done Sent Done false
29797 8447 Acked Write 1 Done Sent Done false
29798 8448 Acked Read 4 Done Sent Done false
29799 8449 Acked Write 7 Done Sent Done false
29800 8450 Acked Read 5 Done Sent Done false
29801 8451 Acked Write 6 Done Sent Done false
29802 8452 Acked Write 6 Done Sent Done false
29803 8453 Acked Write 8 Done Sent Done false
29804 8454 Acked Read 7 Done Sent Done false
29805 8455 Acked Read 8 Done Sent Done false
29806 8456 Acked Read 9 Done Sent Done false
29807 8457 Acked Read 8 Done Sent Done false
29808 8458 Acked Read 8 Done Sent Done false
29809 8459 Acked Read 4 Done Sent Done false
29810 8460 Acked Read 4 Done Sent Done false
29811 8461 Acked Read 8 Done Sent Done false
29812 8462 Acked Read 4 Done Sent Done false
29813 8463 Acked Write 4 Done Sent Done false
29814 8464 Acked Flush 0 Done Sent Done false
29815 8465 Acked Read 8 Done Sent Done false
29816 8466 Acked Flush 0 Done Sent Done false
29817 8467 Acked Write 4 Done Sent Done false
29818 8468 Acked Read 6 Done Sent Done false
29819 8469 Acked Read 10 Done Sent Done false
29820 8470 Acked Read 10 Done Sent Done false
29821 8471 Acked Read 6 Done Sent Done false
29822 8472 Acked Read 10 Done Sent Done false
29823 8473 Acked Write 4 Done Sent Done false
29824 8474 Acked Read 1 Done Sent Done false
29825 8475 Acked Write 8 Done Sent Done false
29826 8476 Acked Read 10 Done Sent Done false
29827 8477 Acked Read 5 Done Sent Done false
29828 8478 Acked Read 2 Done Sent Done false
29829 8479 Acked Write 9 Done Sent Done false
29830 8480 Acked Read 8 Done Sent Done false
29831 8481 Acked Write 4 Done Sent Done false
29832 8482 Acked Read 10 Done Sent Done false
29833 8483 Acked Read 9 Done Sent Done false
29834 8484 Acked Read 2 Done Sent Done false
29835 8485 Acked Read 6 Done Sent Done false
29836 8486 Acked Write 10 Done Sent Done false
29837 8487 Acked Write 5 Done Sent Done false
29838 8488 Acked Flush 0 Done Sent Done false
29839 8489 Acked Write 1 Done Sent Done false
29840 8490 Acked Read 7 Done Sent Done false
29841 8491 Acked Read 9 Done Sent Done false
29842 8492 Acked Flush 0 Done Sent Done false
29843 8493 Acked Read 5 Done Sent Done false
29844 8494 Acked Read 2 Done Sent Done false
29845 8495 Acked Write 7 Done Sent Done false
29846 8496 Acked Write 10 Done Sent Done false
29847 8497 Acked Read 9 Done Sent Done false
29848 8498 Acked Read 5 Done Sent Done false
29849 8499 Acked Write 6 Done Sent Done false
29850 8500 Acked Read 6 Done Sent Done false
29851 8501 Acked Read 7 Done Sent Done false
29852 8502 Acked Write 8 Done Sent Done false
29853 8503 Acked Read 7 Done Sent Done false
29854 8504 Acked Write 10 Done Sent Done false
29855 8505 Acked Write 10 Done Sent Done false
29856 8506 Acked Write 10 Done Sent Done false
29857 8507 Acked Read 5 Done Sent Done false
29858 8508 Acked Flush 0 Done Sent Done false
29859 8509 Acked Write 3 Done Sent Done false
29860 8510 Acked Read 5 Done Sent Done false
29861 8511 Acked Write 7 Done Sent Done false
29862 8512 Acked Read 6 Done Sent Done false
29863 8513 Acked Write 10 Done Sent Done false
29864 8514 Acked Write 9 Done Sent Done false
29865 8515 Acked Read 6 Done Sent Done false
29866 8516 Acked Write 2 Done Sent Done false
29867 8517 Acked Write 6 Done Sent Done false
29868 8518 Acked Write 10 Done Sent Done false
29869 8519 Acked Read 6 Done Sent Done false
29870 8520 Acked Flush 0 Done Sent Done false
29871 8521 Acked Write 1 Done Sent Done false
29872 8522 Acked Read 8 Done Sent Done false
29873 8523 Acked Read 6 Done Sent Done false
29874 8524 Acked Write 2 Done Sent Done false
29875 8525 Acked Write 1 Done Sent Done false
29876 8526 Acked Flush 0 Done Sent Done false
29877 8527 Acked Write 10 Done Sent Done false
29878 8528 Acked Flush 0 Done Sent Done false
29879 8529 Acked Read 5 Done Sent Done false
29880 8530 Acked Write 1 Done Sent Done false
29881 8531 Acked Write 6 Done Sent Done false
29882 8532 Acked Write 5 Done Sent Done false
29883 8533 Acked Write 2 Done Sent Done false
29884 8534 Acked Read 3 Done Sent Done false
29885 8535 Acked Read 7 Done Sent Done false
29886 8536 Acked Read 7 Done Sent Done false
29887 8537 Acked Read 10 Done Sent Done false
29888 8538 Acked Write 4 Done Sent Done false
29889 8539 Acked Flush 0 Done Sent Done false
29890 8540 Acked Read 7 Done Sent Done false
29891 8541 Acked Read 5 Done Sent Done false
29892 8542 Acked Read 7 Done Sent Done false
29893 8543 Acked Write 6 Done Sent Done false
29894 8544 Acked Write 10 Done Sent Done false
29895 8545 Acked Read 4 Done Sent Done false
29896 8546 Acked Write 3 Done Sent Done false
29897 STATES DS:0 DS:1 DS:2 TOTAL
29898 Sent 0 301 0 301
29899 Done 301 0 301 602
29900 Skipped 0 0 0 0
29901 Error 0 0 0 0
29902 Last Flush: Some(JobId(8539)) Some(JobId(8245)) Some(JobId(8539))
29903 Downstairs last five completed: 8245 8244 8243 8242 8241
29904 Upstairs last five completed: 8546 8545 8544 8543 8542
29905 CLIENT: Up:0 ds:301 act:2
299062025-03-17T22:22:58.228ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
299072025-03-17T22:22:59.229ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
299082025-03-17T22:22:59.332ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
299092025-03-17T22:22:59.332ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:53261) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
299102025-03-17T22:22:59.333ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
299112025-03-17T22:22:59.334ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
299122025-03-17T22:22:59.334ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
299132025-03-17T22:22:59.334ZINFOcrucible: send last flush ID to this DS: Some(JobId(8245)) = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
299142025-03-17T22:22:59.334ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
299152025-03-17T22:22:59.334ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(8245)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
299162025-03-17T22:22:59.334ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(8245)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
29917 ----------------------------------------------------------------
29918 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
29919 Downstairs last five completed: 8547 8546 8545 8544 8543
29920 Upstairs last five completed: 8547 8546 8545 8544 8543
29921 CLIENT: Up:0 ds:0 act:3
29922 Replay: All jobs finished
29923 Main task has work to do, go find it
29924 got dsc Stop(1)
29925 stop 1
29926 Mar 17 22:23:01.589[8820] Got stop action so:false kr:false
29927 INFO request completed, latency_us: 375, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 2e2e192c-4aef-4666-9020-7993af04c35a, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
299282025-03-17T22:23:01.590ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
29929 Mar 17 22:23:01.590 INFO request completed, latency_us: 160, response_code: 200, uri: /state/cid/1, method: GET, req_id: 41b9ec71-12c1-4a2c-8c15-d54221330c81, remote_addr: 127.0.0.1:60423, local_addr:{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:23:01.590464678Z","hostname":"w-01JPK2S810SFAA9787XEG8GFX5","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"0f8310d7-099e-4ead-8484-91ddc162d307"}
29930 127.0.0.1:9998{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:23:01.590488446Z","hostname":"w-01JPK2S810SFAA9787XEG8GFX5","pid":1206,"session_id":"0f8310d7-099e-4ead-8484-91ddc162d307"}
29931 
299322025-03-17T22:23:01.590ZINFOcrucible: client task is exiting = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
29933 [8820] Exited with: ExitStatus(unix_wait_status(9))
29934 [8820][1] reports Exit
29935 Mar 17 22:23:05.592 INFO request completed, latency_us: 267, response_code: 200, uri: /state/cid/1, method: GET, req_id: 6cefb929-a458-471b-a507-eeb6227bd7de, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
29936 001/300 Read block 3651 len 40960
29937 002/300 Write block 4528 len 12288 data: 4 4 6
29938 003/300 Write block 3275 len 4096 data: 8
29939 004/300 Write block 2054 len 8192 data: 7 6
29940 005/300 Write block 1110 len 24576 data: 2 3 5 6 6 5
29941 006/300 Write block 2000 len 12288 data: 6 6 6
29942 007/300 Read block 394 len 40960
29943 008/300 Write block 1869 len 36864 data: 5 6 7 5 5 6 5 5 5
29944 009/300 Write block 413 len 24576 data: 5 5 7 7 8 7
29945 010/300 Write block 877 len 28672 data: 4 3 4 4 4 4 3
29946 011/300 Read block 2665 len 24576
29947 012/300 Read block 4072 len 28672
29948 013/300 Flush
29949 014/300 Read block 2640 len 32768
29950 015/300 Read block 4547 len 28672
29951 016/300 Write block 1884 len 24576 data: 6 6 7 8 7 6
29952 017/300 Read block 2890 len 28672
29953 018/300 Write block 891 len 36864 data: 7 7 7 5 4 5 4 4 4
29954 019/300 Write block 815 len 16384 data: 6 5 4 4
29955 020/300 Read block 2606 len 12288
29956 021/300 Flush
29957 022/300 Write block 338 len 20480 data: 6 4 3 4 5
29958 023/300 Read block 1151 len 28672
29959 024/300 Read block 316 len 24576
29960 025/300 Read block 1536 len 8192
29961 026/300 Read block 3825 len 20480
29962 027/300 Write block 4161 len 12288 data: 8 9 8
29963 028/300 Write block 3228 len 12288 data: 7 5 7
29964 029/300 Read block 2561 len 24576
29965 030/300 Write block 3935 len 28672 data: 4 3 3 3 4 5 7
29966 031/300 Read block 4322 len 40960
29967 032/300 Write block 4285 len 4096 data: 6
29968 033/300 Write block 4172 len 24576 data: 8 8 7 4 5 5
29969 034/300 Write block 1837 len 40960 data: 4 4 4 4 4 6 5 6 7 5
29970 035/300 Read block 2699 len 4096
29971 036/300 Read block 2159 len 4096
29972 037/300 Write block 2072 len 36864 data: 8 8 8 6 5 4 5 4 4
29973 038/300 Read block 1456 len 20480
29974 039/300 Flush
29975 040/300 Read block 2515 len 4096
29976 041/300 Read block 901 len 8192
29977 042/300 Read block 3170 len 32768
29978 043/300 Write block 3657 len 24576 data: 5 5 5 4 4 4
29979 044/300 Read block 731 len 12288
29980 045/300 Write block 3147 len 20480 data: 6 6 6 6 5
29981 046/300 Write block 2155 len 36864 data: 6 4 4 3 4 4 3 4 4
29982 047/300 Write block 1403 len 32768 data: 9 8 7 8 7 5 5 6
29983 048/300 Read block 407 len 32768
29984 049/300 Read block 1951 len 12288
29985 050/300 Read block 1049 len 32768
29986 051/300 Write block 1504 len 8192 data: 8 7
29987 052/300 Write block 1069 len 12288 data: 4 3 3
29988 053/300 Flush
29989 054/300 Read block 577 len 40960
29990 055/300 Write block 26 len 12288 data: 9 9 10
29991 056/300 Read block 2690 len 8192
29992 057/300 Flush
29993 058/300 Write block 1492 len 4096 data: 4
29994 059/300 Write block 4819 len 24576 data: 4 4 4 8 7 7
29995 060/300 Write block 1031 len 12288 data: 7 9 10
29996 061/300 Write block 574 len 20480 data: 3 4 5 5 5
29997 062/300 Read block 880 len 12288
29998 063/300 Read block 2096 len 32768
29999 064/300 Flush
30000 065/300 Flush
30001 066/300 Read block 2519 len 16384
30002 067/300 Read block 4405 len 12288
30003 068/300 Read block 3312 len 16384
30004 069/300 Write block 98 len 28672 data: 4 5 4 3 3 3 4
30005 070/300 Read block 2414 len 24576
30006 071/300 Flush
30007 072/300 Read block 4873 len 24576
30008 073/300 Read block 3348 len 36864
30009 074/300 Write block 3886 len 28672 data: 6 9 8 7 7 6 7
30010 075/300 Flush
30011 076/300 Read block 1519 len 4096
30012 077/300 Read block 348 len 24576
30013 078/300 Flush
30014 079/300 Write block 1017 len 4096 data: 5
30015 080/300 Read block 4466 len 16384
30016 081/300 Read block 4056 len 12288
30017 082/300 Write block 3059 len 28672 data: 4 5 7 6 6 5 6
30018 083/300 Read block 1007 len 36864
30019 084/300 Read block 2673 len 40960
30020 085/300 Write block 1706 len 20480 data: 5 5 3 4 4
30021 086/300 Read block 343 len 28672
30022 087/300 Read block 3002 len 4096
30023 088/300 Write block 3575 len 32768 data: 3 3 2 2 3 3 4 5
30024 089/300 Write block 1427 len 24576 data: 4 4 3 4 3 3
30025 090/300 Read block 1522 len 32768
30026 091/300 Write block 1551 len 4096 data: 4
30027 092/300 Write block 4660 len 36864 data: 3 5 5 6 5 5 5 6 5
30028 093/300 Read block 4668 len 16384
30029 094/300 Write block 4555 len 28672 data: 5 6 4 5 4 4 3
30030 095/300 Write block 2352 len 36864 data: 5 5 4 4 5 6 8 8 6
30031 096/300 Write block 3251 len 28672 data: 5 4 4 6 6 5 5
30032 097/300 Write block 947 len 12288 data: 5 4 3
30033 098/300 Read block 3586 len 40960
30034 099/300 Read block 1783 len 28672
30035 100/300 Write block 3186 len 8192 data: 2 4
30036 101/300 Read block 4838 len 4096
30037 102/300 Read block 4164 len 32768
30038 103/300 Write block 2007 len 40960 data: 5 7 4 3 4 5 5 4 5 5
30039 104/300 Write block 2078 len 36864 data: 6 5 5 3 3 3 3 4 5
30040 105/300 Write block 917 len 12288 data: 5 4 4
30041 106/300 Write block 746 len 40960 data: 5 4 3 3 4 4 4 4 6 6
30042 107/300 Write block 1718 len 40960 data: 6 6 5 3 4 4 3 3 3 2
30043 108/300 Write block 1334 len 16384 data: 6 6 5 4
30044 109/300 Write block 1823 len 36864 data: 4 5 3 6 7 5 6 5 6
30045 110/300 Read block 1385 len 4096
30046 111/300 Read block 2643 len 24576
30047 112/300 Read block 896 len 4096
30048 113/300 Write block 3935 len 28672 data: 5 4 4 4 5 6 8
30049 114/300 Write block 3827 len 4096 data: 9
30050 115/300 Read block 2478 len 8192
30051 116/300 Write block 1597 len 20480 data: 7 8 8 4 4
30052 117/300 Read block 109 len 40960
30053 118/300 Read block 523 len 8192
30054 119/300 Write block 3964 len 32768 data: 3 3 4 3 4 3 4 4
30055 120/300 Read block 4952 len 8192
30056 121/300 Read block 3845 len 40960
30057 122/300 Write block 4877 len 32768 data: 6 7 6 5 5 5 8 9
30058 123/300 Read block 4217 len 4096
30059 124/300 Read block 3000 len 40960
30060 125/300 Read block 3500 len 16384
30061 126/300 Read block 1319 len 32768
30062 127/300 Read block 580 len 24576
30063 128/300 Write block 4494 len 4096 data: 4
30064 129/300 Flush
30065 130/300 Read block 4518 len 36864
30066 131/300 Write block 1829 len 12288 data: 7 6 7
30067 132/300 Read block 3383 len 28672
30068 133/300 Write block 3025 len 40960 data: 3 5 3 2 3 3 2 2 4 4
30069 134/300 Read block 2023 len 8192
30070 135/300 Read block 4955 len 32768
30071 136/300 Read block 1479 len 24576
30072 137/300 Write block 2542 len 36864 data: 3 5 4 5 4 4 6 7 7
30073 138/300 Write block 485 len 8192 data: 7 5
30074 139/300 Read block 3670 len 28672
30075 140/300 Read block 4660 len 28672
30076 141/300 Read block 1052 len 24576
30077 142/300 Read block 1632 len 16384
30078 143/300 Write block 2670 len 36864 data: 3 5 7 7 7 7 7 6 6
30079 144/300 Write block 2602 len 4096 data: 4
30080 145/300 Read block 2249 len 32768
30081 146/300 Read block 3682 len 40960
30082 147/300 Write block 925 len 24576 data: 5 5 5 6 6 8
30083 148/300 Flush
30084 149/300 Read block 836 len 28672
30085 150/300 Write block 4987 len 20480 data: 5 4 3 5 6
30086 151/300 Write block 1247 len 8192 data: 3 2
30087 152/300 Read block 1936 len 8192
30088 153/300 Write block 3564 len 36864 data: 7 7 7 5 4 4 5 4 5
30089 154/300 Write block 4877 len 16384 data: 7 8 7 6
30090 155/300 Write block 3618 len 24576 data: 6 4 2 3 3 3
30091 156/300 Read block 2991 len 12288
30092 157/300 Write block 4961 len 16384 data: 3 3 4 3
30093 158/300 Write block 819 len 20480 data: 4 3 3 3 2
30094 159/300 Read block 4784 len 4096
30095 160/300 Read block 1073 len 12288
30096 161/300 Write block 4336 len 16384 data: 7 12 11 11
30097 162/300 Write block 2259 len 32768 data: 4 4 3 4 4 4 5 5
30098 163/300 Read block 4192 len 16384
30099 164/300 Read block 452 len 36864
30100 165/300 Write block 3041 len 28672 data: 4 4 4 5 6 6 7
30101 166/300 Read block 1928 len 16384
30102 167/300 Write block 4763 len 28672 data: 5 6 6 6 6 5 5
30103 168/300 Flush
30104 169/300 Write block 4981 len 40960 data: 6 5 5 5 6 7 6 5 4 6
30105 170/300 Read block 897 len 8192
30106 171/300 Read block 4224 len 20480
30107 172/300 Read block 1244 len 40960
30108 173/300 Write block 602 len 36864 data: 4 7 7 8 7 7 7 6 5
30109 174/300 Flush
30110 175/300 Write block 4162 len 40960 data: 10 9 7 5 8 7 8 6 8 8
30111 176/300 Write block 1311 len 32768 data: 7 7 8 6 5 4 4 4
30112 177/300 Write block 1394 len 20480 data: 6 7 8 8 8
30113 178/300 Read block 2164 len 32768
30114 179/300 Write block 553 len 36864 data: 8 8 8 9 11 10 11 11 11
30115 180/300 Write block 3070 len 20480 data: 6 7 6 7 6
30116 181/300 Read block 1867 len 24576
30117 182/300 Write block 4846 len 32768 data: 4 4 6 6 6 6 5 4
30118 183/300 Write block 469 len 20480 data: 5 4 4 5 3
30119 184/300 Write block 3818 len 24576 data: 4 4 5 5 3 3
30120 185/300 Read block 1746 len 40960
30121 186/300 Read block 4120 len 40960
30122 187/300 Read block 2535 len 32768
30123 188/300 Write block 2832 len 8192 data: 4 6
30124 189/300 Read block 1260 len 32768
30125 190/300 Read block 130 len 8192
30126 191/300 Flush
30127 192/300 Read block 1566 len 24576
30128 193/300 Write block 2254 len 20480 data: 4 4 5 4 4
30129 194/300 Write block 4643 len 12288 data: 5 6 5
30130 195/300 Read block 4244 len 16384
30131 196/300 Write block 2708 len 16384 data: 6 5 3 2
30132 197/300 Read block 4990 len 8192
30133 198/300 Write block 2123 len 16384 data: 7 5 4 4
30134 199/300 Read block 4577 len 36864
30135 200/300 Read block 3277 len 16384
30136 201/300 Read block 1404 len 12288
30137 202/300 Read block 1070 len 16384
30138 203/300 Write block 4848 len 12288 data: 7 7 7
30139 204/300 Read block 1049 len 12288
30140 205/300 Write block 2989 len 12288 data: 8 8 7
30141 206/300 Read block 2655 len 12288
30142 207/300 Write block 726 len 28672 data: 5 4 4 3 3 3 3
30143 208/300 Read block 2554 len 20480
30144 209/300 Write block 4603 len 8192 data: 6 9
30145 210/300 Write block 3422 len 16384 data: 8 7 7 6
30146 211/300 Read block 1825 len 32768
30147 212/300 Write block 1154 len 24576 data: 7 6 8 7 6 4
30148 213/300 Read block 1417 len 40960
30149 214/300 Read block 2908 len 12288
30150 215/300 Write block 1731 len 12288 data: 3 4 4
30151 216/300 Read block 3600 len 20480
30152 217/300 Read block 2816 len 16384
30153 218/300 Write block 2304 len 16384 data: 5 5 4 3
30154 219/300 Write block 3366 len 20480 data: 6 6 6 6 6
30155 220/300 Write block 1526 len 4096 data: 11
30156 221/300 Read block 1678 len 12288
30157 222/300 Write block 2363 len 16384 data: 5 5 5 5
30158 223/300 Read block 2561 len 40960
30159 224/300 Read block 2500 len 24576
30160 225/300 Flush
30161 226/300 Read block 3084 len 8192
30162 227/300 Read block 505 len 36864
30163 228/300 Flush
30164 229/300 Read block 4169 len 36864
30165 230/300 Read block 1731 len 32768
30166 231/300 Read block 2844 len 12288
30167 232/300 Read block 4216 len 16384
30168 233/300 Write block 1290 len 20480 data: 5 7 6 6 5
30169 234/300 Write block 1912 len 32768 data: 2 3 4 5 6 6 5 5
30170 235/300 Read block 1539 len 4096
30171 236/300 Read block 1049 len 24576
30172 237/300 Write block 1382 len 24576 data: 4 3 3 4 4 4
30173 238/300 Write block 3924 len 20480 data: 4 3 2 4 5
30174 239/300 Write block 990 len 4096 data: 7
30175 240/300 Write block 1653 len 32768 data: 5 5 4 4 5 5 5 5
30176 241/300 Read block 3254 len 32768
30177 242/300 Write block 3021 len 24576 data: 5 4 4 3 4 6
30178 243/300 Write block 2553 len 4096 data: 5
30179 244/300 Read block 2742 len 20480
30180 245/300 Write block 4974 len 20480 data: 5 6 6 6 6
30181 246/300 Write block 3187 len 12288 data: 5 4 4
30182 247/300 Read block 2697 len 20480
30183 248/300 Write block 1881 len 40960 data: 5 7 5 7 7 8 9 8 7 6
30184 249/300 Read block 4301 len 4096
30185 250/300 Read block 4388 len 36864
30186 251/300 Read block 151 len 20480
30187 252/300 Write block 2901 len 20480 data: 7 6 6 6 6
30188 253/300 Write block 2355 len 40960 data: 5 6 7 9 9 7 5 5 6 6
30189 254/300 Write block 1954 len 20480 data: 5 5 5 4 5
30190 255/300 Write block 4362 len 28672 data: 4 4 4 4 7 9 9
30191 256/300 Write block 4505 len 8192 data: 5 5
30192 257/300 Write block 2151 len 12288 data: 5 4 4
30193 258/300 Write block 2799 len 40960 data: 3 3 3 4 5 5 6 3 4 5
30194 259/300 Write block 853 len 28672 data: 10 10 9 7 6 7 7
30195 260/300 Write block 100 len 8192 data: 5 4
30196 261/300 Read block 3940 len 16384
30197 262/300 Write block 1001 len 16384 data: 3 3 3 3
30198 263/300 Flush
30199 264/300 Flush
30200 265/300 Read block 3093 len 36864
30201 266/300 Flush
30202 267/300 Read block 3381 len 24576
30203 268/300 Write block 1200 len 12288 data: 4 5 4
30204 269/300 Write block 885 len 8192 data: 4 4
30205 270/300 Read block 1076 len 40960
30206 271/300 Flush
30207 272/300 Write block 13 len 24576 data: 8 8 8 6 5 5
30208 273/300 Read block 1947 len 20480
30209 274/300 Read block 3851 len 36864
30210 275/300 Read block 3810 len 8192
30211 276/300 Read block 987 len 12288
30212 277/300 Read block 2065 len 24576
30213 278/300 Read block 213 len 28672
30214 279/300 Read block 2298 len 20480
30215 280/300 Read block 1393 len 12288
30216 281/300 Write block 122 len 12288 data: 7 7 8
30217 282/300 Read block 198 len 36864
30218 283/300 Read block 4106 len 12288
30219 284/300 Write block 481 len 24576 data: 4 5 6 8 8 6
30220 285/300 Read block 4588 len 24576
30221 286/300 Write block 113 len 8192 data: 4 4
30222 287/300 Read block 1492 len 24576
30223 288/300 Read block 4655 len 28672
30224 289/300 Read block 3088 len 8192
30225 290/300 Read block 1617 len 32768
30226 291/300 Write block 98 len 40960 data: 5 6 6 5 4 4 5 4 4 5
30227 292/300 Read block 2284 len 8192
30228 293/300 Flush
30229 294/300 Write block 1981 len 24576 data: 8 6 7 7 8 10
30230 295/300 Flush
30231 296/300 Read block 2157 len 36864
30232 297/300 Write block 1356 len 28672 data: 8 9 11 10 10 8 8
30233 298/300 Write block 917 len 32768 data: 6 5 5 4 4 5 4 5
30234 299/300 Write block 2803 len 28672 data: 6 6 7 4 5 6 4
30235 300/300 Flush
30236 301/300 Read block 3840 len 12288
30237 Main task has work to do, go find it
30238 got dsc Start(1)
30239 start 1
30240 Mar 17 22:23:05.837 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/1, method: POST, req_id: a9209701-fb7f-47d4-b601-19989228bd34, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
30241 Replay: started 1, returned:Ok(())
30242 ----------------------------------------------------------------
30243 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30244 States: Active Offline Active
30245 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30246 8548 Acked Read 10 Done Sent Done false
30247 8549 Acked Write 3 Done Sent Done false
30248 8550 Acked Write 1 Done Sent Done false
30249 8551 Acked Write 2 Done Sent Done false
30250 8552 Acked Write 6 Done Sent Done false
30251 8553 Acked Write 3 Done Sent Done false
30252 8554 Acked Read 10 Done Sent Done false
30253 8555 Acked Write 9 Done Sent Done false
30254 8556 Acked Write 6 Done Sent Done false
30255 8557 Acked Write 7 Done Sent Done false
30256 8558 Acked Read 6 Done Sent Done false
30257 8559 Acked Read 7 Done Sent Done false
30258 8560 Acked Flush 0 Done Sent Done false
30259 8561 Acked Read 8 Done Sent Done false
30260 8562 Acked Read 7 Done Sent Done false
30261 8563 Acked Write 6 Done Sent Done false
30262 8564 Acked Read 7 Done Sent Done false
30263 8565 Acked Write 9 Done Sent Done false
30264 8566 Acked Write 4 Done Sent Done false
30265 8567 Acked Read 3 Done Sent Done false
30266 8568 Acked Flush 0 Done Sent Done false
30267 8569 Acked Write 5 Done Sent Done false
30268 8570 Acked Read 7 Done Sent Done false
30269 8571 Acked Read 6 Done Sent Done false
30270 8572 Acked Read 2 Done Sent Done false
30271 8573 Acked Read 5 Done Sent Done false
30272 8574 Acked Write 3 Done Sent Done false
30273 8575 Acked Write 3 Done Sent Done false
30274 8576 Acked Read 6 Done Sent Done false
30275 8577 Acked Write 7 Done Sent Done false
30276 8578 Acked Read 10 Done Sent Done false
30277 8579 Acked Write 1 Done Sent Done false
30278 8580 Acked Write 6 Done Sent Done false
30279 8581 Acked Write 10 Done Sent Done false
30280 8582 Acked Read 1 Done Sent Done false
30281 8583 Acked Read 1 Done Sent Done false
30282 8584 Acked Write 9 Done Sent Done false
30283 8585 Acked Read 5 Done Sent Done false
30284 8586 Acked Flush 0 Done Sent Done false
30285 8587 Acked Read 1 Done Sent Done false
30286 8588 Acked Read 2 Done Sent Done false
30287 8589 Acked Read 8 Done Sent Done false
30288 8590 Acked Write 6 Done Sent Done false
30289 8591 Acked Read 3 Done Sent Done false
30290 8592 Acked Write 5 Done Sent Done false
30291 8593 Acked Write 9 Done Sent Done false
30292 8594 Acked Write 8 Done Sent Done false
30293 8595 Acked Read 8 Done Sent Done false
30294 8596 Acked Read 3 Done Sent Done false
30295 8597 Acked Read 8 Done Sent Done false
30296 8598 Acked Write 2 Done Sent Done false
30297 8599 Acked Write 3 Done Sent Done false
30298 8600 Acked Flush 0 Done Sent Done false
30299 8601 Acked Read 10 Done Sent Done false
30300 8602 Acked Write 3 Done Sent Done false
30301 8603 Acked Read 2 Done Sent Done false
30302 8604 Acked Flush 0 Done Sent Done false
30303 8605 Acked Write 1 Done Sent Done false
30304 8606 Acked Write 6 Done Sent Done false
30305 8607 Acked Write 3 Done Sent Done false
30306 8608 Acked Write 5 Done Sent Done false
30307 8609 Acked Read 3 Done Sent Done false
30308 8610 Acked Read 8 Done Sent Done false
30309 8611 Acked Flush 0 Done Sent Done false
30310 8612 Acked Flush 0 Done Sent Done false
30311 8613 Acked Read 4 Done Sent Done false
30312 8614 Acked Read 3 Done Sent Done false
30313 8615 Acked Read 4 Done Sent Done false
30314 8616 Acked Write 7 Done Sent Done false
30315 8617 Acked Read 6 Done Sent Done false
30316 8618 Acked Flush 0 Done Sent Done false
30317 8619 Acked Read 6 Done Sent Done false
30318 8620 Acked Read 9 Done Sent Done false
30319 8621 Acked Write 7 Done Sent Done false
30320 8622 Acked Flush 0 Done Sent Done false
30321 8623 Acked Read 1 Done Sent Done false
30322 8624 Acked Read 6 Done Sent Done false
30323 8625 Acked Flush 0 Done Sent Done false
30324 8626 Acked Write 1 Done Sent Done false
30325 8627 Acked Read 4 Done Sent Done false
30326 8628 Acked Read 3 Done Sent Done false
30327 8629 Acked Write 7 Done Sent Done false
30328 8630 Acked Read 9 Done Sent Done false
30329 8631 Acked Read 10 Done Sent Done false
30330 8632 Acked Write 5 Done Sent Done false
30331 8633 Acked Read 7 Done Sent Done false
30332 8634 Acked Read 1 Done Sent Done false
30333 8635 Acked Write 8 Done Sent Done false
30334 8636 Acked Write 6 Done Sent Done false
30335 8637 Acked Read 8 Done Sent Done false
30336 8638 Acked Write 1 Done Sent Done false
30337 8639 Acked Write 9 Done Sent Done false
30338 8640 Acked Read 4 Done Sent Done false
30339 8641 Acked Write 7 Done Sent Done false
30340 8642 Acked Write 9 Done Sent Done false
30341 8643 Acked Write 7 Done Sent Done false
30342 8644 Acked Write 3 Done Sent Done false
30343 8645 Acked Read 10 Done Sent Done false
30344 8646 Acked Read 7 Done Sent Done false
30345 8647 Acked Write 2 Done Sent Done false
30346 8648 Acked Read 1 Done Sent Done false
30347 8649 Acked Read 8 Done Sent Done false
30348 8650 Acked Write 10 Done Sent Done false
30349 8651 Acked Write 9 Done Sent Done false
30350 8652 Acked Write 3 Done Sent Done false
30351 8653 Acked Write 10 Done Sent Done false
30352 8654 Acked Write 10 Done Sent Done false
30353 8655 Acked Write 4 Done Sent Done false
30354 8656 Acked Write 9 Done Sent Done false
30355 8657 Acked Read 1 Done Sent Done false
30356 8658 Acked Read 6 Done Sent Done false
30357 8659 Acked Read 1 Done Sent Done false
30358 8660 Acked Write 7 Done Sent Done false
30359 8661 Acked Write 1 Done Sent Done false
30360 8662 Acked Read 2 Done Sent Done false
30361 8663 Acked Write 5 Done Sent Done false
30362 8664 Acked Read 10 Done Sent Done false
30363 8665 Acked Read 2 Done Sent Done false
30364 8666 Acked Write 8 Done Sent Done false
30365 8667 Acked Read 2 Done Sent Done false
30366 8668 Acked Read 10 Done Sent Done false
30367 8669 Acked Write 8 Done Sent Done false
30368 8670 Acked Read 1 Done Sent Done false
30369 8671 Acked Read 10 Done Sent Done false
30370 8672 Acked Read 4 Done Sent Done false
30371 8673 Acked Read 8 Done Sent Done false
30372 8674 Acked Read 6 Done Sent Done false
30373 8675 Acked Write 1 Done Sent Done false
30374 8676 Acked Flush 0 Done Sent Done false
30375 8677 Acked Read 9 Done Sent Done false
30376 8678 Acked Write 3 Done Sent Done false
30377 8679 Acked Read 7 Done Sent Done false
30378 8680 Acked Write 10 Done Sent Done false
30379 8681 Acked Read 2 Done Sent Done false
30380 8682 Acked Read 8 Done Sent Done false
30381 8683 Acked Read 6 Done Sent Done false
30382 8684 Acked Write 9 Done Sent Done false
30383 8685 Acked Write 2 Done Sent Done false
30384 8686 Acked Read 7 Done Sent Done false
30385 8687 Acked Read 7 Done Sent Done false
30386 8688 Acked Read 6 Done Sent Done false
30387 8689 Acked Read 4 Done Sent Done false
30388 8690 Acked Write 9 Done Sent Done false
30389 8691 Acked Write 1 Done Sent Done false
30390 8692 Acked Read 8 Done Sent Done false
30391 8693 Acked Read 10 Done Sent Done false
30392 8694 Acked Write 6 Done Sent Done false
30393 8695 Acked Flush 0 Done Sent Done false
30394 8696 Acked Read 7 Done Sent Done false
30395 8697 Acked Write 5 Done Sent Done false
30396 8698 Acked Write 2 Done Sent Done false
30397 8699 Acked Read 2 Done Sent Done false
30398 8700 Acked Write 9 Done Sent Done false
30399 8701 Acked Write 4 Done Sent Done false
30400 8702 Acked Write 6 Done Sent Done false
30401 8703 Acked Read 3 Done Sent Done false
30402 8704 Acked Write 4 Done Sent Done false
30403 8705 Acked Write 5 Done Sent Done false
30404 8706 Acked Read 1 Done Sent Done false
30405 8707 Acked Read 3 Done Sent Done false
30406 8708 Acked Write 4 Done Sent Done false
30407 8709 Acked Write 8 Done Sent Done false
30408 8710 Acked Read 4 Done Sent Done false
30409 8711 Acked Read 9 Done Sent Done false
30410 8712 Acked Write 7 Done Sent Done false
30411 8713 Acked Read 4 Done Sent Done false
30412 8714 Acked Write 7 Done Sent Done false
30413 8715 Acked Flush 0 Done Sent Done false
30414 8716 Acked Write 10 Done Sent Done false
30415 8717 Acked Read 2 Done Sent Done false
30416 8718 Acked Read 5 Done Sent Done false
30417 8719 Acked Read 10 Done Sent Done false
30418 8720 Acked Write 9 Done Sent Done false
30419 8721 Acked Flush 0 Done Sent Done false
30420 8722 Acked Write 10 Done Sent Done false
30421 8723 Acked Write 8 Done Sent Done false
30422 8724 Acked Write 5 Done Sent Done false
30423 8725 Acked Read 8 Done Sent Done false
30424 8726 Acked Write 9 Done Sent Done false
30425 8727 Acked Write 5 Done Sent Done false
30426 8728 Acked Read 6 Done Sent Done false
30427 8729 Acked Write 8 Done Sent Done false
30428 8730 Acked Write 5 Done Sent Done false
30429 8731 Acked Write 6 Done Sent Done false
30430 8732 Acked Read 10 Done Sent Done false
30431 8733 Acked Read 10 Done Sent Done false
30432 8734 Acked Read 8 Done Sent Done false
30433 8735 Acked Write 2 Done Sent Done false
30434 8736 Acked Read 8 Done Sent Done false
30435 8737 Acked Read 2 Done Sent Done false
30436 8738 Acked Flush 0 Done Sent Done false
30437 8739 Acked Read 6 Done Sent Done false
30438 8740 Acked Write 5 Done Sent Done false
30439 8741 Acked Write 3 Done Sent Done false
30440 8742 Acked Read 4 Done Sent Done false
30441 8743 Acked Write 4 Done Sent Done false
30442 8744 Acked Read 2 Done Sent Done false
30443 8745 Acked Write 4 Done Sent Done false
30444 8746 Acked Read 9 Done Sent Done false
30445 8747 Acked Read 4 Done Sent Done false
30446 8748 Acked Read 3 Done Sent Done false
30447 8749 Acked Read 4 Done Sent Done false
30448 8750 Acked Write 3 Done Sent Done false
30449 8751 Acked Read 3 Done Sent Done false
30450 8752 Acked Write 3 Done Sent Done false
30451 8753 Acked Read 3 Done Sent Done false
30452 8754 Acked Write 7 Done Sent Done false
30453 8755 Acked Read 5 Done Sent Done false
30454 8756 Acked Write 2 Done Sent Done false
30455 8757 Acked Write 4 Done Sent Done false
30456 8758 Acked Read 8 Done Sent Done false
30457 8759 Acked Write 6 Done Sent Done false
30458 8760 Acked Read 10 Done Sent Done false
30459 8761 Acked Read 3 Done Sent Done false
30460 8762 Acked Write 3 Done Sent Done false
30461 8763 Acked Read 5 Done Sent Done false
30462 8764 Acked Read 4 Done Sent Done false
30463 8765 Acked Write 4 Done Sent Done false
30464 8766 Acked Write 5 Done Sent Done false
30465 8767 Acked Write 1 Done Sent Done false
30466 8768 Acked Read 3 Done Sent Done false
30467 8769 Acked Write 4 Done Sent Done false
30468 8770 Acked Read 10 Done Sent Done false
30469 8771 Acked Read 6 Done Sent Done false
30470 8772 Acked Flush 0 Done Sent Done false
30471 8773 Acked Read 2 Done Sent Done false
30472 8774 Acked Read 9 Done Sent Done false
30473 8775 Acked Flush 0 Done Sent Done false
30474 8776 Acked Read 9 Done Sent Done false
30475 8777 Acked Read 8 Done Sent Done false
30476 8778 Acked Read 3 Done Sent Done false
30477 8779 Acked Read 4 Done Sent Done false
30478 8780 Acked Write 5 Done Sent Done false
30479 8781 Acked Write 8 Done Sent Done false
30480 8782 Acked Read 1 Done Sent Done false
30481 8783 Acked Read 6 Done Sent Done false
30482 8784 Acked Write 6 Done Sent Done false
30483 8785 Acked Write 5 Done Sent Done false
30484 8786 Acked Write 1 Done Sent Done false
30485 8787 Acked Write 8 Done Sent Done false
30486 8788 Acked Read 8 Done Sent Done false
30487 8789 Acked Write 6 Done Sent Done false
30488 8790 Acked Write 1 Done Sent Done false
30489 8791 Acked Read 5 Done Sent Done false
30490 8792 Acked Write 5 Done Sent Done false
30491 8793 Acked Write 3 Done Sent Done false
30492 8794 Acked Read 5 Done Sent Done false
30493 8795 Acked Write 10 Done Sent Done false
30494 8796 Acked Read 1 Done Sent Done false
30495 8797 Acked Read 9 Done Sent Done false
30496 8798 Acked Read 5 Done Sent Done false
30497 8799 Acked Write 5 Done Sent Done false
30498 8800 Acked Write 10 Done Sent Done false
30499 8801 Acked Write 5 Done Sent Done false
30500 8802 Acked Write 7 Done Sent Done false
30501 8803 Acked Write 2 Done Sent Done false
30502 8804 Acked Write 3 Done Sent Done false
30503 8805 Acked Write 10 Done Sent Done false
30504 8806 Acked Write 7 Done Sent Done false
30505 8807 Acked Write 2 Done Sent Done false
30506 8808 Acked Read 4 Done Sent Done false
30507 8809 Acked Write 4 Done Sent Done false
30508 8810 Acked Flush 0 Done Sent Done false
30509 8811 Acked Flush 0 Done Sent Done false
30510 8812 Acked Read 9 Done Sent Done false
30511 8813 Acked Flush 0 Done Sent Done false
30512 8814 Acked Read 6 Done Sent Done false
30513 8815 Acked Write 3 Done Sent Done false
30514 8816 Acked Write 2 Done Sent Done false
30515 8817 Acked Read 10 Done Sent Done false
30516 8818 Acked Flush 0 Done Sent Done false
30517 8819 Acked Write 6 Done Sent Done false
30518 8820 Acked Read 5 Done Sent Done false
30519 8821 Acked Read 9 Done Sent Done false
30520 8822 Acked Read 2 Done Sent Done false
30521 8823 Acked Read 3 Done Sent Done false
30522 8824 Acked Read 6 Done Sent Done false
30523 8825 Acked Read 7 Done Sent Done false
30524 8826 Acked Read 5 Done Sent Done false
30525 8827 Acked Read 3 Done Sent Done false
30526 8828 Acked Write 3 Done Sent Done false
30527 8829 Acked Read 9 Done Sent Done false
30528 8830 Acked Read 3 Done Sent Done false
30529 8831 Acked Write 6 Done Sent Done false
30530 8832 Acked Read 6 Done Sent Done false
30531 8833 Acked Write 2 Done Sent Done false
30532 8834 Acked Read 6 Done Sent Done false
30533 8835 Acked Read 7 Done Sent Done false
30534 8836 Acked Read 2 Done Sent Done false
30535 8837 Acked Read 8 Done Sent Done false
30536 8838 Acked Write 10 Done Sent Done false
30537 8839 Acked Read 2 Done Sent Done false
30538 8840 Acked Flush 0 Done Sent Done false
30539 8841 Acked Write 6 Done Sent Done false
30540 8842 Acked Flush 0 Done Sent Done false
30541 8843 Acked Read 9 Done Sent Done false
30542 8844 Acked Write 7 Done Sent Done false
30543 8845 Acked Write 8 Done Sent Done false
30544 8846 Acked Write 7 Done Sent Done false
30545 8847 Acked Flush 0 Done Sent Done false
30546 8848 Acked Read 3 Done Sent Done false
30547 STATES DS:0 DS:1 DS:2 TOTAL
30548 Sent 0 301 0 301
30549 Done 301 0 301 602
30550 Skipped 0 0 0 0
30551 Error 0 0 0 0
30552 Last Flush: Some(JobId(8847)) Some(JobId(8547)) Some(JobId(8847))
30553 Downstairs last five completed: 8547 8546 8545 8544 8543
30554 Upstairs last five completed: 8848 8847 8846 8845 8844
30555 CLIENT: Up:0 ds:301 act:2
305562025-03-17T22:23:06.339ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
305572025-03-17T22:23:07.239ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
305582025-03-17T22:23:08.242ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
305592025-03-17T22:23:09.242ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
30560 ----------------------------------------------------------------
30561 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30562 States: Active Offline Active
30563 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30564 8548 Acked Read 10 Done Sent Done false
30565 8549 Acked Write 3 Done Sent Done false
30566 8550 Acked Write 1 Done Sent Done false
30567 8551 Acked Write 2 Done Sent Done false
30568 8552 Acked Write 6 Done Sent Done false
30569 8553 Acked Write 3 Done Sent Done false
30570 8554 Acked Read 10 Done Sent Done false
30571 8555 Acked Write 9 Done Sent Done false
30572 8556 Acked Write 6 Done Sent Done false
30573 8557 Acked Write 7 Done Sent Done false
30574 8558 Acked Read 6 Done Sent Done false
30575 8559 Acked Read 7 Done Sent Done false
30576 8560 Acked Flush 0 Done Sent Done false
30577 8561 Acked Read 8 Done Sent Done false
30578 8562 Acked Read 7 Done Sent Done false
30579 8563 Acked Write 6 Done Sent Done false
30580 8564 Acked Read 7 Done Sent Done false
30581 8565 Acked Write 9 Done Sent Done false
30582 8566 Acked Write 4 Done Sent Done false
30583 8567 Acked Read 3 Done Sent Done false
30584 8568 Acked Flush 0 Done Sent Done false
30585 8569 Acked Write 5 Done Sent Done false
30586 8570 Acked Read 7 Done Sent Done false
30587 8571 Acked Read 6 Done Sent Done false
30588 8572 Acked Read 2 Done Sent Done false
30589 8573 Acked Read 5 Done Sent Done false
30590 8574 Acked Write 3 Done Sent Done false
30591 8575 Acked Write 3 Done Sent Done false
30592 8576 Acked Read 6 Done Sent Done false
30593 8577 Acked Write 7 Done Sent Done false
30594 8578 Acked Read 10 Done Sent Done false
30595 8579 Acked Write 1 Done Sent Done false
30596 8580 Acked Write 6 Done Sent Done false
30597 8581 Acked Write 10 Done Sent Done false
30598 8582 Acked Read 1 Done Sent Done false
30599 8583 Acked Read 1 Done Sent Done false
30600 8584 Acked Write 9 Done Sent Done false
30601 8585 Acked Read 5 Done Sent Done false
30602 8586 Acked Flush 0 Done Sent Done false
30603 8587 Acked Read 1 Done Sent Done false
30604 8588 Acked Read 2 Done Sent Done false
30605 8589 Acked Read 8 Done Sent Done false
30606 8590 Acked Write 6 Done Sent Done false
30607 8591 Acked Read 3 Done Sent Done false
30608 8592 Acked Write 5 Done Sent Done false
30609 8593 Acked Write 9 Done Sent Done false
30610 8594 Acked Write 8 Done Sent Done false
30611 8595 Acked Read 8 Done Sent Done false
30612 8596 Acked Read 3 Done Sent Done false
30613 8597 Acked Read 8 Done Sent Done false
30614 8598 Acked Write 2 Done Sent Done false
30615 8599 Acked Write 3 Done Sent Done false
30616 8600 Acked Flush 0 Done Sent Done false
30617 8601 Acked Read 10 Done Sent Done false
30618 8602 Acked Write 3 Done Sent Done false
30619 8603 Acked Read 2 Done Sent Done false
30620 8604 Acked Flush 0 Done Sent Done false
30621 8605 Acked Write 1 Done Sent Done false
30622 8606 Acked Write 6 Done Sent Done false
30623 8607 Acked Write 3 Done Sent Done false
30624 8608 Acked Write 5 Done Sent Done false
30625 8609 Acked Read 3 Done Sent Done false
30626 8610 Acked Read 8 Done Sent Done false
30627 8611 Acked Flush 0 Done Sent Done false
30628 8612 Acked Flush 0 Done Sent Done false
30629 8613 Acked Read 4 Done Sent Done false
30630 8614 Acked Read 3 Done Sent Done false
30631 8615 Acked Read 4 Done Sent Done false
30632 8616 Acked Write 7 Done Sent Done false
30633 8617 Acked Read 6 Done Sent Done false
30634 8618 Acked Flush 0 Done Sent Done false
30635 8619 Acked Read 6 Done Sent Done false
30636 8620 Acked Read 9 Done Sent Done false
30637 8621 Acked Write 7 Done Sent Done false
30638 8622 Acked Flush 0 Done Sent Done false
30639 8623 Acked Read 1 Done Sent Done false
30640 8624 Acked Read 6 Done Sent Done false
30641 8625 Acked Flush 0 Done Sent Done false
30642 8626 Acked Write 1 Done Sent Done false
30643 8627 Acked Read 4 Done Sent Done false
30644 8628 Acked Read 3 Done Sent Done false
30645 8629 Acked Write 7 Done Sent Done false
30646 8630 Acked Read 9 Done Sent Done false
30647 8631 Acked Read 10 Done Sent Done false
30648 8632 Acked Write 5 Done Sent Done false
30649 8633 Acked Read 7 Done Sent Done false
30650 8634 Acked Read 1 Done Sent Done false
30651 8635 Acked Write 8 Done Sent Done false
30652 8636 Acked Write 6 Done Sent Done false
30653 8637 Acked Read 8 Done Sent Done false
30654 8638 Acked Write 1 Done Sent Done false
30655 8639 Acked Write 9 Done Sent Done false
30656 8640 Acked Read 4 Done Sent Done false
30657 8641 Acked Write 7 Done Sent Done false
30658 8642 Acked Write 9 Done Sent Done false
30659 8643 Acked Write 7 Done Sent Done false
30660 8644 Acked Write 3 Done Sent Done false
30661 8645 Acked Read 10 Done Sent Done false
30662 8646 Acked Read 7 Done Sent Done false
30663 8647 Acked Write 2 Done Sent Done false
30664 8648 Acked Read 1 Done Sent Done false
30665 8649 Acked Read 8 Done Sent Done false
30666 8650 Acked Write 10 Done Sent Done false
30667 8651 Acked Write 9 Done Sent Done false
30668 8652 Acked Write 3 Done Sent Done false
30669 8653 Acked Write 10 Done Sent Done false
30670 8654 Acked Write 10 Done Sent Done false
30671 8655 Acked Write 4 Done Sent Done false
30672 8656 Acked Write 9 Done Sent Done false
30673 8657 Acked Read 1 Done Sent Done false
30674 8658 Acked Read 6 Done Sent Done false
30675 8659 Acked Read 1 Done Sent Done false
30676 8660 Acked Write 7 Done Sent Done false
30677 8661 Acked Write 1 Done Sent Done false
30678 8662 Acked Read 2 Done Sent Done false
30679 8663 Acked Write 5 Done Sent Done false
30680 8664 Acked Read 10 Done Sent Done false
30681 8665 Acked Read 2 Done Sent Done false
30682 8666 Acked Write 8 Done Sent Done false
30683 8667 Acked Read 2 Done Sent Done false
30684 8668 Acked Read 10 Done Sent Done false
30685 8669 Acked Write 8 Done Sent Done false
30686 8670 Acked Read 1 Done Sent Done false
30687 8671 Acked Read 10 Done Sent Done false
30688 8672 Acked Read 4 Done Sent Done false
30689 8673 Acked Read 8 Done Sent Done false
30690 8674 Acked Read 6 Done Sent Done false
30691 8675 Acked Write 1 Done Sent Done false
30692 8676 Acked Flush 0 Done Sent Done false
30693 8677 Acked Read 9 Done Sent Done false
30694 8678 Acked Write 3 Done Sent Done false
30695 8679 Acked Read 7 Done Sent Done false
30696 8680 Acked Write 10 Done Sent Done false
30697 8681 Acked Read 2 Done Sent Done false
30698 8682 Acked Read 8 Done Sent Done false
30699 8683 Acked Read 6 Done Sent Done false
30700 8684 Acked Write 9 Done Sent Done false
30701 8685 Acked Write 2 Done Sent Done false
30702 8686 Acked Read 7 Done Sent Done false
30703 8687 Acked Read 7 Done Sent Done false
30704 8688 Acked Read 6 Done Sent Done false
30705 8689 Acked Read 4 Done Sent Done false
30706 8690 Acked Write 9 Done Sent Done false
30707 8691 Acked Write 1 Done Sent Done false
30708 8692 Acked Read 8 Done Sent Done false
30709 8693 Acked Read 10 Done Sent Done false
30710 8694 Acked Write 6 Done Sent Done false
30711 8695 Acked Flush 0 Done Sent Done false
30712 8696 Acked Read 7 Done Sent Done false
30713 8697 Acked Write 5 Done Sent Done false
30714 8698 Acked Write 2 Done Sent Done false
30715 8699 Acked Read 2 Done Sent Done false
30716 8700 Acked Write 9 Done Sent Done false
30717 8701 Acked Write 4 Done Sent Done false
30718 8702 Acked Write 6 Done Sent Done false
30719 8703 Acked Read 3 Done Sent Done false
30720 8704 Acked Write 4 Done Sent Done false
30721 8705 Acked Write 5 Done Sent Done false
30722 8706 Acked Read 1 Done Sent Done false
30723 8707 Acked Read 3 Done Sent Done false
30724 8708 Acked Write 4 Done Sent Done false
30725 8709 Acked Write 8 Done Sent Done false
30726 8710 Acked Read 4 Done Sent Done false
30727 8711 Acked Read 9 Done Sent Done false
30728 8712 Acked Write 7 Done Sent Done false
30729 8713 Acked Read 4 Done Sent Done false
30730 8714 Acked Write 7 Done Sent Done false
30731 8715 Acked Flush 0 Done Sent Done false
30732 8716 Acked Write 10 Done Sent Done false
30733 8717 Acked Read 2 Done Sent Done false
30734 8718 Acked Read 5 Done Sent Done false
30735 8719 Acked Read 10 Done Sent Done false
30736 8720 Acked Write 9 Done Sent Done false
30737 8721 Acked Flush 0 Done Sent Done false
30738 8722 Acked Write 10 Done Sent Done false
30739 8723 Acked Write 8 Done Sent Done false
30740 8724 Acked Write 5 Done Sent Done false
30741 8725 Acked Read 8 Done Sent Done false
30742 8726 Acked Write 9 Done Sent Done false
30743 8727 Acked Write 5 Done Sent Done false
30744 8728 Acked Read 6 Done Sent Done false
30745 8729 Acked Write 8 Done Sent Done false
30746 8730 Acked Write 5 Done Sent Done false
30747 8731 Acked Write 6 Done Sent Done false
30748 8732 Acked Read 10 Done Sent Done false
30749 8733 Acked Read 10 Done Sent Done false
30750 8734 Acked Read 8 Done Sent Done false
30751 8735 Acked Write 2 Done Sent Done false
30752 8736 Acked Read 8 Done Sent Done false
30753 8737 Acked Read 2 Done Sent Done false
30754 8738 Acked Flush 0 Done Sent Done false
30755 8739 Acked Read 6 Done Sent Done false
30756 8740 Acked Write 5 Done Sent Done false
30757 8741 Acked Write 3 Done Sent Done false
30758 8742 Acked Read 4 Done Sent Done false
30759 8743 Acked Write 4 Done Sent Done false
30760 8744 Acked Read 2 Done Sent Done false
30761 8745 Acked Write 4 Done Sent Done false
30762 8746 Acked Read 9 Done Sent Done false
30763 8747 Acked Read 4 Done Sent Done false
30764 8748 Acked Read 3 Done Sent Done false
30765 8749 Acked Read 4 Done Sent Done false
30766 8750 Acked Write 3 Done Sent Done false
30767 8751 Acked Read 3 Done Sent Done false
30768 8752 Acked Write 3 Done Sent Done false
30769 8753 Acked Read 3 Done Sent Done false
30770 8754 Acked Write 7 Done Sent Done false
30771 8755 Acked Read 5 Done Sent Done false
30772 8756 Acked Write 2 Done Sent Done false
30773 8757 Acked Write 4 Done Sent Done false
30774 8758 Acked Read 8 Done Sent Done false
30775 8759 Acked Write 6 Done Sent Done false
30776 8760 Acked Read 10 Done Sent Done false
30777 8761 Acked Read 3 Done Sent Done false
30778 8762 Acked Write 3 Done Sent Done false
30779 8763 Acked Read 5 Done Sent Done false
30780 8764 Acked Read 4 Done Sent Done false
30781 8765 Acked Write 4 Done Sent Done false
30782 8766 Acked Write 5 Done Sent Done false
30783 8767 Acked Write 1 Done Sent Done false
30784 8768 Acked Read 3 Done Sent Done false
30785 8769 Acked Write 4 Done Sent Done false
30786 8770 Acked Read 10 Done Sent Done false
30787 8771 Acked Read 6 Done Sent Done false
30788 8772 Acked Flush 0 Done Sent Done false
30789 8773 Acked Read 2 Done Sent Done false
30790 8774 Acked Read 9 Done Sent Done false
30791 8775 Acked Flush 0 Done Sent Done false
30792 8776 Acked Read 9 Done Sent Done false
30793 8777 Acked Read 8 Done Sent Done false
30794 8778 Acked Read 3 Done Sent Done false
30795 8779 Acked Read 4 Done Sent Done false
30796 8780 Acked Write 5 Done Sent Done false
30797 8781 Acked Write 8 Done Sent Done false
30798 8782 Acked Read 1 Done Sent Done false
30799 8783 Acked Read 6 Done Sent Done false
30800 8784 Acked Write 6 Done Sent Done false
30801 8785 Acked Write 5 Done Sent Done false
30802 8786 Acked Write 1 Done Sent Done false
30803 8787 Acked Write 8 Done Sent Done false
30804 8788 Acked Read 8 Done Sent Done false
30805 8789 Acked Write 6 Done Sent Done false
30806 8790 Acked Write 1 Done Sent Done false
30807 8791 Acked Read 5 Done Sent Done false
30808 8792 Acked Write 5 Done Sent Done false
30809 8793 Acked Write 3 Done Sent Done false
30810 8794 Acked Read 5 Done Sent Done false
30811 8795 Acked Write 10 Done Sent Done false
30812 8796 Acked Read 1 Done Sent Done false
30813 8797 Acked Read 9 Done Sent Done false
30814 8798 Acked Read 5 Done Sent Done false
30815 8799 Acked Write 5 Done Sent Done false
30816 8800 Acked Write 10 Done Sent Done false
30817 8801 Acked Write 5 Done Sent Done false
30818 8802 Acked Write 7 Done Sent Done false
30819 8803 Acked Write 2 Done Sent Done false
30820 8804 Acked Write 3 Done Sent Done false
30821 8805 Acked Write 10 Done Sent Done false
30822 8806 Acked Write 7 Done Sent Done false
30823 8807 Acked Write 2 Done Sent Done false
30824 8808 Acked Read 4 Done Sent Done false
30825 8809 Acked Write 4 Done Sent Done false
30826 8810 Acked Flush 0 Done Sent Done false
30827 8811 Acked Flush 0 Done Sent Done false
30828 8812 Acked Read 9 Done Sent Done false
30829 8813 Acked Flush 0 Done Sent Done false
30830 8814 Acked Read 6 Done Sent Done false
30831 8815 Acked Write 3 Done Sent Done false
30832 8816 Acked Write 2 Done Sent Done false
30833 8817 Acked Read 10 Done Sent Done false
30834 8818 Acked Flush 0 Done Sent Done false
30835 8819 Acked Write 6 Done Sent Done false
30836 8820 Acked Read 5 Done Sent Done false
30837 8821 Acked Read 9 Done Sent Done false
30838 8822 Acked Read 2 Done Sent Done false
30839 8823 Acked Read 3 Done Sent Done false
30840 8824 Acked Read 6 Done Sent Done false
30841 8825 Acked Read 7 Done Sent Done false
30842 8826 Acked Read 5 Done Sent Done false
30843 8827 Acked Read 3 Done Sent Done false
30844 8828 Acked Write 3 Done Sent Done false
30845 8829 Acked Read 9 Done Sent Done false
30846 8830 Acked Read 3 Done Sent Done false
30847 8831 Acked Write 6 Done Sent Done false
30848 8832 Acked Read 6 Done Sent Done false
30849 8833 Acked Write 2 Done Sent Done false
30850 8834 Acked Read 6 Done Sent Done false
30851 8835 Acked Read 7 Done Sent Done false
30852 8836 Acked Read 2 Done Sent Done false
30853 8837 Acked Read 8 Done Sent Done false
30854 8838 Acked Write 10 Done Sent Done false
30855 8839 Acked Read 2 Done Sent Done false
30856 8840 Acked Flush 0 Done Sent Done false
30857 8841 Acked Write 6 Done Sent Done false
30858 8842 Acked Flush 0 Done Sent Done false
30859 8843 Acked Read 9 Done Sent Done false
30860 8844 Acked Write 7 Done Sent Done false
30861 8845 Acked Write 8 Done Sent Done false
30862 8846 Acked Write 7 Done Sent Done false
30863 8847 Acked Flush 0 Done Sent Done false
30864 8848 Acked Read 3 Done Sent Done false
30865 STATES DS:0 DS:1 DS:2 TOTAL
30866 Sent 0 301 0 301
30867 Done 301 0 301 602
30868 Skipped 0 0 0 0
30869 Error 0 0 0 0
30870 Last Flush: Some(JobId(8847)) Some(JobId(8547)) Some(JobId(8847))
30871 Downstairs last five completed: 8547 8546 8545 8544 8543
30872 Upstairs last five completed: 8848 8847 8846 8845 8844
30873 CLIENT: Up:0 ds:301 act:2
308742025-03-17T22:23:10.585ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
308752025-03-17T22:23:11.331ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
308762025-03-17T22:23:11.592ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
308772025-03-17T22:23:11.592ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
308782025-03-17T22:23:11.592ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
308792025-03-17T22:23:11.592ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
308802025-03-17T22:23:11.592ZINFOcrucible: client task is exiting = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
30881 [8820] got start action
30882 [8820] I am going to restart
30883 Starting downstairs at port 8820
30884 Make output file at "/tmp/dsc/downstairs-8820.txt"
30885 [8820][1] reports Starting
30886 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1232)
30887 [8820][1] reports Running
308882025-03-17T22:23:12.248ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
308892025-03-17T22:23:13.249ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
30890 ----------------------------------------------------------------
30891 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
30892 States: Active Offline Active
30893 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
30894 8548 Acked Read 10 Done Sent Done false
30895 8549 Acked Write 3 Done Sent Done false
30896 8550 Acked Write 1 Done Sent Done false
30897 8551 Acked Write 2 Done Sent Done false
30898 8552 Acked Write 6 Done Sent Done false
30899 8553 Acked Write 3 Done Sent Done false
30900 8554 Acked Read 10 Done Sent Done false
30901 8555 Acked Write 9 Done Sent Done false
30902 8556 Acked Write 6 Done Sent Done false
30903 8557 Acked Write 7 Done Sent Done false
30904 8558 Acked Read 6 Done Sent Done false
30905 8559 Acked Read 7 Done Sent Done false
30906 8560 Acked Flush 0 Done Sent Done false
30907 8561 Acked Read 8 Done Sent Done false
30908 8562 Acked Read 7 Done Sent Done false
30909 8563 Acked Write 6 Done Sent Done false
30910 8564 Acked Read 7 Done Sent Done false
30911 8565 Acked Write 9 Done Sent Done false
30912 8566 Acked Write 4 Done Sent Done false
30913 8567 Acked Read 3 Done Sent Done false
30914 8568 Acked Flush 0 Done Sent Done false
30915 8569 Acked Write 5 Done Sent Done false
30916 8570 Acked Read 7 Done Sent Done false
30917 8571 Acked Read 6 Done Sent Done false
30918 8572 Acked Read 2 Done Sent Done false
30919 8573 Acked Read 5 Done Sent Done false
30920 8574 Acked Write 3 Done Sent Done false
30921 8575 Acked Write 3 Done Sent Done false
30922 8576 Acked Read 6 Done Sent Done false
30923 8577 Acked Write 7 Done Sent Done false
30924 8578 Acked Read 10 Done Sent Done false
30925 8579 Acked Write 1 Done Sent Done false
30926 8580 Acked Write 6 Done Sent Done false
30927 8581 Acked Write 10 Done Sent Done false
30928 8582 Acked Read 1 Done Sent Done false
30929 8583 Acked Read 1 Done Sent Done false
30930 8584 Acked Write 9 Done Sent Done false
30931 8585 Acked Read 5 Done Sent Done false
30932 8586 Acked Flush 0 Done Sent Done false
30933 8587 Acked Read 1 Done Sent Done false
30934 8588 Acked Read 2 Done Sent Done false
30935 8589 Acked Read 8 Done Sent Done false
30936 8590 Acked Write 6 Done Sent Done false
30937 8591 Acked Read 3 Done Sent Done false
30938 8592 Acked Write 5 Done Sent Done false
30939 8593 Acked Write 9 Done Sent Done false
30940 8594 Acked Write 8 Done Sent Done false
30941 8595 Acked Read 8 Done Sent Done false
30942 8596 Acked Read 3 Done Sent Done false
30943 8597 Acked Read 8 Done Sent Done false
30944 8598 Acked Write 2 Done Sent Done false
30945 8599 Acked Write 3 Done Sent Done false
30946 8600 Acked Flush 0 Done Sent Done false
30947 8601 Acked Read 10 Done Sent Done false
30948 8602 Acked Write 3 Done Sent Done false
30949 8603 Acked Read 2 Done Sent Done false
30950 8604 Acked Flush 0 Done Sent Done false
30951 8605 Acked Write 1 Done Sent Done false
30952 8606 Acked Write 6 Done Sent Done false
30953 8607 Acked Write 3 Done Sent Done false
30954 8608 Acked Write 5 Done Sent Done false
30955 8609 Acked Read 3 Done Sent Done false
30956 8610 Acked Read 8 Done Sent Done false
30957 8611 Acked Flush 0 Done Sent Done false
30958 8612 Acked Flush 0 Done Sent Done false
30959 8613 Acked Read 4 Done Sent Done false
30960 8614 Acked Read 3 Done Sent Done false
30961 8615 Acked Read 4 Done Sent Done false
30962 8616 Acked Write 7 Done Sent Done false
30963 8617 Acked Read 6 Done Sent Done false
30964 8618 Acked Flush 0 Done Sent Done false
30965 8619 Acked Read 6 Done Sent Done false
30966 8620 Acked Read 9 Done Sent Done false
30967 8621 Acked Write 7 Done Sent Done false
30968 8622 Acked Flush 0 Done Sent Done false
30969 8623 Acked Read 1 Done Sent Done false
30970 8624 Acked Read 6 Done Sent Done false
30971 8625 Acked Flush 0 Done Sent Done false
30972 8626 Acked Write 1 Done Sent Done false
30973 8627 Acked Read 4 Done Sent Done false
30974 8628 Acked Read 3 Done Sent Done false
30975 8629 Acked Write 7 Done Sent Done false
30976 8630 Acked Read 9 Done Sent Done false
30977 8631 Acked Read 10 Done Sent Done false
30978 8632 Acked Write 5 Done Sent Done false
30979 8633 Acked Read 7 Done Sent Done false
30980 8634 Acked Read 1 Done Sent Done false
30981 8635 Acked Write 8 Done Sent Done false
30982 8636 Acked Write 6 Done Sent Done false
30983 8637 Acked Read 8 Done Sent Done false
30984 8638 Acked Write 1 Done Sent Done false
30985 8639 Acked Write 9 Done Sent Done false
30986 8640 Acked Read 4 Done Sent Done false
30987 8641 Acked Write 7 Done Sent Done false
30988 8642 Acked Write 9 Done Sent Done false
30989 8643 Acked Write 7 Done Sent Done false
30990 8644 Acked Write 3 Done Sent Done false
30991 8645 Acked Read 10 Done Sent Done false
30992 8646 Acked Read 7 Done Sent Done false
30993 8647 Acked Write 2 Done Sent Done false
30994 8648 Acked Read 1 Done Sent Done false
30995 8649 Acked Read 8 Done Sent Done false
30996 8650 Acked Write 10 Done Sent Done false
30997 8651 Acked Write 9 Done Sent Done false
30998 8652 Acked Write 3 Done Sent Done false
30999 8653 Acked Write 10 Done Sent Done false
31000 8654 Acked Write 10 Done Sent Done false
31001 8655 Acked Write 4 Done Sent Done false
31002 8656 Acked Write 9 Done Sent Done false
31003 8657 Acked Read 1 Done Sent Done false
31004 8658 Acked Read 6 Done Sent Done false
31005 8659 Acked Read 1 Done Sent Done false
31006 8660 Acked Write 7 Done Sent Done false
31007 8661 Acked Write 1 Done Sent Done false
31008 8662 Acked Read 2 Done Sent Done false
31009 8663 Acked Write 5 Done Sent Done false
31010 8664 Acked Read 10 Done Sent Done false
31011 8665 Acked Read 2 Done Sent Done false
31012 8666 Acked Write 8 Done Sent Done false
31013 8667 Acked Read 2 Done Sent Done false
31014 8668 Acked Read 10 Done Sent Done false
31015 8669 Acked Write 8 Done Sent Done false
31016 8670 Acked Read 1 Done Sent Done false
31017 8671 Acked Read 10 Done Sent Done false
31018 8672 Acked Read 4 Done Sent Done false
31019 8673 Acked Read 8 Done Sent Done false
31020 8674 Acked Read 6 Done Sent Done false
31021 8675 Acked Write 1 Done Sent Done false
31022 8676 Acked Flush 0 Done Sent Done false
31023 8677 Acked Read 9 Done Sent Done false
31024 8678 Acked Write 3 Done Sent Done false
31025 8679 Acked Read 7 Done Sent Done false
31026 8680 Acked Write 10 Done Sent Done false
31027 8681 Acked Read 2 Done Sent Done false
31028 8682 Acked Read 8 Done Sent Done false
31029 8683 Acked Read 6 Done Sent Done false
31030 8684 Acked Write 9 Done Sent Done false
31031 8685 Acked Write 2 Done Sent Done false
31032 8686 Acked Read 7 Done Sent Done false
31033 8687 Acked Read 7 Done Sent Done false
31034 8688 Acked Read 6 Done Sent Done false
31035 8689 Acked Read 4 Done Sent Done false
31036 8690 Acked Write 9 Done Sent Done false
31037 8691 Acked Write 1 Done Sent Done false
31038 8692 Acked Read 8 Done Sent Done false
31039 8693 Acked Read 10 Done Sent Done false
31040 8694 Acked Write 6 Done Sent Done false
31041 8695 Acked Flush 0 Done Sent Done false
31042 8696 Acked Read 7 Done Sent Done false
31043 8697 Acked Write 5 Done Sent Done false
31044 8698 Acked Write 2 Done Sent Done false
31045 8699 Acked Read 2 Done Sent Done false
31046 8700 Acked Write 9 Done Sent Done false
31047 8701 Acked Write 4 Done Sent Done false
31048 8702 Acked Write 6 Done Sent Done false
31049 8703 Acked Read 3 Done Sent Done false
31050 8704 Acked Write 4 Done Sent Done false
31051 8705 Acked Write 5 Done Sent Done false
31052 8706 Acked Read 1 Done Sent Done false
31053 8707 Acked Read 3 Done Sent Done false
31054 8708 Acked Write 4 Done Sent Done false
31055 8709 Acked Write 8 Done Sent Done false
31056 8710 Acked Read 4 Done Sent Done false
31057 8711 Acked Read 9 Done Sent Done false
31058 8712 Acked Write 7 Done Sent Done false
31059 8713 Acked Read 4 Done Sent Done false
31060 8714 Acked Write 7 Done Sent Done false
31061 8715 Acked Flush 0 Done Sent Done false
31062 8716 Acked Write 10 Done Sent Done false
31063 8717 Acked Read 2 Done Sent Done false
31064 8718 Acked Read 5 Done Sent Done false
31065 8719 Acked Read 10 Done Sent Done false
31066 8720 Acked Write 9 Done Sent Done false
31067 8721 Acked Flush 0 Done Sent Done false
31068 8722 Acked Write 10 Done Sent Done false
31069 8723 Acked Write 8 Done Sent Done false
31070 8724 Acked Write 5 Done Sent Done false
31071 8725 Acked Read 8 Done Sent Done false
31072 8726 Acked Write 9 Done Sent Done false
31073 8727 Acked Write 5 Done Sent Done false
31074 8728 Acked Read 6 Done Sent Done false
31075 8729 Acked Write 8 Done Sent Done false
31076 8730 Acked Write 5 Done Sent Done false
31077 8731 Acked Write 6 Done Sent Done false
31078 8732 Acked Read 10 Done Sent Done false
31079 8733 Acked Read 10 Done Sent Done false
31080 8734 Acked Read 8 Done Sent Done false
31081 8735 Acked Write 2 Done Sent Done false
31082 8736 Acked Read 8 Done Sent Done false
31083 8737 Acked Read 2 Done Sent Done false
31084 8738 Acked Flush 0 Done Sent Done false
31085 8739 Acked Read 6 Done Sent Done false
31086 8740 Acked Write 5 Done Sent Done false
31087 8741 Acked Write 3 Done Sent Done false
31088 8742 Acked Read 4 Done Sent Done false
31089 8743 Acked Write 4 Done Sent Done false
31090 8744 Acked Read 2 Done Sent Done false
31091 8745 Acked Write 4 Done Sent Done false
31092 8746 Acked Read 9 Done Sent Done false
31093 8747 Acked Read 4 Done Sent Done false
31094 8748 Acked Read 3 Done Sent Done false
31095 8749 Acked Read 4 Done Sent Done false
31096 8750 Acked Write 3 Done Sent Done false
31097 8751 Acked Read 3 Done Sent Done false
31098 8752 Acked Write 3 Done Sent Done false
31099 8753 Acked Read 3 Done Sent Done false
31100 8754 Acked Write 7 Done Sent Done false
31101 8755 Acked Read 5 Done Sent Done false
31102 8756 Acked Write 2 Done Sent Done false
31103 8757 Acked Write 4 Done Sent Done false
31104 8758 Acked Read 8 Done Sent Done false
31105 8759 Acked Write 6 Done Sent Done false
31106 8760 Acked Read 10 Done Sent Done false
31107 8761 Acked Read 3 Done Sent Done false
31108 8762 Acked Write 3 Done Sent Done false
31109 8763 Acked Read 5 Done Sent Done false
31110 8764 Acked Read 4 Done Sent Done false
31111 8765 Acked Write 4 Done Sent Done false
31112 8766 Acked Write 5 Done Sent Done false
31113 8767 Acked Write 1 Done Sent Done false
31114 8768 Acked Read 3 Done Sent Done false
31115 8769 Acked Write 4 Done Sent Done false
31116 8770 Acked Read 10 Done Sent Done false
31117 8771 Acked Read 6 Done Sent Done false
31118 8772 Acked Flush 0 Done Sent Done false
31119 8773 Acked Read 2 Done Sent Done false
31120 8774 Acked Read 9 Done Sent Done false
31121 8775 Acked Flush 0 Done Sent Done false
31122 8776 Acked Read 9 Done Sent Done false
31123 8777 Acked Read 8 Done Sent Done false
31124 8778 Acked Read 3 Done Sent Done false
31125 8779 Acked Read 4 Done Sent Done false
31126 8780 Acked Write 5 Done Sent Done false
31127 8781 Acked Write 8 Done Sent Done false
31128 8782 Acked Read 1 Done Sent Done false
31129 8783 Acked Read 6 Done Sent Done false
31130 8784 Acked Write 6 Done Sent Done false
31131 8785 Acked Write 5 Done Sent Done false
31132 8786 Acked Write 1 Done Sent Done false
31133 8787 Acked Write 8 Done Sent Done false
31134 8788 Acked Read 8 Done Sent Done false
31135 8789 Acked Write 6 Done Sent Done false
31136 8790 Acked Write 1 Done Sent Done false
31137 8791 Acked Read 5 Done Sent Done false
31138 8792 Acked Write 5 Done Sent Done false
31139 8793 Acked Write 3 Done Sent Done false
31140 8794 Acked Read 5 Done Sent Done false
31141 8795 Acked Write 10 Done Sent Done false
31142 8796 Acked Read 1 Done Sent Done false
31143 8797 Acked Read 9 Done Sent Done false
31144 8798 Acked Read 5 Done Sent Done false
31145 8799 Acked Write 5 Done Sent Done false
31146 8800 Acked Write 10 Done Sent Done false
31147 8801 Acked Write 5 Done Sent Done false
31148 8802 Acked Write 7 Done Sent Done false
31149 8803 Acked Write 2 Done Sent Done false
31150 8804 Acked Write 3 Done Sent Done false
31151 8805 Acked Write 10 Done Sent Done false
31152 8806 Acked Write 7 Done Sent Done false
31153 8807 Acked Write 2 Done Sent Done false
31154 8808 Acked Read 4 Done Sent Done false
31155 8809 Acked Write 4 Done Sent Done false
31156 8810 Acked Flush 0 Done Sent Done false
31157 8811 Acked Flush 0 Done Sent Done false
31158 8812 Acked Read 9 Done Sent Done false
31159 8813 Acked Flush 0 Done Sent Done false
31160 8814 Acked Read 6 Done Sent Done false
31161 8815 Acked Write 3 Done Sent Done false
31162 8816 Acked Write 2 Done Sent Done false
31163 8817 Acked Read 10 Done Sent Done false
31164 8818 Acked Flush 0 Done Sent Done false
31165 8819 Acked Write 6 Done Sent Done false
31166 8820 Acked Read 5 Done Sent Done false
31167 8821 Acked Read 9 Done Sent Done false
31168 8822 Acked Read 2 Done Sent Done false
31169 8823 Acked Read 3 Done Sent Done false
31170 8824 Acked Read 6 Done Sent Done false
31171 8825 Acked Read 7 Done Sent Done false
31172 8826 Acked Read 5 Done Sent Done false
31173 8827 Acked Read 3 Done Sent Done false
31174 8828 Acked Write 3 Done Sent Done false
31175 8829 Acked Read 9 Done Sent Done false
31176 8830 Acked Read 3 Done Sent Done false
31177 8831 Acked Write 6 Done Sent Done false
31178 8832 Acked Read 6 Done Sent Done false
31179 8833 Acked Write 2 Done Sent Done false
31180 8834 Acked Read 6 Done Sent Done false
31181 8835 Acked Read 7 Done Sent Done false
31182 8836 Acked Read 2 Done Sent Done false
31183 8837 Acked Read 8 Done Sent Done false
31184 8838 Acked Write 10 Done Sent Done false
31185 8839 Acked Read 2 Done Sent Done false
31186 8840 Acked Flush 0 Done Sent Done false
31187 8841 Acked Write 6 Done Sent Done false
31188 8842 Acked Flush 0 Done Sent Done false
31189 8843 Acked Read 9 Done Sent Done false
31190 8844 Acked Write 7 Done Sent Done false
31191 8845 Acked Write 8 Done Sent Done false
31192 8846 Acked Write 7 Done Sent Done false
31193 8847 Acked Flush 0 Done Sent Done false
31194 8848 Acked Read 3 Done Sent Done false
31195 STATES DS:0 DS:1 DS:2 TOTAL
31196 Sent 0 301 0 301
31197 Done 301 0 301 602
31198 Skipped 0 0 0 0
31199 Error 0 0 0 0
31200 Last Flush: Some(JobId(8847)) Some(JobId(8547)) Some(JobId(8847))
31201 Downstairs last five completed: 8547 8546 8545 8544 8543
31202 Upstairs last five completed: 8848 8847 8846 8845 8844
31203 CLIENT: Up:0 ds:301 act:2
312042025-03-17T22:23:14.344ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
312052025-03-17T22:23:15.587ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
312062025-03-17T22:23:16.332ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
312072025-03-17T22:23:17.257ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
31208 ----------------------------------------------------------------
31209 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
31210 States: Active Offline Active
31211 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31212 8548 Acked Read 10 Done Sent Done false
31213 8549 Acked Write 3 Done Sent Done false
31214 8550 Acked Write 1 Done Sent Done false
31215 8551 Acked Write 2 Done Sent Done false
31216 8552 Acked Write 6 Done Sent Done false
31217 8553 Acked Write 3 Done Sent Done false
31218 8554 Acked Read 10 Done Sent Done false
31219 8555 Acked Write 9 Done Sent Done false
31220 8556 Acked Write 6 Done Sent Done false
31221 8557 Acked Write 7 Done Sent Done false
31222 8558 Acked Read 6 Done Sent Done false
31223 8559 Acked Read 7 Done Sent Done false
31224 8560 Acked Flush 0 Done Sent Done false
31225 8561 Acked Read 8 Done Sent Done false
31226 8562 Acked Read 7 Done Sent Done false
31227 8563 Acked Write 6 Done Sent Done false
31228 8564 Acked Read 7 Done Sent Done false
31229 8565 Acked Write 9 Done Sent Done false
31230 8566 Acked Write 4 Done Sent Done false
31231 8567 Acked Read 3 Done Sent Done false
31232 8568 Acked Flush 0 Done Sent Done false
31233 8569 Acked Write 5 Done Sent Done false
31234 8570 Acked Read 7 Done Sent Done false
31235 8571 Acked Read 6 Done Sent Done false
31236 8572 Acked Read 2 Done Sent Done false
31237 8573 Acked Read 5 Done Sent Done false
31238 8574 Acked Write 3 Done Sent Done false
31239 8575 Acked Write 3 Done Sent Done false
31240 8576 Acked Read 6 Done Sent Done false
31241 8577 Acked Write 7 Done Sent Done false
31242 8578 Acked Read 10 Done Sent Done false
31243 8579 Acked Write 1 Done Sent Done false
31244 8580 Acked Write 6 Done Sent Done false
31245 8581 Acked Write 10 Done Sent Done false
31246 8582 Acked Read 1 Done Sent Done false
31247 8583 Acked Read 1 Done Sent Done false
31248 8584 Acked Write 9 Done Sent Done false
31249 8585 Acked Read 5 Done Sent Done false
31250 8586 Acked Flush 0 Done Sent Done false
31251 8587 Acked Read 1 Done Sent Done false
31252 8588 Acked Read 2 Done Sent Done false
31253 8589 Acked Read 8 Done Sent Done false
31254 8590 Acked Write 6 Done Sent Done false
31255 8591 Acked Read 3 Done Sent Done false
31256 8592 Acked Write 5 Done Sent Done false
31257 8593 Acked Write 9 Done Sent Done false
31258 8594 Acked Write 8 Done Sent Done false
31259 8595 Acked Read 8 Done Sent Done false
31260 8596 Acked Read 3 Done Sent Done false
31261 8597 Acked Read 8 Done Sent Done false
31262 8598 Acked Write 2 Done Sent Done false
31263 8599 Acked Write 3 Done Sent Done false
31264 8600 Acked Flush 0 Done Sent Done false
31265 8601 Acked Read 10 Done Sent Done false
31266 8602 Acked Write 3 Done Sent Done false
31267 8603 Acked Read 2 Done Sent Done false
31268 8604 Acked Flush 0 Done Sent Done false
31269 8605 Acked Write 1 Done Sent Done false
31270 8606 Acked Write 6 Done Sent Done false
31271 8607 Acked Write 3 Done Sent Done false
31272 8608 Acked Write 5 Done Sent Done false
31273 8609 Acked Read 3 Done Sent Done false
31274 8610 Acked Read 8 Done Sent Done false
31275 8611 Acked Flush 0 Done Sent Done false
31276 8612 Acked Flush 0 Done Sent Done false
31277 8613 Acked Read 4 Done Sent Done false
31278 8614 Acked Read 3 Done Sent Done false
31279 8615 Acked Read 4 Done Sent Done false
31280 8616 Acked Write 7 Done Sent Done false
31281 8617 Acked Read 6 Done Sent Done false
31282 8618 Acked Flush 0 Done Sent Done false
31283 8619 Acked Read 6 Done Sent Done false
31284 8620 Acked Read 9 Done Sent Done false
31285 8621 Acked Write 7 Done Sent Done false
31286 8622 Acked Flush 0 Done Sent Done false
31287 8623 Acked Read 1 Done Sent Done false
31288 8624 Acked Read 6 Done Sent Done false
31289 8625 Acked Flush 0 Done Sent Done false
31290 8626 Acked Write 1 Done Sent Done false
31291 8627 Acked Read 4 Done Sent Done false
31292 8628 Acked Read 3 Done Sent Done false
31293 8629 Acked Write 7 Done Sent Done false
31294 8630 Acked Read 9 Done Sent Done false
31295 8631 Acked Read 10 Done Sent Done false
31296 8632 Acked Write 5 Done Sent Done false
31297 8633 Acked Read 7 Done Sent Done false
31298 8634 Acked Read 1 Done Sent Done false
31299 8635 Acked Write 8 Done Sent Done false
31300 8636 Acked Write 6 Done Sent Done false
31301 8637 Acked Read 8 Done Sent Done false
31302 8638 Acked Write 1 Done Sent Done false
31303 8639 Acked Write 9 Done Sent Done false
31304 8640 Acked Read 4 Done Sent Done false
31305 8641 Acked Write 7 Done Sent Done false
31306 8642 Acked Write 9 Done Sent Done false
31307 8643 Acked Write 7 Done Sent Done false
31308 8644 Acked Write 3 Done Sent Done false
31309 8645 Acked Read 10 Done Sent Done false
31310 8646 Acked Read 7 Done Sent Done false
31311 8647 Acked Write 2 Done Sent Done false
31312 8648 Acked Read 1 Done Sent Done false
31313 8649 Acked Read 8 Done Sent Done false
31314 8650 Acked Write 10 Done Sent Done false
31315 8651 Acked Write 9 Done Sent Done false
31316 8652 Acked Write 3 Done Sent Done false
31317 8653 Acked Write 10 Done Sent Done false
31318 8654 Acked Write 10 Done Sent Done false
31319 8655 Acked Write 4 Done Sent Done false
31320 8656 Acked Write 9 Done Sent Done false
31321 8657 Acked Read 1 Done Sent Done false
31322 8658 Acked Read 6 Done Sent Done false
31323 8659 Acked Read 1 Done Sent Done false
31324 8660 Acked Write 7 Done Sent Done false
31325 8661 Acked Write 1 Done Sent Done false
31326 8662 Acked Read 2 Done Sent Done false
31327 8663 Acked Write 5 Done Sent Done false
31328 8664 Acked Read 10 Done Sent Done false
31329 8665 Acked Read 2 Done Sent Done false
31330 8666 Acked Write 8 Done Sent Done false
31331 8667 Acked Read 2 Done Sent Done false
31332 8668 Acked Read 10 Done Sent Done false
31333 8669 Acked Write 8 Done Sent Done false
31334 8670 Acked Read 1 Done Sent Done false
31335 8671 Acked Read 10 Done Sent Done false
31336 8672 Acked Read 4 Done Sent Done false
31337 8673 Acked Read 8 Done Sent Done false
31338 8674 Acked Read 6 Done Sent Done false
31339 8675 Acked Write 1 Done Sent Done false
31340 8676 Acked Flush 0 Done Sent Done false
31341 8677 Acked Read 9 Done Sent Done false
31342 8678 Acked Write 3 Done Sent Done false
31343 8679 Acked Read 7 Done Sent Done false
31344 8680 Acked Write 10 Done Sent Done false
31345 8681 Acked Read 2 Done Sent Done false
31346 8682 Acked Read 8 Done Sent Done false
31347 8683 Acked Read 6 Done Sent Done false
31348 8684 Acked Write 9 Done Sent Done false
31349 8685 Acked Write 2 Done Sent Done false
31350 8686 Acked Read 7 Done Sent Done false
31351 8687 Acked Read 7 Done Sent Done false
31352 8688 Acked Read 6 Done Sent Done false
31353 8689 Acked Read 4 Done Sent Done false
31354 8690 Acked Write 9 Done Sent Done false
31355 8691 Acked Write 1 Done Sent Done false
31356 8692 Acked Read 8 Done Sent Done false
31357 8693 Acked Read 10 Done Sent Done false
31358 8694 Acked Write 6 Done Sent Done false
31359 8695 Acked Flush 0 Done Sent Done false
31360 8696 Acked Read 7 Done Sent Done false
31361 8697 Acked Write 5 Done Sent Done false
31362 8698 Acked Write 2 Done Sent Done false
31363 8699 Acked Read 2 Done Sent Done false
31364 8700 Acked Write 9 Done Sent Done false
31365 8701 Acked Write 4 Done Sent Done false
31366 8702 Acked Write 6 Done Sent Done false
31367 8703 Acked Read 3 Done Sent Done false
31368 8704 Acked Write 4 Done Sent Done false
31369 8705 Acked Write 5 Done Sent Done false
31370 8706 Acked Read 1 Done Sent Done false
31371 8707 Acked Read 3 Done Sent Done false
31372 8708 Acked Write 4 Done Sent Done false
31373 8709 Acked Write 8 Done Sent Done false
31374 8710 Acked Read 4 Done Sent Done false
31375 8711 Acked Read 9 Done Sent Done false
31376 8712 Acked Write 7 Done Sent Done false
31377 8713 Acked Read 4 Done Sent Done false
31378 8714 Acked Write 7 Done Sent Done false
31379 8715 Acked Flush 0 Done Sent Done false
31380 8716 Acked Write 10 Done Sent Done false
31381 8717 Acked Read 2 Done Sent Done false
31382 8718 Acked Read 5 Done Sent Done false
31383 8719 Acked Read 10 Done Sent Done false
31384 8720 Acked Write 9 Done Sent Done false
31385 8721 Acked Flush 0 Done Sent Done false
31386 8722 Acked Write 10 Done Sent Done false
31387 8723 Acked Write 8 Done Sent Done false
31388 8724 Acked Write 5 Done Sent Done false
31389 8725 Acked Read 8 Done Sent Done false
31390 8726 Acked Write 9 Done Sent Done false
31391 8727 Acked Write 5 Done Sent Done false
31392 8728 Acked Read 6 Done Sent Done false
31393 8729 Acked Write 8 Done Sent Done false
31394 8730 Acked Write 5 Done Sent Done false
31395 8731 Acked Write 6 Done Sent Done false
31396 8732 Acked Read 10 Done Sent Done false
31397 8733 Acked Read 10 Done Sent Done false
31398 8734 Acked Read 8 Done Sent Done false
31399 8735 Acked Write 2 Done Sent Done false
31400 8736 Acked Read 8 Done Sent Done false
31401 8737 Acked Read 2 Done Sent Done false
31402 8738 Acked Flush 0 Done Sent Done false
31403 8739 Acked Read 6 Done Sent Done false
31404 8740 Acked Write 5 Done Sent Done false
31405 8741 Acked Write 3 Done Sent Done false
31406 8742 Acked Read 4 Done Sent Done false
31407 8743 Acked Write 4 Done Sent Done false
31408 8744 Acked Read 2 Done Sent Done false
31409 8745 Acked Write 4 Done Sent Done false
31410 8746 Acked Read 9 Done Sent Done false
31411 8747 Acked Read 4 Done Sent Done false
31412 8748 Acked Read 3 Done Sent Done false
31413 8749 Acked Read 4 Done Sent Done false
31414 8750 Acked Write 3 Done Sent Done false
31415 8751 Acked Read 3 Done Sent Done false
31416 8752 Acked Write 3 Done Sent Done false
31417 8753 Acked Read 3 Done Sent Done false
31418 8754 Acked Write 7 Done Sent Done false
31419 8755 Acked Read 5 Done Sent Done false
31420 8756 Acked Write 2 Done Sent Done false
31421 8757 Acked Write 4 Done Sent Done false
31422 8758 Acked Read 8 Done Sent Done false
31423 8759 Acked Write 6 Done Sent Done false
31424 8760 Acked Read 10 Done Sent Done false
31425 8761 Acked Read 3 Done Sent Done false
31426 8762 Acked Write 3 Done Sent Done false
31427 8763 Acked Read 5 Done Sent Done false
31428 8764 Acked Read 4 Done Sent Done false
31429 8765 Acked Write 4 Done Sent Done false
31430 8766 Acked Write 5 Done Sent Done false
31431 8767 Acked Write 1 Done Sent Done false
31432 8768 Acked Read 3 Done Sent Done false
31433 8769 Acked Write 4 Done Sent Done false
31434 8770 Acked Read 10 Done Sent Done false
31435 8771 Acked Read 6 Done Sent Done false
31436 8772 Acked Flush 0 Done Sent Done false
31437 8773 Acked Read 2 Done Sent Done false
31438 8774 Acked Read 9 Done Sent Done false
31439 8775 Acked Flush 0 Done Sent Done false
31440 8776 Acked Read 9 Done Sent Done false
31441 8777 Acked Read 8 Done Sent Done false
31442 8778 Acked Read 3 Done Sent Done false
31443 8779 Acked Read 4 Done Sent Done false
31444 8780 Acked Write 5 Done Sent Done false
31445 8781 Acked Write 8 Done Sent Done false
31446 8782 Acked Read 1 Done Sent Done false
31447 8783 Acked Read 6 Done Sent Done false
31448 8784 Acked Write 6 Done Sent Done false
31449 8785 Acked Write 5 Done Sent Done false
31450 8786 Acked Write 1 Done Sent Done false
31451 8787 Acked Write 8 Done Sent Done false
31452 8788 Acked Read 8 Done Sent Done false
31453 8789 Acked Write 6 Done Sent Done false
31454 8790 Acked Write 1 Done Sent Done false
31455 8791 Acked Read 5 Done Sent Done false
31456 8792 Acked Write 5 Done Sent Done false
31457 8793 Acked Write 3 Done Sent Done false
31458 8794 Acked Read 5 Done Sent Done false
31459 8795 Acked Write 10 Done Sent Done false
31460 8796 Acked Read 1 Done Sent Done false
31461 8797 Acked Read 9 Done Sent Done false
31462 8798 Acked Read 5 Done Sent Done false
31463 8799 Acked Write 5 Done Sent Done false
31464 8800 Acked Write 10 Done Sent Done false
31465 8801 Acked Write 5 Done Sent Done false
31466 8802 Acked Write 7 Done Sent Done false
31467 8803 Acked Write 2 Done Sent Done false
31468 8804 Acked Write 3 Done Sent Done false
31469 8805 Acked Write 10 Done Sent Done false
31470 8806 Acked Write 7 Done Sent Done false
31471 8807 Acked Write 2 Done Sent Done false
31472 8808 Acked Read 4 Done Sent Done false
31473 8809 Acked Write 4 Done Sent Done false
31474 8810 Acked Flush 0 Done Sent Done false
31475 8811 Acked Flush 0 Done Sent Done false
31476 8812 Acked Read 9 Done Sent Done false
31477 8813 Acked Flush 0 Done Sent Done false
31478 8814 Acked Read 6 Done Sent Done false
31479 8815 Acked Write 3 Done Sent Done false
31480 8816 Acked Write 2 Done Sent Done false
31481 8817 Acked Read 10 Done Sent Done false
31482 8818 Acked Flush 0 Done Sent Done false
31483 8819 Acked Write 6 Done Sent Done false
31484 8820 Acked Read 5 Done Sent Done false
31485 8821 Acked Read 9 Done Sent Done false
31486 8822 Acked Read 2 Done Sent Done false
31487 8823 Acked Read 3 Done Sent Done false
31488 8824 Acked Read 6 Done Sent Done false
31489 8825 Acked Read 7 Done Sent Done false
31490 8826 Acked Read 5 Done Sent Done false
31491 8827 Acked Read 3 Done Sent Done false
31492 8828 Acked Write 3 Done Sent Done false
31493 8829 Acked Read 9 Done Sent Done false
31494 8830 Acked Read 3 Done Sent Done false
31495 8831 Acked Write 6 Done Sent Done false
31496 8832 Acked Read 6 Done Sent Done false
31497 8833 Acked Write 2 Done Sent Done false
31498 8834 Acked Read 6 Done Sent Done false
31499 8835 Acked Read 7 Done Sent Done false
31500 8836 Acked Read 2 Done Sent Done false
31501 8837 Acked Read 8 Done Sent Done false
31502 8838 Acked Write 10 Done Sent Done false
31503 8839 Acked Read 2 Done Sent Done false
31504 8840 Acked Flush 0 Done Sent Done false
31505 8841 Acked Write 6 Done Sent Done false
31506 8842 Acked Flush 0 Done Sent Done false
31507 8843 Acked Read 9 Done Sent Done false
31508 8844 Acked Write 7 Done Sent Done false
31509 8845 Acked Write 8 Done Sent Done false
31510 8846 Acked Write 7 Done Sent Done false
31511 8847 Acked Flush 0 Done Sent Done false
31512 8848 Acked Read 3 Done Sent Done false
31513 STATES DS:0 DS:1 DS:2 TOTAL
31514 Sent 0 301 0 301
31515 Done 301 0 301 602
31516 Skipped 0 0 0 0
31517 Error 0 0 0 0
31518 Last Flush: Some(JobId(8847)) Some(JobId(8547)) Some(JobId(8847))
31519 Downstairs last five completed: 8547 8546 8545 8544 8543
31520 Upstairs last five completed: 8848 8847 8846 8845 8844
31521 CLIENT: Up:0 ds:301 act:2
315222025-03-17T22:23:18.345ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
315232025-03-17T22:23:19.261ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
315242025-03-17T22:23:20.588ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
315252025-03-17T22:23:21.333ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
315262025-03-17T22:23:21.594ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
315272025-03-17T22:23:21.594ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:59005) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
315282025-03-17T22:23:21.595ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
315292025-03-17T22:23:21.595ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
315302025-03-17T22:23:21.595ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
315312025-03-17T22:23:21.595ZINFOcrucible: send last flush ID to this DS: Some(JobId(8547)) = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
315322025-03-17T22:23:21.595ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
315332025-03-17T22:23:21.596ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(8547)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
315342025-03-17T22:23:21.596ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(8547)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
31535 ----------------------------------------------------------------
31536 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:1
31537 States: Active Active Active
31538 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31539 8848 Acked Read 3 Done Done Done true
31540 STATES DS:0 DS:1 DS:2 TOTAL
31541 Sent 0 0 0 0
31542 Done 1 1 1 3
31543 Skipped 0 0 0 0
31544 Error 0 0 0 0
31545 Last Flush: Some(JobId(8847)) Some(JobId(8847)) Some(JobId(8847))
31546 Downstairs last five completed: 8847 8846 8845 8844 8843
31547 Upstairs last five completed: 8848 8847 8846 8845 8844
31548 CLIENT: Up:0 ds:1 act:3
31549 ----------------------------------------------------------------
31550 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
31551 Downstairs last five completed: 8849 8848 8847 8846 8845
31552 Upstairs last five completed: 8849 8848 8847 8846 8845
31553 CLIENT: Up:0 ds:0 act:3
31554 Replay: All jobs finished
31555 Main task has work to do, go find it
31556 got dsc Stop(0)
31557 stop 0
31558 Mar 17 22:23:25.850 INFO request completed, latency_us: 370, response_code: 204, uri: /stop/cid/0, method: POST, req_id: [8810] Got stop action so:false kr:false
31559 03ebb4ba-ce61-4adb-91f7-ee170a453482, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
31560 Mar 17 22:23:25.850 INFO request completed, latency_us: 143, response_code: 200, uri: /state/cid/0, method: GET, req_id: ff1dda38-169e-4e8f-93b1-c6a257d130a4, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
315612025-03-17T22:23:25.850ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
315622025-03-17T22:23:25.850ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
315632025-03-17T22:23:25.850ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
315642025-03-17T22:23:25.850ZINFOcrucible: client task is exiting = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
31565 [8810] Exited with: ExitStatus(unix_wait_status(9))
31566 [8810][0] reports Exit
31567 Mar 17 22:23:29.852 INFO request completed, latency_us: 245, response_code: 200, uri: /state/cid/0, method: GET, req_id: d4398532-c34b-48cc-954b-986b93a65d89, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
31568 001/300 Read block 585 len 16384
31569 002/300 Read block 2065 len 32768
31570 003/300 Read block 1345 len 32768
31571 004/300 Write block 4274 len 20480 data: 6 5 4 4 4
31572 005/300 Read block 1329 len 28672
31573 006/300 Flush
31574 007/300 Read block 2411 len 36864
31575 008/300 Read block 4415 len 12288
31576 009/300 Read block 4300 len 40960
31577 010/300 Read block 48 len 12288
31578 011/300 Read block 2797 len 24576
31579 012/300 Read block 3274 len 40960
31580 013/300 Write block 986 len 32768 data: 6 6 8 8 8 7 8 8
31581 014/300 Read block 4852 len 4096
31582 015/300 Read block 3490 len 4096
31583 016/300 Write block 2918 len 32768 data: 6 7 7 8 8 5 4 4
31584 017/300 Read block 2458 len 28672
31585 018/300 Read block 802 len 8192
31586 019/300 Read block 146 len 4096
31587 020/300 Flush
31588 021/300 Write block 3992 len 24576 data: 3 4 4 4 4 6
31589 022/300 Read block 14 len 40960
31590 023/300 Read block 1331 len 24576
31591 024/300 Write block 315 len 36864 data: 7 9 9 10 10 5 5 5 4
31592 025/300 Write block 554 len 12288 data: 9 9 10
31593 026/300 Write block 4791 len 12288 data: 3 2 4
31594 027/300 Write block 2533 len 24576 data: 6 7 6 5 4 5
31595 028/300 Write block 4704 len 16384 data: 5 6 6 5
31596 029/300 Flush
31597 030/300 Read block 1133 len 20480
31598 031/300 Flush
31599 032/300 Flush
31600 033/300 Read block 3356 len 32768
31601 034/300 Read block 3315 len 24576
31602 035/300 Write block 2643 len 4096 data: 6
31603 036/300 Read block 3946 len 12288
31604 037/300 Read block 219 len 36864
31605 038/300 Write block 2419 len 32768 data: 8 7 8 6 5 5 5 4
31606 039/300 Write block 3318 len 28672 data: 4 5 4 3 3 6 7
31607 040/300 Flush
31608 041/300 Read block 2358 len 4096
31609 042/300 Flush
31610 043/300 Read block 1203 len 20480
31611 044/300 Read block 4711 len 4096
31612 045/300 Read block 4114 len 24576
31613 046/300 Read block 335 len 12288
31614 047/300 Read block 2853 len 28672
31615 048/300 Write block 2599 len 32768 data: 7 6 4 5 5 7 6 8
31616 049/300 Flush
31617 050/300 Write block 1965 len 24576 data: 8 5 5 6 5 5
31618 051/300 Write block 957 len 12288 data: 6 5 7
31619 052/300 Read block 1506 len 20480
31620 053/300 Read block 418 len 20480
31621 054/300 Read block 2564 len 28672
31622 055/300 Write block 940 len 28672 data: 5 5 3 3 5 5 5
31623 056/300 Read block 690 len 24576
31624 057/300 Write block 3574 len 24576 data: 4 4 4 3 3 4
31625 058/300 Flush
31626 059/300 Write block 4734 len 32768 data: 9 5 5 5 4 4 4 4
31627 060/300 Write block 3142 len 32768 data: 5 6 6 6 6 7 7 7
31628 061/300 Write block 1799 len 36864 data: 6 5 6 5 6 7 6 5 7
31629 062/300 Read block 2515 len 36864
31630 063/300 Write block 2764 len 24576 data: 4 4 5 6 7 7
31631 064/300 Read block 4338 len 4096
31632 065/300 Read block 1222 len 24576
31633 066/300 Write block 4879 len 4096 data: 8
31634 067/300 Read block 668 len 16384
31635 068/300 Write block 2814 len 28672 data: 4 4 6 6 7 5 5
31636 069/300 Flush
31637 070/300 Write block 4180 len 28672 data: 4 4 5 4 5 4 6
31638 071/300 Read block 1831 len 8192
31639 072/300 Read block 1462 len 32768
31640 073/300 Write block 2615 len 20480 data: 6 4 5 5 7
31641 074/300 Read block 4471 len 12288
31642 075/300 Read block 3060 len 12288
31643 076/300 Write block 503 len 8192 data: 8 9
31644 077/300 Write block 1362 len 8192 data: 9 6
31645 078/300 Read block 2264 len 4096
31646 079/300 Read block 3617 len 40960
31647 080/300 Write block 329 len 20480 data: 3 6 6 5 8
31648 081/300 Write block 4173 len 16384 data: 9 8 5 6
31649 082/300 Flush
31650 083/300 Read block 2878 len 32768
31651 084/300 Read block 4813 len 28672
31652 085/300 Read block 3705 len 40960
31653 086/300 Write block 478 len 16384 data: 2 3 4 5
31654 087/300 Flush
31655 088/300 Write block 611 len 40960 data: 5 4 6 6 6 5 7 5 5 6
31656 089/300 Read block 4631 len 20480
31657 090/300 Read block 4243 len 16384
31658 091/300 Write block 2241 len 16384 data: 4 4 4 5
31659 092/300 Read block 2595 len 8192
31660 093/300 Write block 4546 len 32768 data: 7 9 7 6 6 4 4 5
31661 094/300 Flush
31662 095/300 Read block 1690 len 28672
31663 096/300 Read block 1889 len 36864
31664 097/300 Read block 399 len 20480
31665 098/300 Write block 4469 len 40960 data: 9 10 10 9 7 4 3 3 4 4
31666 099/300 Write block 927 len 8192 data: 6 7
31667 100/300 Flush
31668 101/300 Write block 301 len 36864 data: 6 4 3 4 4 5 5 4 5
31669 102/300 Read block 4670 len 28672
31670 103/300 Read block 3377 len 16384
31671 104/300 Read block 512 len 12288
31672 105/300 Write block 2311 len 24576 data: 5 5 5 4 5 5
31673 106/300 Write block 3015 len 28672 data: 7 7 9 9 8 6 6
31674 107/300 Write block 698 len 40960 data: 6 5 5 5 4 8 6 5 5 6
31675 108/300 Read block 2671 len 12288
31676 109/300 Read block 3953 len 20480
31677 110/300 Flush
31678 111/300 Read block 1596 len 16384
31679 112/300 Read block 3272 len 40960
31680 113/300 Read block 4417 len 40960
31681 114/300 Flush
31682 115/300 Read block 3364 len 16384
31683 116/300 Flush
31684 117/300 Read block 2245 len 40960
31685 118/300 Write block 2939 len 12288 data: 7 9 10
31686 119/300 Flush
31687 120/300 Read block 3729 len 36864
31688 121/300 Read block 4017 len 4096
31689 122/300 Read block 2396 len 16384
31690 123/300 Read block 4024 len 32768
31691 124/300 Read block 3379 len 24576
31692 125/300 Read block 3677 len 8192
31693 126/300 Read block 1267 len 12288
31694 127/300 Read block 4905 len 20480
31695 128/300 Flush
31696 129/300 Write block 1868 len 20480 data: 4 6 7 8 6
31697 130/300 Read block 1824 len 32768
31698 131/300 Flush
31699 132/300 Read block 1300 len 24576
31700 133/300 Write block 2674 len 24576 data: 8 8 8 7 7 4
31701 134/300 Write block 3734 len 4096 data: 7
31702 135/300 Read block 686 len 28672
31703 136/300 Read block 2984 len 36864
31704 137/300 Read block 177 len 32768
31705 138/300 Flush
31706 139/300 Write block 4742 len 4096 data: 5
31707 140/300 Flush
31708 141/300 Write block 2669 len 28672 data: 4 4 6 8 8 9 9
31709 142/300 Write block 1732 len 28672 data: 5 5 5 5 4 4 4
31710 143/300 Flush
31711 144/300 Read block 1309 len 36864
31712 145/300 Flush
31713 146/300 Read block 2193 len 20480
31714 147/300 Read block 3825 len 16384
31715 148/300 Flush
31716 149/300 Write block 2670 len 20480 data: 5 7 9 9 10
31717 150/300 Read block 3723 len 24576
31718 151/300 Write block 4019 len 12288 data: 8 6 6
31719 152/300 Write block 3489 len 32768 data: 3 6 8 9 8 8 9 8
31720 153/300 Read block 412 len 28672
31721 154/300 Write block 222 len 28672 data: 5 5 5 5 5 4 5
31722 155/300 Write block 430 len 24576 data: 4 4 4 3 3 2
31723 156/300 Write block 2037 len 16384 data: 5 4 7 8
31724 157/300 Read block 1498 len 20480
31725 158/300 Write block 4216 len 40960 data: 5 7 7 5 5 3 4 4 3 3
31726 159/300 Write block 3789 len 28672 data: 3 3 3 2 3 4 5
31727 160/300 Write block 4765 len 28672 data: 7 7 7 6 6 5 6
31728 161/300 Write block 4039 len 40960 data: 6 5 5 5 4 3 4 4 5 3
31729 162/300 Read block 854 len 40960
31730 163/300 Write block 3775 len 24576 data: 5 4 3 4 4 5
31731 164/300 Read block 1869 len 40960
31732 165/300 Read block 4920 len 4096
31733 166/300 Read block 2488 len 20480
31734 167/300 Read block 715 len 20480
31735 168/300 Read block 3026 len 8192
31736 169/300 Read block 242 len 40960
31737 170/300 Write block 1583 len 4096 data: 7
31738 171/300 Write block 4532 len 36864 data: 8 9 8 8 7 7 7 4 4
31739 172/300 Read block 3435 len 16384
31740 173/300 Read block 1497 len 16384
31741 174/300 Read block 3945 len 8192
31742 175/300 Read block 2921 len 20480
31743 176/300 Read block 4107 len 32768
31744 177/300 Read block 2536 len 4096
31745 178/300 Read block 4634 len 4096
31746 179/300 Write block 4527 len 16384 data: 4 5 5 7
31747 180/300 Read block 2207 len 8192
31748 181/300 Write block 4114 len 8192 data: 6 3
31749 182/300 Read block 904 len 20480
31750 183/300 Read block 4548 len 40960
31751 184/300 Write block 3764 len 36864 data: 4 6 7 5 4 4 4 4 5
31752 185/300 Write block 2911 len 28672 data: 4 5 4 4 6 6 6
31753 186/300 Write block 3869 len 4096 data: 6
31754 187/300 Read block 915 len 24576
31755 188/300 Read block 4311 len 16384
31756 189/300 Write block 2481 len 4096 data: 5
31757 190/300 Flush
31758 191/300 Write block 3812 len 20480 data: 4 5 4 3 4
31759 192/300 Flush
31760 193/300 Write block 2058 len 4096 data: 7
31761 194/300 Read block 2816 len 4096
31762 195/300 Read block 3144 len 24576
31763 196/300 Write block 949 len 8192 data: 4 3
31764 197/300 Write block 1648 len 40960 data: 10 8 6 6 4 6 6 5 5 6
31765 198/300 Read block 450 len 8192
31766 199/300 Write block 2735 len 8192 data: 3 3
31767 200/300 Write block 1255 len 16384 data: 7 7 7 6
31768 201/300 Read block 3562 len 12288
31769 202/300 Write block 1064 len 24576 data: 6 5 7 6 4 5
31770 203/300 Write block 3294 len 24576 data: 5 5 5 5 4 5
31771 204/300 Read block 3276 len 4096
31772 205/300 Write block 2340 len 16384 data: 5 5 5 5
31773 206/300 Write block 3431 len 4096 data: 4
31774 207/300 Read block 3241 len 12288
31775 208/300 Write block 2675 len 12288 data: 10 9 8
31776 209/300 Write block 2971 len 20480 data: 5 6 7 5 4
31777 210/300 Read block 3121 len 16384
31778 211/300 Flush
31779 212/300 Flush
31780 213/300 Flush
31781 214/300 Read block 770 len 16384
31782 215/300 Read block 1973 len 28672
31783 216/300 Read block 4962 len 16384
31784 217/300 Read block 4572 len 40960
31785 218/300 Read block 2174 len 8192
31786 219/300 Flush
31787 220/300 Flush
31788 221/300 Read block 2234 len 24576
31789 222/300 Read block 890 len 36864
31790 223/300 Write block 1002 len 40960 data: 4 4 4 3 5 4 4 4 5 6
31791 224/300 Read block 3903 len 32768
31792 225/300 Write block 3307 len 16384 data: 3 3 4 5
31793 226/300 Write block 3643 len 28672 data: 10 11 10 8 8 7 8
31794 227/300 Write block 1238 len 8192 data: 7 6
31795 228/300 Write block 3622 len 28672 data: 4 4 3 3 3 3 3
31796 229/300 Write block 2575 len 28672 data: 3 3 4 3 3 3 3
31797 230/300 Write block 4193 len 20480 data: 3 4 4 4 6
31798 231/300 Write block 3487 len 32768 data: 3 3 4 7 9 10 9 9
31799 232/300 Read block 3759 len 32768
31800 233/300 Write block 3816 len 36864 data: 5 4 5 5 6 6 4 4 5
31801 234/300 Write block 570 len 24576 data: 3 3 3 3 4 5
31802 235/300 Write block 1690 len 16384 data: 6 6 5 6
31803 236/300 Write block 53 len 24576 data: 4 4 4 5 5 5
31804 237/300 Flush
31805 238/300 Read block 4367 len 16384
31806 239/300 Read block 3162 len 40960
31807 240/300 Read block 3589 len 28672
31808 241/300 Read block 4374 len 28672
31809 242/300 Write block 730 len 12288 data: 4 4 4
31810 243/300 Write block 760 len 4096 data: 3
31811 244/300 Read block 1742 len 12288
31812 245/300 Write block 4420 len 8192 data: 8 10
31813 246/300 Write block 1747 len 24576 data: 8 6 6 4 4 5
31814 247/300 Flush
31815 248/300 Write block 554 len 4096 data: 10
31816 249/300 Write block 458 len 36864 data: 4 4 5 5 5 6 8 9 8
31817 250/300 Read block 932 len 28672
31818 251/300 Flush
31819 252/300 Write block 3199 len 20480 data: 5 5 5 6 6
31820 253/300 Write block 3400 len 16384 data: 4 5 8 10
31821 254/300 Read block 1529 len 40960
31822 255/300 Write block 922 len 12288 data: 6 5 6
31823 256/300 Read block 537 len 12288
31824 257/300 Read block 3896 len 16384
31825 258/300 Write block 1356 len 8192 data: 9 10
31826 259/300 Write block 2414 len 32768 data: 5 6 6 8 6 9 8 9
31827 260/300 Read block 3034 len 28672
31828 261/300 Read block 2504 len 28672
31829 262/300 Read block 4745 len 24576
31830 263/300 Write block 2664 len 40960 data: 5 6 7 7 8 5 6 8 10 10
31831 264/300 Write block 1876 len 16384 data: 6 6 4 4
31832 265/300 Write block 1228 len 8192 data: 9 4
31833 266/300 Write block 4099 len 8192 data: 9 6
31834 267/300 Write block 1307 len 20480 data: 6 8 7 7 8
31835 268/300 Write block 259 len 24576 data: 7 6 7 6 3 4
31836 269/300 Read block 1271 len 20480
31837 270/300 Flush
31838 271/300 Write block 2501 len 12288 data: 5 4 4
31839 272/300 Read block 4604 len 8192
31840 273/300 Read block 2793 len 4096
31841 274/300 Read block 3694 len 28672
31842 275/300 Read block 4642 len 12288
31843 276/300 Flush
31844 277/300 Read block 3611 len 8192
31845 278/300 Read block 1624 len 12288
31846 279/300 Write block 1154 len 24576 data: 8 7 9 8 7 5
31847 280/300 Read block 1601 len 8192
31848 281/300 Read block 299 len 32768
31849 282/300 Read block 3190 len 16384
31850 283/300 Read block 1109 len 28672
31851 284/300 Read block 1808 len 32768
31852 285/300 Read block 4677 len 28672
31853 286/300 Write block 4598 len 36864 data: 3 3 3 3 6 7 10 8 11
31854 287/300 Read block 3872 len 4096
31855 288/300 Read block 1763 len 20480
31856 289/300 Read block 3797 len 20480
31857 290/300 Write block 3284 len 4096 data: 5
31858 291/300 Write block 1907 len 20480 data: 2 3 3 2 2
31859 292/300 Write block 3745 len 24576 data: 7 7 7 6 7 4
31860 293/300 Read block 3856 len 16384
31861 294/300 Read block 3492 len 16384
31862 295/300 Read block 1801 len 20480
31863 296/300 Read block 327 len 8192
31864 297/300 Read block 4658 len 12288
31865 298/300 Flush
31866 299/300 Write block 2908 len 32768 data: 4 6 4 5 6 5 5 7
31867 300/300 Write block 3169 len 40960 data: 9 8 7 6 8 8 7 5 5 5
31868 301/300 Flush
31869 Main task has work to do, go find it
31870 got dsc Start(0)
31871 start 0
31872 Mar 17 22:23:30.094 INFO request completed, latency_us: 215, response_code: 204, uri: /start/cid/0, method: POST, req_id: ef1b7dbf-8c7f-45a8-8bcf-fdefc35a9cd8, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
31873 Replay: started 0, returned:Ok(())
31874 ----------------------------------------------------------------
31875 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
31876 States: Offline Active Active
31877 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
31878 8850 Acked Read 4 Sent Done Done false
31879 8851 Acked Read 8 Sent Done Done false
31880 8852 Acked Read 8 Sent Done Done false
31881 8853 Acked Write 5 Sent Done Done false
31882 8854 Acked Read 7 Sent Done Done false
31883 8855 Acked Flush 0 Sent Done Done false
31884 8856 Acked Read 9 Sent Done Done false
31885 8857 Acked Read 3 Sent Done Done false
31886 8858 Acked Read 10 Sent Done Done false
31887 8859 Acked Read 3 Sent Done Done false
31888 8860 Acked Read 6 Sent Done Done false
31889 8861 Acked Read 10 Sent Done Done false
31890 8862 Acked Write 8 Sent Done Done false
31891 8863 Acked Read 1 Sent Done Done false
31892 8864 Acked Read 1 Sent Done Done false
31893 8865 Acked Write 8 Sent Done Done false
31894 8866 Acked Read 7 Sent Done Done false
31895 8867 Acked Read 2 Sent Done Done false
31896 8868 Acked Read 1 Sent Done Done false
31897 8869 Acked Flush 0 Sent Done Done false
31898 8870 Acked Write 6 Sent Done Done false
31899 8871 Acked Read 10 Sent Done Done false
31900 8872 Acked Read 6 Sent Done Done false
31901 8873 Acked Write 9 Sent Done Done false
31902 8874 Acked Write 3 Sent Done Done false
31903 8875 Acked Write 3 Sent Done Done false
31904 8876 Acked Write 6 Sent Done Done false
31905 8877 Acked Write 4 Sent Done Done false
31906 8878 Acked Flush 0 Sent Done Done false
31907 8879 Acked Read 5 Sent Done Done false
31908 8880 Acked Flush 0 Sent Done Done false
31909 8881 Acked Flush 0 Sent Done Done false
31910 8882 Acked Read 8 Sent Done Done false
31911 8883 Acked Read 6 Sent Done Done false
31912 8884 Acked Write 1 Sent Done Done false
31913 8885 Acked Read 3 Sent Done Done false
31914 8886 Acked Read 9 Sent Done Done false
31915 8887 Acked Write 8 Sent Done Done false
31916 8888 Acked Write 7 Sent Done Done false
31917 8889 Acked Flush 0 Sent Done Done false
31918 8890 Acked Read 1 Sent Done Done false
31919 8891 Acked Flush 0 Sent Done Done false
31920 8892 Acked Read 5 Sent Done Done false
31921 8893 Acked Read 1 Sent Done Done false
31922 8894 Acked Read 6 Sent Done Done false
31923 8895 Acked Read 3 Sent Done Done false
31924 8896 Acked Read 7 Sent Done Done false
31925 8897 Acked Write 8 Sent Done Done false
31926 8898 Acked Flush 0 Sent Done Done false
31927 8899 Acked Write 6 Sent Done Done false
31928 8900 Acked Write 3 Sent Done Done false
31929 8901 Acked Read 5 Sent Done Done false
31930 8902 Acked Read 5 Sent Done Done false
31931 8903 Acked Read 7 Sent Done Done false
31932 8904 Acked Write 7 Sent Done Done false
31933 8905 Acked Read 6 Sent Done Done false
31934 8906 Acked Write 6 Sent Done Done false
31935 8907 Acked Flush 0 Sent Done Done false
31936 8908 Acked Write 8 Sent Done Done false
31937 8909 Acked Write 8 Sent Done Done false
31938 8910 Acked Write 9 Sent Done Done false
31939 8911 Acked Read 9 Sent Done Done false
31940 8912 Acked Write 6 Sent Done Done false
31941 8913 Acked Read 1 Sent Done Done false
31942 8914 Acked Read 6 Sent Done Done false
31943 8915 Acked Write 1 Sent Done Done false
31944 8916 Acked Read 4 Sent Done Done false
31945 8917 Acked Write 7 Sent Done Done false
31946 8918 Acked Flush 0 Sent Done Done false
31947 8919 Acked Write 7 Sent Done Done false
31948 8920 Acked Read 2 Sent Done Done false
31949 8921 Acked Read 8 Sent Done Done false
31950 8922 Acked Write 5 Sent Done Done false
31951 8923 Acked Read 3 Sent Done Done false
31952 8924 Acked Read 3 Sent Done Done false
31953 8925 Acked Write 2 Sent Done Done false
31954 8926 Acked Write 2 Sent Done Done false
31955 8927 Acked Read 1 Sent Done Done false
31956 8928 Acked Read 10 Sent Done Done false
31957 8929 Acked Write 5 Sent Done Done false
31958 8930 Acked Write 4 Sent Done Done false
31959 8931 Acked Flush 0 Sent Done Done false
31960 8932 Acked Read 8 Sent Done Done false
31961 8933 Acked Read 7 Sent Done Done false
31962 8934 Acked Read 10 Sent Done Done false
31963 8935 Acked Write 4 Sent Done Done false
31964 8936 Acked Flush 0 Sent Done Done false
31965 8937 Acked Write 10 Sent Done Done false
31966 8938 Acked Read 5 Sent Done Done false
31967 8939 Acked Read 4 Sent Done Done false
31968 8940 Acked Write 4 Sent Done Done false
31969 8941 Acked Read 2 Sent Done Done false
31970 8942 Acked Write 8 Sent Done Done false
31971 8943 Acked Flush 0 Sent Done Done false
31972 8944 Acked Read 7 Sent Done Done false
31973 8945 Acked Read 9 Sent Done Done false
31974 8946 Acked Read 5 Sent Done Done false
31975 8947 Acked Write 10 Sent Done Done false
31976 8948 Acked Write 2 Sent Done Done false
31977 8949 Acked Flush 0 Sent Done Done false
31978 8950 Acked Write 9 Sent Done Done false
31979 8951 Acked Read 7 Sent Done Done false
31980 8952 Acked Read 4 Sent Done Done false
31981 8953 Acked Read 3 Sent Done Done false
31982 8954 Acked Write 6 Sent Done Done false
31983 8955 Acked Write 7 Sent Done Done false
31984 8956 Acked Write 10 Sent Done Done false
31985 8957 Acked Read 3 Sent Done Done false
31986 8958 Acked Read 5 Sent Done Done false
31987 8959 Acked Flush 0 Sent Done Done false
31988 8960 Acked Read 4 Sent Done Done false
31989 8961 Acked Read 10 Sent Done Done false
31990 8962 Acked Read 10 Sent Done Done false
31991 8963 Acked Flush 0 Sent Done Done false
31992 8964 Acked Read 4 Sent Done Done false
31993 8965 Acked Flush 0 Sent Done Done false
31994 8966 Acked Read 10 Sent Done Done false
31995 8967 Acked Write 3 Sent Done Done false
31996 8968 Acked Flush 0 Sent Done Done false
31997 8969 Acked Read 9 Sent Done Done false
31998 8970 Acked Read 1 Sent Done Done false
31999 8971 Acked Read 4 Sent Done Done false
32000 8972 Acked Read 8 Sent Done Done false
32001 8973 Acked Read 6 Sent Done Done false
32002 8974 Acked Read 2 Sent Done Done false
32003 8975 Acked Read 3 Sent Done Done false
32004 8976 Acked Read 5 Sent Done Done false
32005 8977 Acked Flush 0 Sent Done Done false
32006 8978 Acked Write 5 Sent Done Done false
32007 8979 Acked Read 8 Sent Done Done false
32008 8980 Acked Flush 0 Sent Done Done false
32009 8981 Acked Read 6 Sent Done Done false
32010 8982 Acked Write 6 Sent Done Done false
32011 8983 Acked Write 1 Sent Done Done false
32012 8984 Acked Read 7 Sent Done Done false
32013 8985 Acked Read 9 Sent Done Done false
32014 8986 Acked Read 8 Sent Done Done false
32015 8987 Acked Flush 0 Sent Done Done false
32016 8988 Acked Write 1 Sent Done Done false
32017 8989 Acked Flush 0 Sent Done Done false
32018 8990 Acked Write 7 Sent Done Done false
32019 8991 Acked Write 7 Sent Done Done false
32020 8992 Acked Flush 0 Sent Done Done false
32021 8993 Acked Read 9 Sent Done Done false
32022 8994 Acked Flush 0 Sent Done Done false
32023 8995 Acked Read 5 Sent Done Done false
32024 8996 Acked Read 4 Sent Done Done false
32025 8997 Acked Flush 0 Sent Done Done false
32026 8998 Acked Write 5 Sent Done Done false
32027 8999 Acked Read 6 Sent Done Done false
32028 9000 Acked Write 3 Sent Done Done false
32029 9001 Acked Write 8 Sent Done Done false
32030 9002 Acked Read 7 Sent Done Done false
32031 9003 Acked Write 7 Sent Done Done false
32032 9004 Acked Write 6 Sent Done Done false
32033 9005 Acked Write 4 Sent Done Done false
32034 9006 Acked Read 5 Sent Done Done false
32035 9007 Acked Write 10 Sent Done Done false
32036 9008 Acked Write 7 Sent Done Done false
32037 9009 Acked Write 7 Sent Done Done false
32038 9010 Acked Write 10 Sent Done Done false
32039 9011 Acked Read 10 Sent Done Done false
32040 9012 Acked Write 6 Sent Done Done false
32041 9013 Acked Read 10 Sent Done Done false
32042 9014 Acked Read 1 Sent Done Done false
32043 9015 Acked Read 5 Sent Done Done false
32044 9016 Acked Read 5 Sent Done Done false
32045 9017 Acked Read 2 Sent Done Done false
32046 9018 Acked Read 10 Sent Done Done false
32047 9019 Acked Write 1 Sent Done Done false
32048 9020 Acked Write 9 Sent Done Done false
32049 9021 Acked Read 4 Sent Done Done false
32050 9022 Acked Read 4 Sent Done Done false
32051 9023 Acked Read 2 Sent Done Done false
32052 9024 Acked Read 5 Sent Done Done false
32053 9025 Acked Read 8 Sent Done Done false
32054 9026 Acked Read 1 Sent Done Done false
32055 9027 Acked Read 1 Sent Done Done false
32056 9028 Acked Write 4 Sent Done Done false
32057 9029 Acked Read 2 Sent Done Done false
32058 9030 Acked Write 2 Sent Done Done false
32059 9031 Acked Read 5 Sent Done Done false
32060 9032 Acked Read 10 Sent Done Done false
32061 9033 Acked Write 9 Sent Done Done false
32062 9034 Acked Write 7 Sent Done Done false
32063 9035 Acked Write 1 Sent Done Done false
32064 9036 Acked Read 6 Sent Done Done false
32065 9037 Acked Read 4 Sent Done Done false
32066 9038 Acked Write 1 Sent Done Done false
32067 9039 Acked Flush 0 Sent Done Done false
32068 9040 Acked Write 5 Sent Done Done false
32069 9041 Acked Flush 0 Sent Done Done false
32070 9042 Acked Write 1 Sent Done Done false
32071 9043 Acked Read 1 Sent Done Done false
32072 9044 Acked Read 6 Sent Done Done false
32073 9045 Acked Write 2 Sent Done Done false
32074 9046 Acked Write 10 Sent Done Done false
32075 9047 Acked Read 2 Sent Done Done false
32076 9048 Acked Write 2 Sent Done Done false
32077 9049 Acked Write 4 Sent Done Done false
32078 9050 Acked Read 3 Sent Done Done false
32079 9051 Acked Write 6 Sent Done Done false
32080 9052 Acked Write 6 Sent Done Done false
32081 9053 Acked Read 1 Sent Done Done false
32082 9054 Acked Write 4 Sent Done Done false
32083 9055 Acked Write 1 Sent Done Done false
32084 9056 Acked Read 3 Sent Done Done false
32085 9057 Acked Write 3 Sent Done Done false
32086 9058 Acked Write 5 Sent Done Done false
32087 9059 Acked Read 4 Sent Done Done false
32088 9060 Acked Flush 0 Sent Done Done false
32089 9061 Acked Flush 0 Sent Done Done false
32090 9062 Acked Flush 0 Sent Done Done false
32091 9063 Acked Read 4 Sent Done Done false
32092 9064 Acked Read 7 Sent Done Done false
32093 9065 Acked Read 4 Sent Done Done false
32094 9066 Acked Read 10 Sent Done Done false
32095 9067 Acked Read 2 Sent Done Done false
32096 9068 Acked Flush 0 Sent Done Done false
32097 9069 Acked Flush 0 Sent Done Done false
32098 9070 Acked Read 6 Sent Done Done false
32099 9071 Acked Read 9 Sent Done Done false
32100 9072 Acked Write 10 Sent Done Done false
32101 9073 Acked Read 8 Sent Done Done false
32102 9074 Acked Write 4 Sent Done Done false
32103 9075 Acked Write 7 Sent Done Done false
32104 9076 Acked Write 2 Sent Done Done false
32105 9077 Acked Write 7 Sent Done Done false
32106 9078 Acked Write 7 Sent Done Done false
32107 9079 Acked Write 5 Sent Done Done false
32108 9080 Acked Write 8 Sent Done Done false
32109 9081 Acked Read 8 Sent Done Done false
32110 9082 Acked Write 9 Sent Done Done false
32111 9083 Acked Write 6 Sent Done Done false
32112 9084 Acked Write 4 Sent Done Done false
32113 9085 Acked Write 6 Sent Done Done false
32114 9086 Acked Flush 0 Sent Done Done false
32115 9087 Acked Read 4 Sent Done Done false
32116 9088 Acked Read 10 Sent Done Done false
32117 9089 Acked Read 7 Sent Done Done false
32118 9090 Acked Read 7 Sent Done Done false
32119 9091 Acked Write 3 Sent Done Done false
32120 9092 Acked Write 1 Sent Done Done false
32121 9093 Acked Read 3 Sent Done Done false
32122 9094 Acked Write 2 Sent Done Done false
32123 9095 Acked Write 6 Sent Done Done false
32124 9096 Acked Flush 0 Sent Done Done false
32125 9097 Acked Write 1 Sent Done Done false
32126 9098 Acked Write 9 Sent Done Done false
32127 9099 Acked Read 7 Sent Done Done false
32128 9100 Acked Flush 0 Sent Done Done false
32129 9101 Acked Write 5 Sent Done Done false
32130 9102 Acked Write 4 Sent Done Done false
32131 9103 Acked Read 10 Sent Done Done false
32132 9104 Acked Write 3 Sent Done Done false
32133 9105 Acked Read 3 Sent Done Done false
32134 9106 Acked Read 4 Sent Done Done false
32135 9107 Acked Write 2 Sent Done Done false
32136 9108 Acked Write 8 Sent Done Done false
32137 9109 Acked Read 7 Sent Done Done false
32138 9110 Acked Read 7 Sent Done Done false
32139 9111 Acked Read 6 Sent Done Done false
32140 9112 Acked Write 10 Sent Done Done false
32141 9113 Acked Write 4 Sent Done Done false
32142 9114 Acked Write 2 Sent Done Done false
32143 9115 Acked Write 2 Sent Done Done false
32144 9116 Acked Write 5 Sent Done Done false
32145 9117 Acked Write 6 Sent Done Done false
32146 9118 Acked Read 5 Sent Done Done false
32147 9119 Acked Flush 0 Sent Done Done false
32148 9120 Acked Write 3 Sent Done Done false
32149 9121 Acked Read 2 Sent Done Done false
32150 9122 Acked Read 1 Sent Done Done false
32151 9123 Acked Read 7 Sent Done Done false
32152 9124 Acked Read 3 Sent Done Done false
32153 9125 Acked Flush 0 Sent Done Done false
32154 9126 Acked Read 2 Sent Done Done false
32155 9127 Acked Read 3 Sent Done Done false
32156 9128 Acked Write 6 Sent Done Done false
32157 9129 Acked Read 2 Sent Done Done false
32158 9130 Acked Read 8 Sent Done Done false
32159 9131 Acked Read 4 Sent Done Done false
32160 9132 Acked Read 7 Sent Done Done false
32161 9133 Acked Read 8 Sent Done Done false
32162 9134 Acked Read 7 Sent Done Done false
32163 9135 Acked Write 9 Sent Done Done false
32164 9136 Acked Read 1 Sent Done Done false
32165 9137 Acked Read 5 Sent Done Done false
32166 9138 Acked Read 5 Sent Done Done false
32167 9139 Acked Write 1 Sent Done Done false
32168 9140 Acked Write 5 Sent Done Done false
32169 9141 Acked Write 6 Sent Done Done false
32170 9142 Acked Read 4 Sent Done Done false
32171 9143 Acked Read 4 Sent Done Done false
32172 9144 Acked Read 5 Sent Done Done false
32173 9145 Acked Read 2 Sent Done Done false
32174 9146 Acked Read 3 Sent Done Done false
32175 9147 Acked Flush 0 Sent Done Done false
32176 9148 Acked Write 8 Sent Done Done false
32177 9149 Acked Write 10 Sent Done Done false
32178 9150 Acked Flush 0 Sent Done Done false
32179 STATES DS:0 DS:1 DS:2 TOTAL
32180 Sent 301 0 0 301
32181 Done 0 301 301 602
32182 Skipped 0 0 0 0
32183 Error 0 0 0 0
32184 Last Flush: Some(JobId(8849)) Some(JobId(9150)) Some(JobId(9150))
32185 Downstairs last five completed: 8849 8848 8847 8846 8845
32186 Upstairs last five completed: 9150 9149 9148 9147 9146
32187 CLIENT: Up:0 ds:301 act:2
321882025-03-17T22:23:30.596ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
321892025-03-17T22:23:31.276ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
32190 [8810] got start action
32191 [8810] I am going to restart
32192 Starting downstairs at port 8810
32193 Make output file at "/tmp/dsc/downstairs-8810.txt"
32194 [8810][0] reports Starting
32195 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1233)
32196 [8810][0] reports Running
321972025-03-17T22:23:32.279ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
321982025-03-17T22:23:33.281ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
32199 ----------------------------------------------------------------
32200 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32201 States: Offline Active Active
32202 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32203 8850 Acked Read 4 Sent Done Done false
32204 8851 Acked Read 8 Sent Done Done false
32205 8852 Acked Read 8 Sent Done Done false
32206 8853 Acked Write 5 Sent Done Done false
32207 8854 Acked Read 7 Sent Done Done false
32208 8855 Acked Flush 0 Sent Done Done false
32209 8856 Acked Read 9 Sent Done Done false
32210 8857 Acked Read 3 Sent Done Done false
32211 8858 Acked Read 10 Sent Done Done false
32212 8859 Acked Read 3 Sent Done Done false
32213 8860 Acked Read 6 Sent Done Done false
32214 8861 Acked Read 10 Sent Done Done false
32215 8862 Acked Write 8 Sent Done Done false
32216 8863 Acked Read 1 Sent Done Done false
32217 8864 Acked Read 1 Sent Done Done false
32218 8865 Acked Write 8 Sent Done Done false
32219 8866 Acked Read 7 Sent Done Done false
32220 8867 Acked Read 2 Sent Done Done false
32221 8868 Acked Read 1 Sent Done Done false
32222 8869 Acked Flush 0 Sent Done Done false
32223 8870 Acked Write 6 Sent Done Done false
32224 8871 Acked Read 10 Sent Done Done false
32225 8872 Acked Read 6 Sent Done Done false
32226 8873 Acked Write 9 Sent Done Done false
32227 8874 Acked Write 3 Sent Done Done false
32228 8875 Acked Write 3 Sent Done Done false
32229 8876 Acked Write 6 Sent Done Done false
32230 8877 Acked Write 4 Sent Done Done false
32231 8878 Acked Flush 0 Sent Done Done false
32232 8879 Acked Read 5 Sent Done Done false
32233 8880 Acked Flush 0 Sent Done Done false
32234 8881 Acked Flush 0 Sent Done Done false
32235 8882 Acked Read 8 Sent Done Done false
32236 8883 Acked Read 6 Sent Done Done false
32237 8884 Acked Write 1 Sent Done Done false
32238 8885 Acked Read 3 Sent Done Done false
32239 8886 Acked Read 9 Sent Done Done false
32240 8887 Acked Write 8 Sent Done Done false
32241 8888 Acked Write 7 Sent Done Done false
32242 8889 Acked Flush 0 Sent Done Done false
32243 8890 Acked Read 1 Sent Done Done false
32244 8891 Acked Flush 0 Sent Done Done false
32245 8892 Acked Read 5 Sent Done Done false
32246 8893 Acked Read 1 Sent Done Done false
32247 8894 Acked Read 6 Sent Done Done false
32248 8895 Acked Read 3 Sent Done Done false
32249 8896 Acked Read 7 Sent Done Done false
32250 8897 Acked Write 8 Sent Done Done false
32251 8898 Acked Flush 0 Sent Done Done false
32252 8899 Acked Write 6 Sent Done Done false
32253 8900 Acked Write 3 Sent Done Done false
32254 8901 Acked Read 5 Sent Done Done false
32255 8902 Acked Read 5 Sent Done Done false
32256 8903 Acked Read 7 Sent Done Done false
32257 8904 Acked Write 7 Sent Done Done false
32258 8905 Acked Read 6 Sent Done Done false
32259 8906 Acked Write 6 Sent Done Done false
32260 8907 Acked Flush 0 Sent Done Done false
32261 8908 Acked Write 8 Sent Done Done false
32262 8909 Acked Write 8 Sent Done Done false
32263 8910 Acked Write 9 Sent Done Done false
32264 8911 Acked Read 9 Sent Done Done false
32265 8912 Acked Write 6 Sent Done Done false
32266 8913 Acked Read 1 Sent Done Done false
32267 8914 Acked Read 6 Sent Done Done false
32268 8915 Acked Write 1 Sent Done Done false
32269 8916 Acked Read 4 Sent Done Done false
32270 8917 Acked Write 7 Sent Done Done false
32271 8918 Acked Flush 0 Sent Done Done false
32272 8919 Acked Write 7 Sent Done Done false
32273 8920 Acked Read 2 Sent Done Done false
32274 8921 Acked Read 8 Sent Done Done false
32275 8922 Acked Write 5 Sent Done Done false
32276 8923 Acked Read 3 Sent Done Done false
32277 8924 Acked Read 3 Sent Done Done false
32278 8925 Acked Write 2 Sent Done Done false
32279 8926 Acked Write 2 Sent Done Done false
32280 8927 Acked Read 1 Sent Done Done false
32281 8928 Acked Read 10 Sent Done Done false
32282 8929 Acked Write 5 Sent Done Done false
32283 8930 Acked Write 4 Sent Done Done false
32284 8931 Acked Flush 0 Sent Done Done false
32285 8932 Acked Read 8 Sent Done Done false
32286 8933 Acked Read 7 Sent Done Done false
32287 8934 Acked Read 10 Sent Done Done false
32288 8935 Acked Write 4 Sent Done Done false
32289 8936 Acked Flush 0 Sent Done Done false
32290 8937 Acked Write 10 Sent Done Done false
32291 8938 Acked Read 5 Sent Done Done false
32292 8939 Acked Read 4 Sent Done Done false
32293 8940 Acked Write 4 Sent Done Done false
32294 8941 Acked Read 2 Sent Done Done false
32295 8942 Acked Write 8 Sent Done Done false
32296 8943 Acked Flush 0 Sent Done Done false
32297 8944 Acked Read 7 Sent Done Done false
32298 8945 Acked Read 9 Sent Done Done false
32299 8946 Acked Read 5 Sent Done Done false
32300 8947 Acked Write 10 Sent Done Done false
32301 8948 Acked Write 2 Sent Done Done false
32302 8949 Acked Flush 0 Sent Done Done false
32303 8950 Acked Write 9 Sent Done Done false
32304 8951 Acked Read 7 Sent Done Done false
32305 8952 Acked Read 4 Sent Done Done false
32306 8953 Acked Read 3 Sent Done Done false
32307 8954 Acked Write 6 Sent Done Done false
32308 8955 Acked Write 7 Sent Done Done false
32309 8956 Acked Write 10 Sent Done Done false
32310 8957 Acked Read 3 Sent Done Done false
32311 8958 Acked Read 5 Sent Done Done false
32312 8959 Acked Flush 0 Sent Done Done false
32313 8960 Acked Read 4 Sent Done Done false
32314 8961 Acked Read 10 Sent Done Done false
32315 8962 Acked Read 10 Sent Done Done false
32316 8963 Acked Flush 0 Sent Done Done false
32317 8964 Acked Read 4 Sent Done Done false
32318 8965 Acked Flush 0 Sent Done Done false
32319 8966 Acked Read 10 Sent Done Done false
32320 8967 Acked Write 3 Sent Done Done false
32321 8968 Acked Flush 0 Sent Done Done false
32322 8969 Acked Read 9 Sent Done Done false
32323 8970 Acked Read 1 Sent Done Done false
32324 8971 Acked Read 4 Sent Done Done false
32325 8972 Acked Read 8 Sent Done Done false
32326 8973 Acked Read 6 Sent Done Done false
32327 8974 Acked Read 2 Sent Done Done false
32328 8975 Acked Read 3 Sent Done Done false
32329 8976 Acked Read 5 Sent Done Done false
32330 8977 Acked Flush 0 Sent Done Done false
32331 8978 Acked Write 5 Sent Done Done false
32332 8979 Acked Read 8 Sent Done Done false
32333 8980 Acked Flush 0 Sent Done Done false
32334 8981 Acked Read 6 Sent Done Done false
32335 8982 Acked Write 6 Sent Done Done false
32336 8983 Acked Write 1 Sent Done Done false
32337 8984 Acked Read 7 Sent Done Done false
32338 8985 Acked Read 9 Sent Done Done false
32339 8986 Acked Read 8 Sent Done Done false
32340 8987 Acked Flush 0 Sent Done Done false
32341 8988 Acked Write 1 Sent Done Done false
32342 8989 Acked Flush 0 Sent Done Done false
32343 8990 Acked Write 7 Sent Done Done false
32344 8991 Acked Write 7 Sent Done Done false
32345 8992 Acked Flush 0 Sent Done Done false
32346 8993 Acked Read 9 Sent Done Done false
32347 8994 Acked Flush 0 Sent Done Done false
32348 8995 Acked Read 5 Sent Done Done false
32349 8996 Acked Read 4 Sent Done Done false
32350 8997 Acked Flush 0 Sent Done Done false
32351 8998 Acked Write 5 Sent Done Done false
32352 8999 Acked Read 6 Sent Done Done false
32353 9000 Acked Write 3 Sent Done Done false
32354 9001 Acked Write 8 Sent Done Done false
32355 9002 Acked Read 7 Sent Done Done false
32356 9003 Acked Write 7 Sent Done Done false
32357 9004 Acked Write 6 Sent Done Done false
32358 9005 Acked Write 4 Sent Done Done false
32359 9006 Acked Read 5 Sent Done Done false
32360 9007 Acked Write 10 Sent Done Done false
32361 9008 Acked Write 7 Sent Done Done false
32362 9009 Acked Write 7 Sent Done Done false
32363 9010 Acked Write 10 Sent Done Done false
32364 9011 Acked Read 10 Sent Done Done false
32365 9012 Acked Write 6 Sent Done Done false
32366 9013 Acked Read 10 Sent Done Done false
32367 9014 Acked Read 1 Sent Done Done false
32368 9015 Acked Read 5 Sent Done Done false
32369 9016 Acked Read 5 Sent Done Done false
32370 9017 Acked Read 2 Sent Done Done false
32371 9018 Acked Read 10 Sent Done Done false
32372 9019 Acked Write 1 Sent Done Done false
32373 9020 Acked Write 9 Sent Done Done false
32374 9021 Acked Read 4 Sent Done Done false
32375 9022 Acked Read 4 Sent Done Done false
32376 9023 Acked Read 2 Sent Done Done false
32377 9024 Acked Read 5 Sent Done Done false
32378 9025 Acked Read 8 Sent Done Done false
32379 9026 Acked Read 1 Sent Done Done false
32380 9027 Acked Read 1 Sent Done Done false
32381 9028 Acked Write 4 Sent Done Done false
32382 9029 Acked Read 2 Sent Done Done false
32383 9030 Acked Write 2 Sent Done Done false
32384 9031 Acked Read 5 Sent Done Done false
32385 9032 Acked Read 10 Sent Done Done false
32386 9033 Acked Write 9 Sent Done Done false
32387 9034 Acked Write 7 Sent Done Done false
32388 9035 Acked Write 1 Sent Done Done false
32389 9036 Acked Read 6 Sent Done Done false
32390 9037 Acked Read 4 Sent Done Done false
32391 9038 Acked Write 1 Sent Done Done false
32392 9039 Acked Flush 0 Sent Done Done false
32393 9040 Acked Write 5 Sent Done Done false
32394 9041 Acked Flush 0 Sent Done Done false
32395 9042 Acked Write 1 Sent Done Done false
32396 9043 Acked Read 1 Sent Done Done false
32397 9044 Acked Read 6 Sent Done Done false
32398 9045 Acked Write 2 Sent Done Done false
32399 9046 Acked Write 10 Sent Done Done false
32400 9047 Acked Read 2 Sent Done Done false
32401 9048 Acked Write 2 Sent Done Done false
32402 9049 Acked Write 4 Sent Done Done false
32403 9050 Acked Read 3 Sent Done Done false
32404 9051 Acked Write 6 Sent Done Done false
32405 9052 Acked Write 6 Sent Done Done false
32406 9053 Acked Read 1 Sent Done Done false
32407 9054 Acked Write 4 Sent Done Done false
32408 9055 Acked Write 1 Sent Done Done false
32409 9056 Acked Read 3 Sent Done Done false
32410 9057 Acked Write 3 Sent Done Done false
32411 9058 Acked Write 5 Sent Done Done false
32412 9059 Acked Read 4 Sent Done Done false
32413 9060 Acked Flush 0 Sent Done Done false
32414 9061 Acked Flush 0 Sent Done Done false
32415 9062 Acked Flush 0 Sent Done Done false
32416 9063 Acked Read 4 Sent Done Done false
32417 9064 Acked Read 7 Sent Done Done false
32418 9065 Acked Read 4 Sent Done Done false
32419 9066 Acked Read 10 Sent Done Done false
32420 9067 Acked Read 2 Sent Done Done false
32421 9068 Acked Flush 0 Sent Done Done false
32422 9069 Acked Flush 0 Sent Done Done false
32423 9070 Acked Read 6 Sent Done Done false
32424 9071 Acked Read 9 Sent Done Done false
32425 9072 Acked Write 10 Sent Done Done false
32426 9073 Acked Read 8 Sent Done Done false
32427 9074 Acked Write 4 Sent Done Done false
32428 9075 Acked Write 7 Sent Done Done false
32429 9076 Acked Write 2 Sent Done Done false
32430 9077 Acked Write 7 Sent Done Done false
32431 9078 Acked Write 7 Sent Done Done false
32432 9079 Acked Write 5 Sent Done Done false
32433 9080 Acked Write 8 Sent Done Done false
32434 9081 Acked Read 8 Sent Done Done false
32435 9082 Acked Write 9 Sent Done Done false
32436 9083 Acked Write 6 Sent Done Done false
32437 9084 Acked Write 4 Sent Done Done false
32438 9085 Acked Write 6 Sent Done Done false
32439 9086 Acked Flush 0 Sent Done Done false
32440 9087 Acked Read 4 Sent Done Done false
32441 9088 Acked Read 10 Sent Done Done false
32442 9089 Acked Read 7 Sent Done Done false
32443 9090 Acked Read 7 Sent Done Done false
32444 9091 Acked Write 3 Sent Done Done false
32445 9092 Acked Write 1 Sent Done Done false
32446 9093 Acked Read 3 Sent Done Done false
32447 9094 Acked Write 2 Sent Done Done false
32448 9095 Acked Write 6 Sent Done Done false
32449 9096 Acked Flush 0 Sent Done Done false
32450 9097 Acked Write 1 Sent Done Done false
32451 9098 Acked Write 9 Sent Done Done false
32452 9099 Acked Read 7 Sent Done Done false
32453 9100 Acked Flush 0 Sent Done Done false
32454 9101 Acked Write 5 Sent Done Done false
32455 9102 Acked Write 4 Sent Done Done false
32456 9103 Acked Read 10 Sent Done Done false
32457 9104 Acked Write 3 Sent Done Done false
32458 9105 Acked Read 3 Sent Done Done false
32459 9106 Acked Read 4 Sent Done Done false
32460 9107 Acked Write 2 Sent Done Done false
32461 9108 Acked Write 8 Sent Done Done false
32462 9109 Acked Read 7 Sent Done Done false
32463 9110 Acked Read 7 Sent Done Done false
32464 9111 Acked Read 6 Sent Done Done false
32465 9112 Acked Write 10 Sent Done Done false
32466 9113 Acked Write 4 Sent Done Done false
32467 9114 Acked Write 2 Sent Done Done false
32468 9115 Acked Write 2 Sent Done Done false
32469 9116 Acked Write 5 Sent Done Done false
32470 9117 Acked Write 6 Sent Done Done false
32471 9118 Acked Read 5 Sent Done Done false
32472 9119 Acked Flush 0 Sent Done Done false
32473 9120 Acked Write 3 Sent Done Done false
32474 9121 Acked Read 2 Sent Done Done false
32475 9122 Acked Read 1 Sent Done Done false
32476 9123 Acked Read 7 Sent Done Done false
32477 9124 Acked Read 3 Sent Done Done false
32478 9125 Acked Flush 0 Sent Done Done false
32479 9126 Acked Read 2 Sent Done Done false
32480 9127 Acked Read 3 Sent Done Done false
32481 9128 Acked Write 6 Sent Done Done false
32482 9129 Acked Read 2 Sent Done Done false
32483 9130 Acked Read 8 Sent Done Done false
32484 9131 Acked Read 4 Sent Done Done false
32485 9132 Acked Read 7 Sent Done Done false
32486 9133 Acked Read 8 Sent Done Done false
32487 9134 Acked Read 7 Sent Done Done false
32488 9135 Acked Write 9 Sent Done Done false
32489 9136 Acked Read 1 Sent Done Done false
32490 9137 Acked Read 5 Sent Done Done false
32491 9138 Acked Read 5 Sent Done Done false
32492 9139 Acked Write 1 Sent Done Done false
32493 9140 Acked Write 5 Sent Done Done false
32494 9141 Acked Write 6 Sent Done Done false
32495 9142 Acked Read 4 Sent Done Done false
32496 9143 Acked Read 4 Sent Done Done false
32497 9144 Acked Read 5 Sent Done Done false
32498 9145 Acked Read 2 Sent Done Done false
32499 9146 Acked Read 3 Sent Done Done false
32500 9147 Acked Flush 0 Sent Done Done false
32501 9148 Acked Write 8 Sent Done Done false
32502 9149 Acked Write 10 Sent Done Done false
32503 9150 Acked Flush 0 Sent Done Done false
32504 STATES DS:0 DS:1 DS:2 TOTAL
32505 Sent 301 0 0 301
32506 Done 0 301 301 602
32507 Skipped 0 0 0 0
32508 Error 0 0 0 0
32509 Last Flush: Some(JobId(8849)) Some(JobId(9150)) Some(JobId(9150))
32510 Downstairs last five completed: 8849 8848 8847 8846 8845
32511 Upstairs last five completed: 9150 9149 9148 9147 9146
32512 CLIENT: Up:0 ds:301 act:2
325132025-03-17T22:23:34.596ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
325142025-03-17T22:23:35.591ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
325152025-03-17T22:23:35.852ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
325162025-03-17T22:23:35.852ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:64346) = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
325172025-03-17T22:23:35.853ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
325182025-03-17T22:23:35.854ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
325192025-03-17T22:23:35.854ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
325202025-03-17T22:23:35.854ZINFOcrucible: send last flush ID to this DS: Some(JobId(8849)) = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
325212025-03-17T22:23:35.854ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
325222025-03-17T22:23:35.854ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(8849)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
325232025-03-17T22:23:35.854ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(8849)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
32524 ----------------------------------------------------------------
32525 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
32526 Downstairs last five completed: 9150 9149 9148 9147 9146
32527 Upstairs last five completed: 9150 9149 9148 9147 9146
32528 CLIENT: Up:0 ds:0 act:3
32529 Replay: All jobs finished
32530 Main task has work to do, go find it
32531 got dsc Stop(0)
32532 stop 0
32533 Mar 17 22:23:38.098 INFO request completed, latency_us: 420, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 5ef9cad6-65ca-4a96-8d0d-d609598e0519, remote_addr: 127.0.0.1:60423[8810] Got stop action so:false kr:false
32534 , local_addr: 127.0.0.1:9998
32535 Mar 17 22:23:38.099 INFO request completed, latency_us: 119, response_code: 200, uri: /state/cid/0, method: GET, req_id: c0427553-ea16-4df6-b6eb-f98ebdf1d651, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
325362025-03-17T22:23:38.099ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
325372025-03-17T22:23:38.099ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
325382025-03-17T22:23:38.099ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
325392025-03-17T22:23:38.099ZINFOcrucible: client task is exiting = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
32540 [8810] Exited with: ExitStatus(unix_wait_status(9))
32541 [8810][0] reports Exit
32542 Mar 17 22:23:42.101 INFO request completed, latency_us: 231, response_code: 200, uri: /state/cid/0, method: GET, req_id: 465814c1-bd11-42e7-ab57-d54670f99eb7, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
32543 001/300 Write block 3437 len 32768 data: 8 6 6 6 6 5 6 7
32544 002/300 Read block 681 len 40960
32545 003/300 Flush
32546 004/300 Read block 2781 len 32768
32547 005/300 Read block 1959 len 16384
32548 006/300 Read block 1057 len 4096
32549 007/300 Read block 2203 len 20480
32550 008/300 Write block 480 len 32768 data: 5 6 6 7 9 9 7 5
32551 009/300 Flush
32552 010/300 Read block 1544 len 36864
32553 011/300 Read block 3534 len 32768
32554 012/300 Read block 3428 len 12288
32555 013/300 Write block 53 len 24576 data: 5 5 5 6 6 6
32556 014/300 Flush
32557 015/300 Flush
32558 016/300 Read block 3161 len 8192
32559 017/300 Read block 3548 len 4096
32560 018/300 Read block 289 len 36864
32561 019/300 Read block 1624 len 40960
32562 020/300 Write block 2658 len 8192 data: 5 6
32563 021/300 Write block 2339 len 24576 data: 7 6 6 6 6 6
32564 022/300 Write block 1251 len 16384 data: 3 6 7 8
32565 023/300 Read block 4594 len 36864
32566 024/300 Read block 2970 len 20480
32567 025/300 Write block 827 len 16384 data: 3 3 6 5
32568 026/300 Write block 946 len 4096 data: 6
32569 027/300 Flush
32570 028/300 Flush
32571 029/300 Read block 1421 len 24576
32572 030/300 Write block 4828 len 16384 data: 5 5 5 5
32573 031/300 Flush
32574 032/300 Read block 3117 len 20480
32575 033/300 Read block 2100 len 12288
32576 034/300 Read block 1845 len 36864
32577 035/300 Read block 2872 len 40960
32578 036/300 Write block 2505 len 12288 data: 3 4 4
32579 037/300 Write block 3558 len 36864 data: 6 6 6 6 6 6 8 8 8
32580 038/300 Write block 2017 len 24576 data: 6 4 4 5 5 5
32581 039/300 Write block 2095 len 32768 data: 2 3 3 4 4 3 3 3
32582 040/300 Write block 4084 len 36864 data: 7 8 8 10 6 6 6 5 5
32583 041/300 Write block 189 len 12288 data: 7 6 5
32584 042/300 Write block 1807 len 40960 data: 8 7 6 5 4 4 4 4 5 5
32585 043/300 Write block 3200 len 8192 data: 6 6
32586 044/300 Read block 1529 len 8192
32587 045/300 Read block 2347 len 36864
32588 046/300 Write block 3181 len 24576 data: 3 5 4 3 2 3
32589 047/300 Write block 2617 len 8192 data: 6 6
32590 048/300 Flush
32591 049/300 Flush
32592 050/300 Write block 716 len 36864 data: 5 4 4 5 6 8 7 5 5
32593 051/300 Write block 590 len 24576 data: 10 9 11 9 11 11
32594 052/300 Write block 2531 len 36864 data: 7 6 7 8 7 6 5 6 4
32595 053/300 Read block 2211 len 8192
32596 054/300 Write block 4318 len 32768 data: 5 5 4 4 4 3 3 4
32597 055/300 Read block 4876 len 12288
32598 056/300 Read block 4058 len 28672
32599 057/300 Flush
32600 058/300 Write block 3873 len 36864 data: 4 4 4 4 4 4 4 5 5
32601 059/300 Write block 1404 len 20480 data: 9 8 9 8 6
32602 060/300 Read block 3776 len 20480
32603 061/300 Write block 117 len 40960 data: 7 6 7 7 7 8 8 9 7 8
32604 062/300 Write block 960 len 16384 data: 7 6 5 4
32605 063/300 Write block 4514 len 24576 data: 6 6 7 5 4 4
32606 064/300 Write block 3433 len 20480 data: 8 9 8 7 9
32607 065/300 Write block 2767 len 4096 data: 7
32608 066/300 Read block 3944 len 40960
32609 067/300 Read block 2330 len 36864
32610 068/300 Read block 2640 len 8192
32611 069/300 Read block 871 len 32768
32612 070/300 Flush
32613 071/300 Write block 4423 len 24576 data: 9 10 10 9 9 6
32614 072/300 Write block 2007 len 16384 data: 6 8 5 4
32615 073/300 Read block 3200 len 16384
32616 074/300 Read block 923 len 40960
32617 075/300 Read block 4148 len 24576
32618 076/300 Read block 3998 len 32768
32619 077/300 Write block 1123 len 20480 data: 3 3 4 5 7
32620 078/300 Read block 2884 len 20480
32621 079/300 Write block 303 len 12288 data: 4 5 5
32622 080/300 Read block 862 len 20480
32623 081/300 Read block 4195 len 16384
32624 082/300 Flush
32625 083/300 Write block 3994 len 4096 data: 5
32626 084/300 Write block 4396 len 4096 data: 4
32627 085/300 Write block 3005 len 20480 data: 7 9 9 6 6
32628 086/300 Read block 325 len 32768
32629 087/300 Flush
32630 088/300 Read block 950 len 12288
32631 089/300 Write block 2361 len 8192 data: 6 6
32632 090/300 Write block 1831 len 36864 data: 8 6 6 6 5 4 5 5 5
32633 091/300 Read block 3766 len 8192
32634 092/300 Read block 4391 len 4096
32635 093/300 Write block 873 len 36864 data: 5 5 5 4 5 4 5 5 5
32636 094/300 Read block 662 len 8192
32637 095/300 Read block 4661 len 32768
32638 096/300 Read block 598 len 28672
32639 097/300 Write block 3484 len 40960 data: 3 3 2 4 4 5 8 10 11 10
32640 098/300 Read block 26 len 28672
32641 099/300 Read block 2548 len 4096
32642 100/300 Write block 1608 len 40960 data: 4 5 5 6 6 7 7 7 7 7
32643 101/300 Read block 878 len 40960
32644 102/300 Read block 4386 len 32768
32645 103/300 Read block 4162 len 20480
32646 104/300 Write block 1606 len 20480 data: 3 3 5 6 6
32647 105/300 Read block 4100 len 20480
32648 106/300 Read block 491 len 28672
32649 107/300 Read block 415 len 24576
32650 108/300 Write block 4908 len 16384 data: 3 4 4 4
32651 109/300 Write block 700 len 8192 data: 6 6
32652 110/300 Write block 946 len 20480 data: 7 6 5 5 4
32653 111/300 Write block 2992 len 32768 data: 7 7 6 5 5 5 7 7
32654 112/300 Write block 1100 len 28672 data: 6 6 8 6 6 5 5
32655 113/300 Read block 4101 len 20480
32656 114/300 Write block 277 len 4096 data: 5
32657 115/300 Read block 3913 len 32768
32658 116/300 Read block 430 len 40960
32659 117/300 Read block 2997 len 4096
32660 118/300 Read block 1280 len 32768
32661 119/300 Read block 128 len 36864
32662 120/300 Read block 131 len 28672
32663 121/300 Read block 4784 len 32768
32664 122/300 Write block 1008 len 40960 data: 5 5 6 7 6 5 6 6 5 6
32665 123/300 Write block 2293 len 4096 data: 6
32666 124/300 Read block 730 len 12288
32667 125/300 Read block 4587 len 4096
32668 126/300 Read block 1567 len 20480
32669 127/300 Flush
32670 128/300 Write block 2247 len 24576 data: 7 7 6 5 4 4
32671 129/300 Read block 4093 len 36864
32672 130/300 Read block 2056 len 16384
32673 131/300 Read block 3623 len 40960
32674 132/300 Read block 4854 len 4096
32675 133/300 Write block 2318 len 20480 data: 3 3 4 3 3
32676 134/300 Read block 2002 len 40960
32677 135/300 Read block 1886 len 28672
32678 136/300 Write block 4726 len 4096 data: 6
32679 137/300 Read block 4033 len 32768
32680 138/300 Write block 4337 len 12288 data: 13 12 12
32681 139/300 Write block 4765 len 12288 data: 8 8 8
32682 140/300 Write block 3928 len 12288 data: 6 4 4
32683 141/300 Flush
32684 142/300 Write block 827 len 4096 data: 4
32685 143/300 Write block 4643 len 16384 data: 6 7 6 4
32686 144/300 Read block 2118 len 32768
32687 145/300 Write block 3339 len 20480 data: 5 4 3 2 2
32688 146/300 Write block 4752 len 4096 data: 7
32689 147/300 Write block 854 len 32768 data: 11 10 8 7 8 8 7 7
32690 148/300 Write block 334 len 32768 data: 8 9 9 9 7 5 4 5
32691 149/300 Read block 4144 len 4096
32692 150/300 Flush
32693 151/300 Write block 3974 len 28672 data: 4 3 4 4 4 3 3
32694 152/300 Read block 3234 len 12288
32695 153/300 Write block 2937 len 16384 data: 6 6 8 10
32696 154/300 Read block 3208 len 16384
32697 155/300 Write block 4233 len 40960 data: 7 6 7 6 5 5 8 7 8 7
32698 156/300 Write block 782 len 24576 data: 4 4 5 6 6 6
32699 157/300 Write block 3949 len 20480 data: 6 6 5 6 7
32700 158/300 Write block 170 len 40960 data: 9 9 8 7 6 4 4 4 4 3
32701 159/300 Write block 181 len 16384 data: 4 4 5 6
32702 160/300 Read block 1300 len 8192
32703 161/300 Write block 1626 len 36864 data: 4 4 4 4 2 3 4 4 4
32704 162/300 Write block 931 len 20480 data: 8 7 6 5 4
32705 163/300 Write block 4856 len 8192 data: 5 8
32706 164/300 Read block 3233 len 8192
32707 165/300 Read block 4545 len 28672
32708 166/300 Read block 1782 len 4096
32709 167/300 Read block 2881 len 32768
32710 168/300 Read block 2855 len 36864
32711 169/300 Read block 3946 len 28672
32712 170/300 Write block 966 len 32768 data: 3 3 3 3 4 4 3 3
32713 171/300 Read block 2143 len 24576
32714 172/300 Write block 3273 len 20480 data: 7 7 9 7 4
32715 173/300 Write block 2847 len 40960 data: 8 7 8 8 7 9 7 6 7 6
32716 174/300 Write block 2787 len 8192 data: 6 6
32717 175/300 Read block 2051 len 12288
32718 176/300 Read block 475 len 32768
32719 177/300 Write block 4101 len 24576 data: 6 4 2 2 6 8
32720 178/300 Read block 1476 len 12288
32721 179/300 Write block 2300 len 24576 data: 8 7 7 5 6 6
32722 180/300 Write block 4116 len 24576 data: 3 5 5 4 4 4
32723 181/300 Write block 583 len 20480 data: 3 4 5 5 6
32724 182/300 Write block 3758 len 4096 data: 3
32725 183/300 Write block 2425 len 28672 data: 6 5 6 4 5 5 4
32726 184/300 Read block 3754 len 16384
32727 185/300 Write block 4580 len 4096 data: 4
32728 186/300 Write block 4364 len 12288 data: 5 5 8
32729 187/300 Write block 3239 len 28672 data: 8 7 5 6 6 6 5
32730 188/300 Flush
32731 189/300 Write block 687 len 20480 data: 4 3 3 3 2
32732 190/300 Read block 392 len 28672
32733 191/300 Read block 3605 len 36864
32734 192/300 Read block 214 len 4096
32735 193/300 Read block 1432 len 24576
32736 194/300 Write block 2718 len 40960 data: 5 5 6 5 3 2 3 3 3 3
32737 195/300 Read block 1935 len 28672
32738 196/300 Write block 1367 len 4096 data: 5
32739 197/300 Read block 314 len 20480
32740 198/300 Read block 4875 len 20480
32741 199/300 Flush
32742 200/300 Read block 2826 len 8192
32743 201/300 Read block 4332 len 16384
32744 202/300 Write block 2352 len 24576 data: 6 6 5 6 7 8
32745 203/300 Flush
32746 204/300 Write block 1901 len 20480 data: 4 4 3 2 2
32747 205/300 Read block 1524 len 12288
32748 206/300 Read block 2527 len 8192
32749 207/300 Read block 1099 len 32768
32750 208/300 Read block 4147 len 36864
32751 209/300 Read block 3113 len 28672
32752 210/300 Write block 2514 len 40960 data: 4 5 5 4 3 3 2 2 3 4
32753 211/300 Write block 2691 len 24576 data: 6 7 6 6 7 9
32754 212/300 Read block 2066 len 16384
32755 213/300 Flush
32756 214/300 Write block 3896 len 40960 data: 9 9 9 10 9 9 8 5 5 3
32757 215/300 Read block 2639 len 4096
32758 216/300 Write block 3939 len 20480 data: 6 7 9 6 4
32759 217/300 Read block 4 len 16384
32760 218/300 Read block 526 len 24576
32761 219/300 Write block 3843 len 32768 data: 5 3 3 4 4 4 4 3
32762 220/300 Write block 2968 len 36864 data: 8 7 5 6 7 8 6 5 4
32763 221/300 Write block 2671 len 28672 data: 9 11 11 11 11 10 9
32764 222/300 Read block 564 len 20480
32765 223/300 Read block 1284 len 28672
32766 224/300 Read block 4397 len 28672
32767 225/300 Read block 385 len 8192
32768 226/300 Read block 2391 len 40960
32769 227/300 Read block 2531 len 4096
32770 228/300 Write block 1086 len 4096 data: 6
32771 229/300 Write block 3384 len 12288 data: 3 4 6
32772 230/300 Write block 4630 len 36864 data: 8 8 7 9 8 7 7 7 7
32773 231/300 Read block 4544 len 16384
32774 232/300 Write block 2554 len 24576 data: 5 6 5 5 5 4
32775 233/300 Read block 566 len 12288
32776 234/300 Read block 3656 len 36864
32777 235/300 Write block 4825 len 32768 data: 7 6 5 6 6 6 6 4
32778 236/300 Read block 1650 len 28672
32779 237/300 Write block 1081 len 12288 data: 6 6 5
32780 238/300 Write block 1747 len 24576 data: 9 7 7 5 5 6
32781 239/300 Flush
32782 240/300 Read block 1365 len 28672
32783 241/300 Write block 2537 len 24576 data: 6 7 5 3 3 4
32784 242/300 Read block 2665 len 8192
32785 243/300 Read block 2025 len 36864
32786 244/300 Read block 1911 len 36864
32787 245/300 Read block 4126 len 4096
32788 246/300 Flush
32789 247/300 Read block 2893 len 4096
32790 248/300 Read block 182 len 8192
32791 249/300 Read block 690 len 16384
32792 250/300 Read block 405 len 16384
32793 251/300 Write block 2171 len 20480 data: 6 6 6 5 5
32794 252/300 Read block 4921 len 16384
32795 253/300 Write block 3478 len 4096 data: 2
32796 254/300 Read block 3805 len 8192
32797 255/300 Write block 4685 len 8192 data: 9 8
32798 256/300 Write block 2295 len 4096 data: 6
32799 257/300 Write block 458 len 40960 data: 5 5 6 6 6 7 9 10 9 7
32800 258/300 Read block 4108 len 16384
32801 259/300 Read block 1180 len 40960
32802 260/300 Read block 4462 len 4096
32803 261/300 Read block 4517 len 24576
32804 262/300 Write block 4513 len 16384 data: 5 7 7 8
32805 263/300 Read block 4516 len 12288
32806 264/300 Write block 612 len 24576 data: 5 7 7 7 6 8
32807 265/300 Write block 2479 len 24576 data: 8 6 6 6 7 6
32808 266/300 Read block 4719 len 8192
32809 267/300 Read block 1920 len 24576
32810 268/300 Write block 4931 len 12288 data: 3 3 2
32811 269/300 Write block 3520 len 8192 data: 3 3
32812 270/300 Read block 3007 len 8192
32813 271/300 Write block 4566 len 12288 data: 4 4 5
32814 272/300 Write block 375 len 28672 data: 8 7 7 6 6 5 4
32815 273/300 Write block 3993 len 4096 data: 5
32816 274/300 Read block 3772 len 12288
32817 275/300 Read block 2738 len 32768
32818 276/300 Read block 4188 len 4096
32819 277/300 Write block 4950 len 32768 data: 7 6 5 5 4 3 3 3
32820 278/300 Write block 4400 len 40960 data: 6 4 4 5 7 8 8 8 8 9
32821 279/300 Read block 3516 len 20480
32822 280/300 Read block 710 len 20480
32823 281/300 Flush
32824 282/300 Write block 1444 len 12288 data: 3 4 4
32825 283/300 Read block 4986 len 8192
32826 284/300 Read block 1151 len 8192
32827 285/300 Read block 4145 len 36864
32828 286/300 Read block 4850 len 16384
32829 287/300 Read block 2595 len 8192
32830 288/300 Write block 2442 len 40960 data: 6 6 5 5 5 4 3 3 5 4
32831 289/300 Read block 165 len 32768
32832 290/300 Write block 3375 len 16384 data: 5 6 6 7
32833 291/300 Read block 1627 len 12288
32834 292/300 Read block 1914 len 36864
32835 293/300 Write block 2340 len 36864 data: 7 7 7 7 7 6 4 4 3
32836 294/300 Write block 2687 len 24576 data: 7 7 6 6 7 8
32837 295/300 Read block 3517 len 4096
32838 296/300 Read block 4134 len 32768
32839 297/300 Read block 2285 len 20480
32840 298/300 Read block 3005 len 28672
32841 299/300 Write block 1589 len 4096 data: 3
32842 300/300 Read block 412 len 28672
32843 301/300 Flush
32844 Main task has work to do, go find it
32845 got dsc Start(0)
32846 start 0
32847 Mar 17 22:23:42.350 INFO request completed, latency_us: 247, response_code: 204, uri: /start/cid/0, method: POST, req_id: f8bd6667-d1f1-4fc7-8f6a-9db12ae0622c, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
32848 Replay: started 0, returned:Ok(())
32849 ----------------------------------------------------------------
32850 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
32851 States: Offline Active Active
32852 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
32853 9151 Acked Write 8 Sent Done Done false
32854 9152 Acked Read 10 Sent Done Done false
32855 9153 Acked Flush 0 Sent Done Done false
32856 9154 Acked Read 8 Sent Done Done false
32857 9155 Acked Read 4 Sent Done Done false
32858 9156 Acked Read 1 Sent Done Done false
32859 9157 Acked Read 5 Sent Done Done false
32860 9158 Acked Write 8 Sent Done Done false
32861 9159 Acked Flush 0 Sent Done Done false
32862 9160 Acked Read 9 Sent Done Done false
32863 9161 Acked Read 8 Sent Done Done false
32864 9162 Acked Read 3 Sent Done Done false
32865 9163 Acked Write 6 Sent Done Done false
32866 9164 Acked Flush 0 Sent Done Done false
32867 9165 Acked Flush 0 Sent Done Done false
32868 9166 Acked Read 2 Sent Done Done false
32869 9167 Acked Read 1 Sent Done Done false
32870 9168 Acked Read 9 Sent Done Done false
32871 9169 Acked Read 10 Sent Done Done false
32872 9170 Acked Write 2 Sent Done Done false
32873 9171 Acked Write 6 Sent Done Done false
32874 9172 Acked Write 4 Sent Done Done false
32875 9173 Acked Read 9 Sent Done Done false
32876 9174 Acked Read 5 Sent Done Done false
32877 9175 Acked Write 4 Sent Done Done false
32878 9176 Acked Write 1 Sent Done Done false
32879 9177 Acked Flush 0 Sent Done Done false
32880 9178 Acked Flush 0 Sent Done Done false
32881 9179 Acked Read 6 Sent Done Done false
32882 9180 Acked Write 4 Sent Done Done false
32883 9181 Acked Flush 0 Sent Done Done false
32884 9182 Acked Read 5 Sent Done Done false
32885 9183 Acked Read 3 Sent Done Done false
32886 9184 Acked Read 9 Sent Done Done false
32887 9185 Acked Read 10 Sent Done Done false
32888 9186 Acked Write 3 Sent Done Done false
32889 9187 Acked Write 9 Sent Done Done false
32890 9188 Acked Write 6 Sent Done Done false
32891 9189 Acked Write 8 Sent Done Done false
32892 9190 Acked Write 9 Sent Done Done false
32893 9191 Acked Write 3 Sent Done Done false
32894 9192 Acked Write 10 Sent Done Done false
32895 9193 Acked Write 2 Sent Done Done false
32896 9194 Acked Read 2 Sent Done Done false
32897 9195 Acked Read 9 Sent Done Done false
32898 9196 Acked Write 6 Sent Done Done false
32899 9197 Acked Write 2 Sent Done Done false
32900 9198 Acked Flush 0 Sent Done Done false
32901 9199 Acked Flush 0 Sent Done Done false
32902 9200 Acked Write 9 Sent Done Done false
32903 9201 Acked Write 6 Sent Done Done false
32904 9202 Acked Write 9 Sent Done Done false
32905 9203 Acked Read 2 Sent Done Done false
32906 9204 Acked Write 8 Sent Done Done false
32907 9205 Acked Read 3 Sent Done Done false
32908 9206 Acked Read 7 Sent Done Done false
32909 9207 Acked Flush 0 Sent Done Done false
32910 9208 Acked Write 9 Sent Done Done false
32911 9209 Acked Write 5 Sent Done Done false
32912 9210 Acked Read 5 Sent Done Done false
32913 9211 Acked Write 10 Sent Done Done false
32914 9212 Acked Write 4 Sent Done Done false
32915 9213 Acked Write 6 Sent Done Done false
32916 9214 Acked Write 5 Sent Done Done false
32917 9215 Acked Write 1 Sent Done Done false
32918 9216 Acked Read 10 Sent Done Done false
32919 9217 Acked Read 9 Sent Done Done false
32920 9218 Acked Read 2 Sent Done Done false
32921 9219 Acked Read 8 Sent Done Done false
32922 9220 Acked Flush 0 Sent Done Done false
32923 9221 Acked Write 6 Sent Done Done false
32924 9222 Acked Write 4 Sent Done Done false
32925 9223 Acked Read 4 Sent Done Done false
32926 9224 Acked Read 10 Sent Done Done false
32927 9225 Acked Read 6 Sent Done Done false
32928 9226 Acked Read 8 Sent Done Done false
32929 9227 Acked Write 5 Sent Done Done false
32930 9228 Acked Read 5 Sent Done Done false
32931 9229 Acked Write 3 Sent Done Done false
32932 9230 Acked Read 5 Sent Done Done false
32933 9231 Acked Read 4 Sent Done Done false
32934 9232 Acked Flush 0 Sent Done Done false
32935 9233 Acked Write 1 Sent Done Done false
32936 9234 Acked Write 1 Sent Done Done false
32937 9235 Acked Write 5 Sent Done Done false
32938 9236 Acked Read 8 Sent Done Done false
32939 9237 Acked Flush 0 Sent Done Done false
32940 9238 Acked Read 3 Sent Done Done false
32941 9239 Acked Write 2 Sent Done Done false
32942 9240 Acked Write 9 Sent Done Done false
32943 9241 Acked Read 2 Sent Done Done false
32944 9242 Acked Read 1 Sent Done Done false
32945 9243 Acked Write 9 Sent Done Done false
32946 9244 Acked Read 2 Sent Done Done false
32947 9245 Acked Read 8 Sent Done Done false
32948 9246 Acked Read 7 Sent Done Done false
32949 9247 Acked Write 10 Sent Done Done false
32950 9248 Acked Read 7 Sent Done Done false
32951 9249 Acked Read 1 Sent Done Done false
32952 9250 Acked Write 10 Sent Done Done false
32953 9251 Acked Read 10 Sent Done Done false
32954 9252 Acked Read 8 Sent Done Done false
32955 9253 Acked Read 5 Sent Done Done false
32956 9254 Acked Write 5 Sent Done Done false
32957 9255 Acked Read 5 Sent Done Done false
32958 9256 Acked Read 7 Sent Done Done false
32959 9257 Acked Read 6 Sent Done Done false
32960 9258 Acked Write 4 Sent Done Done false
32961 9259 Acked Write 2 Sent Done Done false
32962 9260 Acked Write 5 Sent Done Done false
32963 9261 Acked Write 8 Sent Done Done false
32964 9262 Acked Write 7 Sent Done Done false
32965 9263 Acked Read 5 Sent Done Done false
32966 9264 Acked Write 1 Sent Done Done false
32967 9265 Acked Read 8 Sent Done Done false
32968 9266 Acked Read 10 Sent Done Done false
32969 9267 Acked Read 1 Sent Done Done false
32970 9268 Acked Read 8 Sent Done Done false
32971 9269 Acked Read 9 Sent Done Done false
32972 9270 Acked Read 7 Sent Done Done false
32973 9271 Acked Read 8 Sent Done Done false
32974 9272 Acked Write 10 Sent Done Done false
32975 9273 Acked Write 1 Sent Done Done false
32976 9274 Acked Read 3 Sent Done Done false
32977 9275 Acked Read 1 Sent Done Done false
32978 9276 Acked Read 5 Sent Done Done false
32979 9277 Acked Flush 0 Sent Done Done false
32980 9278 Acked Write 6 Sent Done Done false
32981 9279 Acked Read 9 Sent Done Done false
32982 9280 Acked Read 4 Sent Done Done false
32983 9281 Acked Read 10 Sent Done Done false
32984 9282 Acked Read 1 Sent Done Done false
32985 9283 Acked Write 5 Sent Done Done false
32986 9284 Acked Read 10 Sent Done Done false
32987 9285 Acked Read 7 Sent Done Done false
32988 9286 Acked Write 1 Sent Done Done false
32989 9287 Acked Read 8 Sent Done Done false
32990 9288 Acked Write 3 Sent Done Done false
32991 9289 Acked Write 3 Sent Done Done false
32992 9290 Acked Write 3 Sent Done Done false
32993 9291 Acked Flush 0 Sent Done Done false
32994 9292 Acked Write 1 Sent Done Done false
32995 9293 Acked Write 4 Sent Done Done false
32996 9294 Acked Read 8 Sent Done Done false
32997 9295 Acked Write 5 Sent Done Done false
32998 9296 Acked Write 1 Sent Done Done false
32999 9297 Acked Write 8 Sent Done Done false
33000 9298 Acked Write 8 Sent Done Done false
33001 9299 Acked Read 1 Sent Done Done false
33002 9300 Acked Flush 0 Sent Done Done false
33003 9301 Acked Write 7 Sent Done Done false
33004 9302 Acked Read 3 Sent Done Done false
33005 9303 Acked Write 4 Sent Done Done false
33006 9304 Acked Read 4 Sent Done Done false
33007 9305 Acked Write 10 Sent Done Done false
33008 9306 Acked Write 6 Sent Done Done false
33009 9307 Acked Write 5 Sent Done Done false
33010 9308 Acked Write 10 Sent Done Done false
33011 9309 Acked Write 4 Sent Done Done false
33012 9310 Acked Read 2 Sent Done Done false
33013 9311 Acked Write 9 Sent Done Done false
33014 9312 Acked Write 5 Sent Done Done false
33015 9313 Acked Write 2 Sent Done Done false
33016 9314 Acked Read 2 Sent Done Done false
33017 9315 Acked Read 7 Sent Done Done false
33018 9316 Acked Read 1 Sent Done Done false
33019 9317 Acked Read 8 Sent Done Done false
33020 9318 Acked Read 9 Sent Done Done false
33021 9319 Acked Read 7 Sent Done Done false
33022 9320 Acked Write 8 Sent Done Done false
33023 9321 Acked Read 6 Sent Done Done false
33024 9322 Acked Write 5 Sent Done Done false
33025 9323 Acked Write 10 Sent Done Done false
33026 9324 Acked Write 2 Sent Done Done false
33027 9325 Acked Read 3 Sent Done Done false
33028 9326 Acked Read 8 Sent Done Done false
33029 9327 Acked Write 6 Sent Done Done false
33030 9328 Acked Read 3 Sent Done Done false
33031 9329 Acked Write 6 Sent Done Done false
33032 9330 Acked Write 6 Sent Done Done false
33033 9331 Acked Write 5 Sent Done Done false
33034 9332 Acked Write 1 Sent Done Done false
33035 9333 Acked Write 7 Sent Done Done false
33036 9334 Acked Read 4 Sent Done Done false
33037 9335 Acked Write 1 Sent Done Done false
33038 9336 Acked Write 3 Sent Done Done false
33039 9337 Acked Write 7 Sent Done Done false
33040 9338 Acked Flush 0 Sent Done Done false
33041 9339 Acked Write 5 Sent Done Done false
33042 9340 Acked Read 7 Sent Done Done false
33043 9341 Acked Read 9 Sent Done Done false
33044 9342 Acked Read 1 Sent Done Done false
33045 9343 Acked Read 6 Sent Done Done false
33046 9344 Acked Write 10 Sent Done Done false
33047 9345 Acked Read 7 Sent Done Done false
33048 9346 Acked Write 1 Sent Done Done false
33049 9347 Acked Read 5 Sent Done Done false
33050 9348 Acked Read 5 Sent Done Done false
33051 9349 Acked Flush 0 Sent Done Done false
33052 9350 Acked Read 2 Sent Done Done false
33053 9351 Acked Read 4 Sent Done Done false
33054 9352 Acked Write 6 Sent Done Done false
33055 9353 Acked Flush 0 Sent Done Done false
33056 9354 Acked Write 5 Sent Done Done false
33057 9355 Acked Read 3 Sent Done Done false
33058 9356 Acked Read 2 Sent Done Done false
33059 9357 Acked Read 8 Sent Done Done false
33060 9358 Acked Read 9 Sent Done Done false
33061 9359 Acked Read 7 Sent Done Done false
33062 9360 Acked Write 10 Sent Done Done false
33063 9361 Acked Write 6 Sent Done Done false
33064 9362 Acked Read 4 Sent Done Done false
33065 9363 Acked Flush 0 Sent Done Done false
33066 9364 Acked Write 10 Sent Done Done false
33067 9365 Acked Read 1 Sent Done Done false
33068 9366 Acked Write 5 Sent Done Done false
33069 9367 Acked Read 4 Sent Done Done false
33070 9368 Acked Read 6 Sent Done Done false
33071 9369 Acked Write 8 Sent Done Done false
33072 9370 Acked Write 9 Sent Done Done false
33073 9371 Acked Write 7 Sent Done Done false
33074 9372 Acked Read 5 Sent Done Done false
33075 9373 Acked Read 7 Sent Done Done false
33076 9374 Acked Read 7 Sent Done Done false
33077 9375 Acked Read 2 Sent Done Done false
33078 9376 Acked Read 10 Sent Done Done false
33079 9377 Acked Read 1 Sent Done Done false
33080 9378 Acked Write 1 Sent Done Done false
33081 9379 Acked Write 3 Sent Done Done false
33082 9380 Acked Write 9 Sent Done Done false
33083 9381 Acked Read 4 Sent Done Done false
33084 9382 Acked Write 6 Sent Done Done false
33085 9383 Acked Read 3 Sent Done Done false
33086 9384 Acked Read 9 Sent Done Done false
33087 9385 Acked Write 8 Sent Done Done false
33088 9386 Acked Read 7 Sent Done Done false
33089 9387 Acked Write 3 Sent Done Done false
33090 9388 Acked Write 6 Sent Done Done false
33091 9389 Acked Flush 0 Sent Done Done false
33092 9390 Acked Read 7 Sent Done Done false
33093 9391 Acked Write 6 Sent Done Done false
33094 9392 Acked Read 2 Sent Done Done false
33095 9393 Acked Read 9 Sent Done Done false
33096 9394 Acked Read 9 Sent Done Done false
33097 9395 Acked Read 1 Sent Done Done false
33098 9396 Acked Flush 0 Sent Done Done false
33099 9397 Acked Read 1 Sent Done Done false
33100 9398 Acked Read 2 Sent Done Done false
33101 9399 Acked Read 4 Sent Done Done false
33102 9400 Acked Read 4 Sent Done Done false
33103 9401 Acked Write 5 Sent Done Done false
33104 9402 Acked Read 4 Sent Done Done false
33105 9403 Acked Write 1 Sent Done Done false
33106 9404 Acked Read 2 Sent Done Done false
33107 9405 Acked Write 2 Sent Done Done false
33108 9406 Acked Write 1 Sent Done Done false
33109 9407 Acked Write 10 Sent Done Done false
33110 9408 Acked Read 4 Sent Done Done false
33111 9409 Acked Read 10 Sent Done Done false
33112 9410 Acked Read 1 Sent Done Done false
33113 9411 Acked Read 6 Sent Done Done false
33114 9412 Acked Write 4 Sent Done Done false
33115 9413 Acked Read 3 Sent Done Done false
33116 9414 Acked Write 6 Sent Done Done false
33117 9415 Acked Write 6 Sent Done Done false
33118 9416 Acked Read 2 Sent Done Done false
33119 9417 Acked Read 6 Sent Done Done false
33120 9418 Acked Write 3 Sent Done Done false
33121 9419 Acked Write 2 Sent Done Done false
33122 9420 Acked Read 2 Sent Done Done false
33123 9421 Acked Write 3 Sent Done Done false
33124 9422 Acked Write 7 Sent Done Done false
33125 9423 Acked Write 1 Sent Done Done false
33126 9424 Acked Read 3 Sent Done Done false
33127 9425 Acked Read 8 Sent Done Done false
33128 9426 Acked Read 1 Sent Done Done false
33129 9427 Acked Write 8 Sent Done Done false
33130 9428 Acked Write 10 Sent Done Done false
33131 9429 Acked Read 5 Sent Done Done false
33132 9430 Acked Read 5 Sent Done Done false
33133 9431 Acked Flush 0 Sent Done Done false
33134 9432 Acked Write 3 Sent Done Done false
33135 9433 Acked Read 2 Sent Done Done false
33136 9434 Acked Read 2 Sent Done Done false
33137 9435 Acked Read 9 Sent Done Done false
33138 9436 Acked Read 4 Sent Done Done false
33139 9437 Acked Read 2 Sent Done Done false
33140 9438 Acked Write 10 Sent Done Done false
33141 9439 Acked Read 8 Sent Done Done false
33142 9440 Acked Write 4 Sent Done Done false
33143 9441 Acked Read 3 Sent Done Done false
33144 9442 Acked Read 9 Sent Done Done false
33145 9443 Acked Write 9 Sent Done Done false
33146 9444 Acked Write 6 Sent Done Done false
33147 9445 Acked Read 1 Sent Done Done false
33148 9446 Acked Read 8 Sent Done Done false
33149 9447 Acked Read 5 Sent Done Done false
33150 9448 Acked Read 7 Sent Done Done false
33151 9449 Acked Write 1 Sent Done Done false
33152 9450 Acked Read 7 Sent Done Done false
33153 9451 Acked Flush 0 Sent Done Done false
33154 STATES DS:0 DS:1 DS:2 TOTAL
33155 Sent 301 0 0 301
33156 Done 0 301 301 602
33157 Skipped 0 0 0 0
33158 Error 0 0 0 0
33159 Last Flush: Some(JobId(9150)) Some(JobId(9451)) Some(JobId(9451))
33160 Downstairs last five completed: 9150 9149 9148 9147 9146
33161 Upstairs last five completed: 9451 9450 9449 9448 9447
33162 CLIENT: Up:0 ds:301 act:2
331632025-03-17T22:23:43.296ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
331642025-03-17T22:23:44.296ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
331652025-03-17T22:23:45.592ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
331662025-03-17T22:23:46.299ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
33167 ----------------------------------------------------------------
33168 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
33169 States: Offline Active Active
33170 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33171 9151 Acked Write 8 Sent Done Done false
33172 9152 Acked Read 10 Sent Done Done false
33173 9153 Acked Flush 0 Sent Done Done false
33174 9154 Acked Read 8 Sent Done Done false
33175 9155 Acked Read 4 Sent Done Done false
33176 9156 Acked Read 1 Sent Done Done false
33177 9157 Acked Read 5 Sent Done Done false
33178 9158 Acked Write 8 Sent Done Done false
33179 9159 Acked Flush 0 Sent Done Done false
33180 9160 Acked Read 9 Sent Done Done false
33181 9161 Acked Read 8 Sent Done Done false
33182 9162 Acked Read 3 Sent Done Done false
33183 9163 Acked Write 6 Sent Done Done false
33184 9164 Acked Flush 0 Sent Done Done false
33185 9165 Acked Flush 0 Sent Done Done false
33186 9166 Acked Read 2 Sent Done Done false
33187 9167 Acked Read 1 Sent Done Done false
33188 9168 Acked Read 9 Sent Done Done false
33189 9169 Acked Read 10 Sent Done Done false
33190 9170 Acked Write 2 Sent Done Done false
33191 9171 Acked Write 6 Sent Done Done false
33192 9172 Acked Write 4 Sent Done Done false
33193 9173 Acked Read 9 Sent Done Done false
33194 9174 Acked Read 5 Sent Done Done false
33195 9175 Acked Write 4 Sent Done Done false
33196 9176 Acked Write 1 Sent Done Done false
33197 9177 Acked Flush 0 Sent Done Done false
33198 9178 Acked Flush 0 Sent Done Done false
33199 9179 Acked Read 6 Sent Done Done false
33200 9180 Acked Write 4 Sent Done Done false
33201 9181 Acked Flush 0 Sent Done Done false
33202 9182 Acked Read 5 Sent Done Done false
33203 9183 Acked Read 3 Sent Done Done false
33204 9184 Acked Read 9 Sent Done Done false
33205 9185 Acked Read 10 Sent Done Done false
33206 9186 Acked Write 3 Sent Done Done false
33207 9187 Acked Write 9 Sent Done Done false
33208 9188 Acked Write 6 Sent Done Done false
33209 9189 Acked Write 8 Sent Done Done false
33210 9190 Acked Write 9 Sent Done Done false
33211 9191 Acked Write 3 Sent Done Done false
33212 9192 Acked Write 10 Sent Done Done false
33213 9193 Acked Write 2 Sent Done Done false
33214 9194 Acked Read 2 Sent Done Done false
33215 9195 Acked Read 9 Sent Done Done false
33216 9196 Acked Write 6 Sent Done Done false
33217 9197 Acked Write 2 Sent Done Done false
33218 9198 Acked Flush 0 Sent Done Done false
33219 9199 Acked Flush 0 Sent Done Done false
33220 9200 Acked Write 9 Sent Done Done false
33221 9201 Acked Write 6 Sent Done Done false
33222 9202 Acked Write 9 Sent Done Done false
33223 9203 Acked Read 2 Sent Done Done false
33224 9204 Acked Write 8 Sent Done Done false
33225 9205 Acked Read 3 Sent Done Done false
33226 9206 Acked Read 7 Sent Done Done false
33227 9207 Acked Flush 0 Sent Done Done false
33228 9208 Acked Write 9 Sent Done Done false
33229 9209 Acked Write 5 Sent Done Done false
33230 9210 Acked Read 5 Sent Done Done false
33231 9211 Acked Write 10 Sent Done Done false
33232 9212 Acked Write 4 Sent Done Done false
33233 9213 Acked Write 6 Sent Done Done false
33234 9214 Acked Write 5 Sent Done Done false
33235 9215 Acked Write 1 Sent Done Done false
33236 9216 Acked Read 10 Sent Done Done false
33237 9217 Acked Read 9 Sent Done Done false
33238 9218 Acked Read 2 Sent Done Done false
33239 9219 Acked Read 8 Sent Done Done false
33240 9220 Acked Flush 0 Sent Done Done false
33241 9221 Acked Write 6 Sent Done Done false
33242 9222 Acked Write 4 Sent Done Done false
33243 9223 Acked Read 4 Sent Done Done false
33244 9224 Acked Read 10 Sent Done Done false
33245 9225 Acked Read 6 Sent Done Done false
33246 9226 Acked Read 8 Sent Done Done false
33247 9227 Acked Write 5 Sent Done Done false
33248 9228 Acked Read 5 Sent Done Done false
33249 9229 Acked Write 3 Sent Done Done false
33250 9230 Acked Read 5 Sent Done Done false
33251 9231 Acked Read 4 Sent Done Done false
33252 9232 Acked Flush 0 Sent Done Done false
33253 9233 Acked Write 1 Sent Done Done false
33254 9234 Acked Write 1 Sent Done Done false
33255 9235 Acked Write 5 Sent Done Done false
33256 9236 Acked Read 8 Sent Done Done false
33257 9237 Acked Flush 0 Sent Done Done false
33258 9238 Acked Read 3 Sent Done Done false
33259 9239 Acked Write 2 Sent Done Done false
33260 9240 Acked Write 9 Sent Done Done false
33261 9241 Acked Read 2 Sent Done Done false
33262 9242 Acked Read 1 Sent Done Done false
33263 9243 Acked Write 9 Sent Done Done false
33264 9244 Acked Read 2 Sent Done Done false
33265 9245 Acked Read 8 Sent Done Done false
33266 9246 Acked Read 7 Sent Done Done false
33267 9247 Acked Write 10 Sent Done Done false
33268 9248 Acked Read 7 Sent Done Done false
33269 9249 Acked Read 1 Sent Done Done false
33270 9250 Acked Write 10 Sent Done Done false
33271 9251 Acked Read 10 Sent Done Done false
33272 9252 Acked Read 8 Sent Done Done false
33273 9253 Acked Read 5 Sent Done Done false
33274 9254 Acked Write 5 Sent Done Done false
33275 9255 Acked Read 5 Sent Done Done false
33276 9256 Acked Read 7 Sent Done Done false
33277 9257 Acked Read 6 Sent Done Done false
33278 9258 Acked Write 4 Sent Done Done false
33279 9259 Acked Write 2 Sent Done Done false
33280 9260 Acked Write 5 Sent Done Done false
33281 9261 Acked Write 8 Sent Done Done false
33282 9262 Acked Write 7 Sent Done Done false
33283 9263 Acked Read 5 Sent Done Done false
33284 9264 Acked Write 1 Sent Done Done false
33285 9265 Acked Read 8 Sent Done Done false
33286 9266 Acked Read 10 Sent Done Done false
33287 9267 Acked Read 1 Sent Done Done false
33288 9268 Acked Read 8 Sent Done Done false
33289 9269 Acked Read 9 Sent Done Done false
33290 9270 Acked Read 7 Sent Done Done false
33291 9271 Acked Read 8 Sent Done Done false
33292 9272 Acked Write 10 Sent Done Done false
33293 9273 Acked Write 1 Sent Done Done false
33294 9274 Acked Read 3 Sent Done Done false
33295 9275 Acked Read 1 Sent Done Done false
33296 9276 Acked Read 5 Sent Done Done false
33297 9277 Acked Flush 0 Sent Done Done false
33298 9278 Acked Write 6 Sent Done Done false
33299 9279 Acked Read 9 Sent Done Done false
33300 9280 Acked Read 4 Sent Done Done false
33301 9281 Acked Read 10 Sent Done Done false
33302 9282 Acked Read 1 Sent Done Done false
33303 9283 Acked Write 5 Sent Done Done false
33304 9284 Acked Read 10 Sent Done Done false
33305 9285 Acked Read 7 Sent Done Done false
33306 9286 Acked Write 1 Sent Done Done false
33307 9287 Acked Read 8 Sent Done Done false
33308 9288 Acked Write 3 Sent Done Done false
33309 9289 Acked Write 3 Sent Done Done false
33310 9290 Acked Write 3 Sent Done Done false
33311 9291 Acked Flush 0 Sent Done Done false
33312 9292 Acked Write 1 Sent Done Done false
33313 9293 Acked Write 4 Sent Done Done false
33314 9294 Acked Read 8 Sent Done Done false
33315 9295 Acked Write 5 Sent Done Done false
33316 9296 Acked Write 1 Sent Done Done false
33317 9297 Acked Write 8 Sent Done Done false
33318 9298 Acked Write 8 Sent Done Done false
33319 9299 Acked Read 1 Sent Done Done false
33320 9300 Acked Flush 0 Sent Done Done false
33321 9301 Acked Write 7 Sent Done Done false
33322 9302 Acked Read 3 Sent Done Done false
33323 9303 Acked Write 4 Sent Done Done false
33324 9304 Acked Read 4 Sent Done Done false
33325 9305 Acked Write 10 Sent Done Done false
33326 9306 Acked Write 6 Sent Done Done false
33327 9307 Acked Write 5 Sent Done Done false
33328 9308 Acked Write 10 Sent Done Done false
33329 9309 Acked Write 4 Sent Done Done false
33330 9310 Acked Read 2 Sent Done Done false
33331 9311 Acked Write 9 Sent Done Done false
33332 9312 Acked Write 5 Sent Done Done false
33333 9313 Acked Write 2 Sent Done Done false
33334 9314 Acked Read 2 Sent Done Done false
33335 9315 Acked Read 7 Sent Done Done false
33336 9316 Acked Read 1 Sent Done Done false
33337 9317 Acked Read 8 Sent Done Done false
33338 9318 Acked Read 9 Sent Done Done false
33339 9319 Acked Read 7 Sent Done Done false
33340 9320 Acked Write 8 Sent Done Done false
33341 9321 Acked Read 6 Sent Done Done false
33342 9322 Acked Write 5 Sent Done Done false
33343 9323 Acked Write 10 Sent Done Done false
33344 9324 Acked Write 2 Sent Done Done false
33345 9325 Acked Read 3 Sent Done Done false
33346 9326 Acked Read 8 Sent Done Done false
33347 9327 Acked Write 6 Sent Done Done false
33348 9328 Acked Read 3 Sent Done Done false
33349 9329 Acked Write 6 Sent Done Done false
33350 9330 Acked Write 6 Sent Done Done false
33351 9331 Acked Write 5 Sent Done Done false
33352 9332 Acked Write 1 Sent Done Done false
33353 9333 Acked Write 7 Sent Done Done false
33354 9334 Acked Read 4 Sent Done Done false
33355 9335 Acked Write 1 Sent Done Done false
33356 9336 Acked Write 3 Sent Done Done false
33357 9337 Acked Write 7 Sent Done Done false
33358 9338 Acked Flush 0 Sent Done Done false
33359 9339 Acked Write 5 Sent Done Done false
33360 9340 Acked Read 7 Sent Done Done false
33361 9341 Acked Read 9 Sent Done Done false
33362 9342 Acked Read 1 Sent Done Done false
33363 9343 Acked Read 6 Sent Done Done false
33364 9344 Acked Write 10 Sent Done Done false
33365 9345 Acked Read 7 Sent Done Done false
33366 9346 Acked Write 1 Sent Done Done false
33367 9347 Acked Read 5 Sent Done Done false
33368 9348 Acked Read 5 Sent Done Done false
33369 9349 Acked Flush 0 Sent Done Done false
33370 9350 Acked Read 2 Sent Done Done false
33371 9351 Acked Read 4 Sent Done Done false
33372 9352 Acked Write 6 Sent Done Done false
33373 9353 Acked Flush 0 Sent Done Done false
33374 9354 Acked Write 5 Sent Done Done false
33375 9355 Acked Read 3 Sent Done Done false
33376 9356 Acked Read 2 Sent Done Done false
33377 9357 Acked Read 8 Sent Done Done false
33378 9358 Acked Read 9 Sent Done Done false
33379 9359 Acked Read 7 Sent Done Done false
33380 9360 Acked Write 10 Sent Done Done false
33381 9361 Acked Write 6 Sent Done Done false
33382 9362 Acked Read 4 Sent Done Done false
33383 9363 Acked Flush 0 Sent Done Done false
33384 9364 Acked Write 10 Sent Done Done false
33385 9365 Acked Read 1 Sent Done Done false
33386 9366 Acked Write 5 Sent Done Done false
33387 9367 Acked Read 4 Sent Done Done false
33388 9368 Acked Read 6 Sent Done Done false
33389 9369 Acked Write 8 Sent Done Done false
33390 9370 Acked Write 9 Sent Done Done false
33391 9371 Acked Write 7 Sent Done Done false
33392 9372 Acked Read 5 Sent Done Done false
33393 9373 Acked Read 7 Sent Done Done false
33394 9374 Acked Read 7 Sent Done Done false
33395 9375 Acked Read 2 Sent Done Done false
33396 9376 Acked Read 10 Sent Done Done false
33397 9377 Acked Read 1 Sent Done Done false
33398 9378 Acked Write 1 Sent Done Done false
33399 9379 Acked Write 3 Sent Done Done false
33400 9380 Acked Write 9 Sent Done Done false
33401 9381 Acked Read 4 Sent Done Done false
33402 9382 Acked Write 6 Sent Done Done false
33403 9383 Acked Read 3 Sent Done Done false
33404 9384 Acked Read 9 Sent Done Done false
33405 9385 Acked Write 8 Sent Done Done false
33406 9386 Acked Read 7 Sent Done Done false
33407 9387 Acked Write 3 Sent Done Done false
33408 9388 Acked Write 6 Sent Done Done false
33409 9389 Acked Flush 0 Sent Done Done false
33410 9390 Acked Read 7 Sent Done Done false
33411 9391 Acked Write 6 Sent Done Done false
33412 9392 Acked Read 2 Sent Done Done false
33413 9393 Acked Read 9 Sent Done Done false
33414 9394 Acked Read 9 Sent Done Done false
33415 9395 Acked Read 1 Sent Done Done false
33416 9396 Acked Flush 0 Sent Done Done false
33417 9397 Acked Read 1 Sent Done Done false
33418 9398 Acked Read 2 Sent Done Done false
33419 9399 Acked Read 4 Sent Done Done false
33420 9400 Acked Read 4 Sent Done Done false
33421 9401 Acked Write 5 Sent Done Done false
33422 9402 Acked Read 4 Sent Done Done false
33423 9403 Acked Write 1 Sent Done Done false
33424 9404 Acked Read 2 Sent Done Done false
33425 9405 Acked Write 2 Sent Done Done false
33426 9406 Acked Write 1 Sent Done Done false
33427 9407 Acked Write 10 Sent Done Done false
33428 9408 Acked Read 4 Sent Done Done false
33429 9409 Acked Read 10 Sent Done Done false
33430 9410 Acked Read 1 Sent Done Done false
33431 9411 Acked Read 6 Sent Done Done false
33432 9412 Acked Write 4 Sent Done Done false
33433 9413 Acked Read 3 Sent Done Done false
33434 9414 Acked Write 6 Sent Done Done false
33435 9415 Acked Write 6 Sent Done Done false
33436 9416 Acked Read 2 Sent Done Done false
33437 9417 Acked Read 6 Sent Done Done false
33438 9418 Acked Write 3 Sent Done Done false
33439 9419 Acked Write 2 Sent Done Done false
33440 9420 Acked Read 2 Sent Done Done false
33441 9421 Acked Write 3 Sent Done Done false
33442 9422 Acked Write 7 Sent Done Done false
33443 9423 Acked Write 1 Sent Done Done false
33444 9424 Acked Read 3 Sent Done Done false
33445 9425 Acked Read 8 Sent Done Done false
33446 9426 Acked Read 1 Sent Done Done false
33447 9427 Acked Write 8 Sent Done Done false
33448 9428 Acked Write 10 Sent Done Done false
33449 9429 Acked Read 5 Sent Done Done false
33450 9430 Acked Read 5 Sent Done Done false
33451 9431 Acked Flush 0 Sent Done Done false
33452 9432 Acked Write 3 Sent Done Done false
33453 9433 Acked Read 2 Sent Done Done false
33454 9434 Acked Read 2 Sent Done Done false
33455 9435 Acked Read 9 Sent Done Done false
33456 9436 Acked Read 4 Sent Done Done false
33457 9437 Acked Read 2 Sent Done Done false
33458 9438 Acked Write 10 Sent Done Done false
33459 9439 Acked Read 8 Sent Done Done false
33460 9440 Acked Write 4 Sent Done Done false
33461 9441 Acked Read 3 Sent Done Done false
33462 9442 Acked Read 9 Sent Done Done false
33463 9443 Acked Write 9 Sent Done Done false
33464 9444 Acked Write 6 Sent Done Done false
33465 9445 Acked Read 1 Sent Done Done false
33466 9446 Acked Read 8 Sent Done Done false
33467 9447 Acked Read 5 Sent Done Done false
33468 9448 Acked Read 7 Sent Done Done false
33469 9449 Acked Write 1 Sent Done Done false
33470 9450 Acked Read 7 Sent Done Done false
33471 9451 Acked Flush 0 Sent Done Done false
33472 STATES DS:0 DS:1 DS:2 TOTAL
33473 Sent 301 0 0 301
33474 Done 0 301 301 602
33475 Skipped 0 0 0 0
33476 Error 0 0 0 0
33477 Last Flush: Some(JobId(9150)) Some(JobId(9451)) Some(JobId(9451))
33478 Downstairs last five completed: 9150 9149 9148 9147 9146
33479 Upstairs last five completed: 9451 9450 9449 9448 9447
33480 CLIENT: Up:0 ds:301 act:2
334812025-03-17T22:23:47.301ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
334822025-03-17T22:23:48.099ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
334832025-03-17T22:23:48.099ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
334842025-03-17T22:23:48.099ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
334852025-03-17T22:23:48.099ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
334862025-03-17T22:23:48.100ZINFOcrucible: client task is exiting = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
334872025-03-17T22:23:48.601ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
334882025-03-17T22:23:49.303ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
33489 ----------------------------------------------------------------
33490 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
33491 States: Offline Active Active
33492 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33493 9151 Acked Write 8 Sent Done Done false
33494 9152 Acked Read 10 Sent Done Done false
33495 9153 Acked Flush 0 Sent Done Done false
33496 9154 Acked Read 8 Sent Done Done false
33497 9155 Acked Read 4 Sent Done Done false
33498 9156 Acked Read 1 Sent Done Done false
33499 9157 Acked Read 5 Sent Done Done false
33500 9158 Acked Write 8 Sent Done Done false
33501 9159 Acked Flush 0 Sent Done Done false
33502 9160 Acked Read 9 Sent Done Done false
33503 9161 Acked Read 8 Sent Done Done false
33504 9162 Acked Read 3 Sent Done Done false
33505 9163 Acked Write 6 Sent Done Done false
33506 9164 Acked Flush 0 Sent Done Done false
33507 9165 Acked Flush 0 Sent Done Done false
33508 9166 Acked Read 2 Sent Done Done false
33509 9167 Acked Read 1 Sent Done Done false
33510 9168 Acked Read 9 Sent Done Done false
33511 9169 Acked Read 10 Sent Done Done false
33512 9170 Acked Write 2 Sent Done Done false
33513 9171 Acked Write 6 Sent Done Done false
33514 9172 Acked Write 4 Sent Done Done false
33515 9173 Acked Read 9 Sent Done Done false
33516 9174 Acked Read 5 Sent Done Done false
33517 9175 Acked Write 4 Sent Done Done false
33518 9176 Acked Write 1 Sent Done Done false
33519 9177 Acked Flush 0 Sent Done Done false
33520 9178 Acked Flush 0 Sent Done Done false
33521 9179 Acked Read 6 Sent Done Done false
33522 9180 Acked Write 4 Sent Done Done false
33523 9181 Acked Flush 0 Sent Done Done false
33524 9182 Acked Read 5 Sent Done Done false
33525 9183 Acked Read 3 Sent Done Done false
33526 9184 Acked Read 9 Sent Done Done false
33527 9185 Acked Read 10 Sent Done Done false
33528 9186 Acked Write 3 Sent Done Done false
33529 9187 Acked Write 9 Sent Done Done false
33530 9188 Acked Write 6 Sent Done Done false
33531 9189 Acked Write 8 Sent Done Done false
33532 9190 Acked Write 9 Sent Done Done false
33533 9191 Acked Write 3 Sent Done Done false
33534 9192 Acked Write 10 Sent Done Done false
33535 9193 Acked Write 2 Sent Done Done false
33536 9194 Acked Read 2 Sent Done Done false
33537 9195 Acked Read 9 Sent Done Done false
33538 9196 Acked Write 6 Sent Done Done false
33539 9197 Acked Write 2 Sent Done Done false
33540 9198 Acked Flush 0 Sent Done Done false
33541 9199 Acked Flush 0 Sent Done Done false
33542 9200 Acked Write 9 Sent Done Done false
33543 9201 Acked Write 6 Sent Done Done false
33544 9202 Acked Write 9 Sent Done Done false
33545 9203 Acked Read 2 Sent Done Done false
33546 9204 Acked Write 8 Sent Done Done false
33547 9205 Acked Read 3 Sent Done Done false
33548 9206 Acked Read 7 Sent Done Done false
33549 9207 Acked Flush 0 Sent Done Done false
33550 9208 Acked Write 9 Sent Done Done false
33551 9209 Acked Write 5 Sent Done Done false
33552 9210 Acked Read 5 Sent Done Done false
33553 9211 Acked Write 10 Sent Done Done false
33554 9212 Acked Write 4 Sent Done Done false
33555 9213 Acked Write 6 Sent Done Done false
33556 9214 Acked Write 5 Sent Done Done false
33557 9215 Acked Write 1 Sent Done Done false
33558 9216 Acked Read 10 Sent Done Done false
33559 9217 Acked Read 9 Sent Done Done false
33560 9218 Acked Read 2 Sent Done Done false
33561 9219 Acked Read 8 Sent Done Done false
33562 9220 Acked Flush 0 Sent Done Done false
33563 9221 Acked Write 6 Sent Done Done false
33564 9222 Acked Write 4 Sent Done Done false
33565 9223 Acked Read 4 Sent Done Done false
33566 9224 Acked Read 10 Sent Done Done false
33567 9225 Acked Read 6 Sent Done Done false
33568 9226 Acked Read 8 Sent Done Done false
33569 9227 Acked Write 5 Sent Done Done false
33570 9228 Acked Read 5 Sent Done Done false
33571 9229 Acked Write 3 Sent Done Done false
33572 9230 Acked Read 5 Sent Done Done false
33573 9231 Acked Read 4 Sent Done Done false
33574 9232 Acked Flush 0 Sent Done Done false
33575 9233 Acked Write 1 Sent Done Done false
33576 9234 Acked Write 1 Sent Done Done false
33577 9235 Acked Write 5 Sent Done Done false
33578 9236 Acked Read 8 Sent Done Done false
33579 9237 Acked Flush 0 Sent Done Done false
33580 9238 Acked Read 3 Sent Done Done false
33581 9239 Acked Write 2 Sent Done Done false
33582 9240 Acked Write 9 Sent Done Done false
33583 9241 Acked Read 2 Sent Done Done false
33584 9242 Acked Read 1 Sent Done Done false
33585 9243 Acked Write 9 Sent Done Done false
33586 9244 Acked Read 2 Sent Done Done false
33587 9245 Acked Read 8 Sent Done Done false
33588 9246 Acked Read 7 Sent Done Done false
33589 9247 Acked Write 10 Sent Done Done false
33590 9248 Acked Read 7 Sent Done Done false
33591 9249 Acked Read 1 Sent Done Done false
33592 9250 Acked Write 10 Sent Done Done false
33593 9251 Acked Read 10 Sent Done Done false
33594 9252 Acked Read 8 Sent Done Done false
33595 9253 Acked Read 5 Sent Done Done false
33596 9254 Acked Write 5 Sent Done Done false
33597 9255 Acked Read 5 Sent Done Done false
33598 9256 Acked Read 7 Sent Done Done false
33599 9257 Acked Read 6 Sent Done Done false
33600 9258 Acked Write 4 Sent Done Done false
33601 9259 Acked Write 2 Sent Done Done false
33602 9260 Acked Write 5 Sent Done Done false
33603 9261 Acked Write 8 Sent Done Done false
33604 9262 Acked Write 7 Sent Done Done false
33605 9263 Acked Read 5 Sent Done Done false
33606 9264 Acked Write 1 Sent Done Done false
33607 9265 Acked Read 8 Sent Done Done false
33608 9266 Acked Read 10 Sent Done Done false
33609 9267 Acked Read 1 Sent Done Done false
33610 9268 Acked Read 8 Sent Done Done false
33611 9269 Acked Read 9 Sent Done Done false
33612 9270 Acked Read 7 Sent Done Done false
33613 9271 Acked Read 8 Sent Done Done false
33614 9272 Acked Write 10 Sent Done Done false
33615 9273 Acked Write 1 Sent Done Done false
33616 9274 Acked Read 3 Sent Done Done false
33617 9275 Acked Read 1 Sent Done Done false
33618 9276 Acked Read 5 Sent Done Done false
33619 9277 Acked Flush 0 Sent Done Done false
33620 9278 Acked Write 6 Sent Done Done false
33621 9279 Acked Read 9 Sent Done Done false
33622 9280 Acked Read 4 Sent Done Done false
33623 9281 Acked Read 10 Sent Done Done false
33624 9282 Acked Read 1 Sent Done Done false
33625 9283 Acked Write 5 Sent Done Done false
33626 9284 Acked Read 10 Sent Done Done false
33627 9285 Acked Read 7 Sent Done Done false
33628 9286 Acked Write 1 Sent Done Done false
33629 9287 Acked Read 8 Sent Done Done false
33630 9288 Acked Write 3 Sent Done Done false
33631 9289 Acked Write 3 Sent Done Done false
33632 9290 Acked Write 3 Sent Done Done false
33633 9291 Acked Flush 0 Sent Done Done false
33634 9292 Acked Write 1 Sent Done Done false
33635 9293 Acked Write 4 Sent Done Done false
33636 9294 Acked Read 8 Sent Done Done false
33637 9295 Acked Write 5 Sent Done Done false
33638 9296 Acked Write 1 Sent Done Done false
33639 9297 Acked Write 8 Sent Done Done false
33640 9298 Acked Write 8 Sent Done Done false
33641 9299 Acked Read 1 Sent Done Done false
33642 9300 Acked Flush 0 Sent Done Done false
33643 9301 Acked Write 7 Sent Done Done false
33644 9302 Acked Read 3 Sent Done Done false
33645 9303 Acked Write 4 Sent Done Done false
33646 9304 Acked Read 4 Sent Done Done false
33647 9305 Acked Write 10 Sent Done Done false
33648 9306 Acked Write 6 Sent Done Done false
33649 9307 Acked Write 5 Sent Done Done false
33650 9308 Acked Write 10 Sent Done Done false
33651 9309 Acked Write 4 Sent Done Done false
33652 9310 Acked Read 2 Sent Done Done false
33653 9311 Acked Write 9 Sent Done Done false
33654 9312 Acked Write 5 Sent Done Done false
33655 9313 Acked Write 2 Sent Done Done false
33656 9314 Acked Read 2 Sent Done Done false
33657 9315 Acked Read 7 Sent Done Done false
33658 9316 Acked Read 1 Sent Done Done false
33659 9317 Acked Read 8 Sent Done Done false
33660 9318 Acked Read 9 Sent Done Done false
33661 9319 Acked Read 7 Sent Done Done false
33662 9320 Acked Write 8 Sent Done Done false
33663 9321 Acked Read 6 Sent Done Done false
33664 9322 Acked Write 5 Sent Done Done false
33665 9323 Acked Write 10 Sent Done Done false
33666 9324 Acked Write 2 Sent Done Done false
33667 9325 Acked Read 3 Sent Done Done false
33668 9326 Acked Read 8 Sent Done Done false
33669 9327 Acked Write 6 Sent Done Done false
33670 9328 Acked Read 3 Sent Done Done false
33671 9329 Acked Write 6 Sent Done Done false
33672 9330 Acked Write 6 Sent Done Done false
33673 9331 Acked Write 5 Sent Done Done false
33674 9332 Acked Write 1 Sent Done Done false
33675 9333 Acked Write 7 Sent Done Done false
33676 9334 Acked Read 4 Sent Done Done false
33677 9335 Acked Write 1 Sent Done Done false
33678 9336 Acked Write 3 Sent Done Done false
33679 9337 Acked Write 7 Sent Done Done false
33680 9338 Acked Flush 0 Sent Done Done false
33681 9339 Acked Write 5 Sent Done Done false
33682 9340 Acked Read 7 Sent Done Done false
33683 9341 Acked Read 9 Sent Done Done false
33684 9342 Acked Read 1 Sent Done Done false
33685 9343 Acked Read 6 Sent Done Done false
33686 9344 Acked Write 10 Sent Done Done false
33687 9345 Acked Read 7 Sent Done Done false
33688 9346 Acked Write 1 Sent Done Done false
33689 9347 Acked Read 5 Sent Done Done false
33690 9348 Acked Read 5 Sent Done Done false
33691 9349 Acked Flush 0 Sent Done Done false
33692 9350 Acked Read 2 Sent Done Done false
33693 9351 Acked Read 4 Sent Done Done false
33694 9352 Acked Write 6 Sent Done Done false
33695 9353 Acked Flush 0 Sent Done Done false
33696 9354 Acked Write 5 Sent Done Done false
33697 9355 Acked Read 3 Sent Done Done false
33698 9356 Acked Read 2 Sent Done Done false
33699 9357 Acked Read 8 Sent Done Done false
33700 9358 Acked Read 9 Sent Done Done false
33701 9359 Acked Read 7 Sent Done Done false
33702 9360 Acked Write 10 Sent Done Done false
33703 9361 Acked Write 6 Sent Done Done false
33704 9362 Acked Read 4 Sent Done Done false
33705 9363 Acked Flush 0 Sent Done Done false
33706 9364 Acked Write 10 Sent Done Done false
33707 9365 Acked Read 1 Sent Done Done false
33708 9366 Acked Write 5 Sent Done Done false
33709 9367 Acked Read 4 Sent Done Done false
33710 9368 Acked Read 6 Sent Done Done false
33711 9369 Acked Write 8 Sent Done Done false
33712 9370 Acked Write 9 Sent Done Done false
33713 9371 Acked Write 7 Sent Done Done false
33714 9372 Acked Read 5 Sent Done Done false
33715 9373 Acked Read 7 Sent Done Done false
33716 9374 Acked Read 7 Sent Done Done false
33717 9375 Acked Read 2 Sent Done Done false
33718 9376 Acked Read 10 Sent Done Done false
33719 9377 Acked Read 1 Sent Done Done false
33720 9378 Acked Write 1 Sent Done Done false
33721 9379 Acked Write 3 Sent Done Done false
33722 9380 Acked Write 9 Sent Done Done false
33723 9381 Acked Read 4 Sent Done Done false
33724 9382 Acked Write 6 Sent Done Done false
33725 9383 Acked Read 3 Sent Done Done false
33726 9384 Acked Read 9 Sent Done Done false
33727 9385 Acked Write 8 Sent Done Done false
33728 9386 Acked Read 7 Sent Done Done false
33729 9387 Acked Write 3 Sent Done Done false
33730 9388 Acked Write 6 Sent Done Done false
33731 9389 Acked Flush 0 Sent Done Done false
33732 9390 Acked Read 7 Sent Done Done false
33733 9391 Acked Write 6 Sent Done Done false
33734 9392 Acked Read 2 Sent Done Done false
33735 9393 Acked Read 9 Sent Done Done false
33736 9394 Acked Read 9 Sent Done Done false
33737 9395 Acked Read 1 Sent Done Done false
33738 9396 Acked Flush 0 Sent Done Done false
33739 9397 Acked Read 1 Sent Done Done false
33740 9398 Acked Read 2 Sent Done Done false
33741 9399 Acked Read 4 Sent Done Done false
33742 9400 Acked Read 4 Sent Done Done false
33743 9401 Acked Write 5 Sent Done Done false
33744 9402 Acked Read 4 Sent Done Done false
33745 9403 Acked Write 1 Sent Done Done false
33746 9404 Acked Read 2 Sent Done Done false
33747 9405 Acked Write 2 Sent Done Done false
33748 9406 Acked Write 1 Sent Done Done false
33749 9407 Acked Write 10 Sent Done Done false
33750 9408 Acked Read 4 Sent Done Done false
33751 9409 Acked Read 10 Sent Done Done false
33752 9410 Acked Read 1 Sent Done Done false
33753 9411 Acked Read 6 Sent Done Done false
33754 9412 Acked Write 4 Sent Done Done false
33755 9413 Acked Read 3 Sent Done Done false
33756 9414 Acked Write 6 Sent Done Done false
33757 9415 Acked Write 6 Sent Done Done false
33758 9416 Acked Read 2 Sent Done Done false
33759 9417 Acked Read 6 Sent Done Done false
33760 9418 Acked Write 3 Sent Done Done false
33761 9419 Acked Write 2 Sent Done Done false
33762 9420 Acked Read 2 Sent Done Done false
33763 9421 Acked Write 3 Sent Done Done false
33764 9422 Acked Write 7 Sent Done Done false
33765 9423 Acked Write 1 Sent Done Done false
33766 9424 Acked Read 3 Sent Done Done false
33767 9425 Acked Read 8 Sent Done Done false
33768 9426 Acked Read 1 Sent Done Done false
33769 9427 Acked Write 8 Sent Done Done false
33770 9428 Acked Write 10 Sent Done Done false
33771 9429 Acked Read 5 Sent Done Done false
33772 9430 Acked Read 5 Sent Done Done false
33773 9431 Acked Flush 0 Sent Done Done false
33774 9432 Acked Write 3 Sent Done Done false
33775 9433 Acked Read 2 Sent Done Done false
33776 9434 Acked Read 2 Sent Done Done false
33777 9435 Acked Read 9 Sent Done Done false
33778 9436 Acked Read 4 Sent Done Done false
33779 9437 Acked Read 2 Sent Done Done false
33780 9438 Acked Write 10 Sent Done Done false
33781 9439 Acked Read 8 Sent Done Done false
33782 9440 Acked Write 4 Sent Done Done false
33783 9441 Acked Read 3 Sent Done Done false
33784 9442 Acked Read 9 Sent Done Done false
33785 9443 Acked Write 9 Sent Done Done false
33786 9444 Acked Write 6 Sent Done Done false
33787 9445 Acked Read 1 Sent Done Done false
33788 9446 Acked Read 8 Sent Done Done false
33789 9447 Acked Read 5 Sent Done Done false
33790 9448 Acked Read 7 Sent Done Done false
33791 9449 Acked Write 1 Sent Done Done false
33792 9450 Acked Read 7 Sent Done Done false
33793 9451 Acked Flush 0 Sent Done Done false
33794 STATES DS:0 DS:1 DS:2 TOTAL
33795 Sent 301 0 0 301
33796 Done 0 301 301 602
33797 Skipped 0 0 0 0
33798 Error 0 0 0 0
33799 Last Flush: Some(JobId(9150)) Some(JobId(9451)) Some(JobId(9451))
33800 Downstairs last five completed: 9150 9149 9148 9147 9146
33801 Upstairs last five completed: 9451 9450 9449 9448 9447
33802 CLIENT: Up:0 ds:301 act:2
338032025-03-17T22:23:51.304ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
33804 [8810] got start action
33805 [8810] I am going to restart
33806 Starting downstairs at port 8810
33807 Make output file at "/tmp/dsc/downstairs-8810.txt"
33808 [8810][0] reports Starting
33809 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1234)
33810 [8810][0] reports Running
338112025-03-17T22:23:52.307ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
338122025-03-17T22:23:52.807ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
338132025-03-17T22:23:53.308ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
338142025-03-17T22:23:54.310ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
33815 ----------------------------------------------------------------
33816 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
33817 States: Offline Active Active
33818 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
33819 9151 Acked Write 8 Sent Done Done false
33820 9152 Acked Read 10 Sent Done Done false
33821 9153 Acked Flush 0 Sent Done Done false
33822 9154 Acked Read 8 Sent Done Done false
33823 9155 Acked Read 4 Sent Done Done false
33824 9156 Acked Read 1 Sent Done Done false
33825 9157 Acked Read 5 Sent Done Done false
33826 9158 Acked Write 8 Sent Done Done false
33827 9159 Acked Flush 0 Sent Done Done false
33828 9160 Acked Read 9 Sent Done Done false
33829 9161 Acked Read 8 Sent Done Done false
33830 9162 Acked Read 3 Sent Done Done false
33831 9163 Acked Write 6 Sent Done Done false
33832 9164 Acked Flush 0 Sent Done Done false
33833 9165 Acked Flush 0 Sent Done Done false
33834 9166 Acked Read 2 Sent Done Done false
33835 9167 Acked Read 1 Sent Done Done false
33836 9168 Acked Read 9 Sent Done Done false
33837 9169 Acked Read 10 Sent Done Done false
33838 9170 Acked Write 2 Sent Done Done false
33839 9171 Acked Write 6 Sent Done Done false
33840 9172 Acked Write 4 Sent Done Done false
33841 9173 Acked Read 9 Sent Done Done false
33842 9174 Acked Read 5 Sent Done Done false
33843 9175 Acked Write 4 Sent Done Done false
33844 9176 Acked Write 1 Sent Done Done false
33845 9177 Acked Flush 0 Sent Done Done false
33846 9178 Acked Flush 0 Sent Done Done false
33847 9179 Acked Read 6 Sent Done Done false
33848 9180 Acked Write 4 Sent Done Done false
33849 9181 Acked Flush 0 Sent Done Done false
33850 9182 Acked Read 5 Sent Done Done false
33851 9183 Acked Read 3 Sent Done Done false
33852 9184 Acked Read 9 Sent Done Done false
33853 9185 Acked Read 10 Sent Done Done false
33854 9186 Acked Write 3 Sent Done Done false
33855 9187 Acked Write 9 Sent Done Done false
33856 9188 Acked Write 6 Sent Done Done false
33857 9189 Acked Write 8 Sent Done Done false
33858 9190 Acked Write 9 Sent Done Done false
33859 9191 Acked Write 3 Sent Done Done false
33860 9192 Acked Write 10 Sent Done Done false
33861 9193 Acked Write 2 Sent Done Done false
33862 9194 Acked Read 2 Sent Done Done false
33863 9195 Acked Read 9 Sent Done Done false
33864 9196 Acked Write 6 Sent Done Done false
33865 9197 Acked Write 2 Sent Done Done false
33866 9198 Acked Flush 0 Sent Done Done false
33867 9199 Acked Flush 0 Sent Done Done false
33868 9200 Acked Write 9 Sent Done Done false
33869 9201 Acked Write 6 Sent Done Done false
33870 9202 Acked Write 9 Sent Done Done false
33871 9203 Acked Read 2 Sent Done Done false
33872 9204 Acked Write 8 Sent Done Done false
33873 9205 Acked Read 3 Sent Done Done false
33874 9206 Acked Read 7 Sent Done Done false
33875 9207 Acked Flush 0 Sent Done Done false
33876 9208 Acked Write 9 Sent Done Done false
33877 9209 Acked Write 5 Sent Done Done false
33878 9210 Acked Read 5 Sent Done Done false
33879 9211 Acked Write 10 Sent Done Done false
33880 9212 Acked Write 4 Sent Done Done false
33881 9213 Acked Write 6 Sent Done Done false
33882 9214 Acked Write 5 Sent Done Done false
33883 9215 Acked Write 1 Sent Done Done false
33884 9216 Acked Read 10 Sent Done Done false
33885 9217 Acked Read 9 Sent Done Done false
33886 9218 Acked Read 2 Sent Done Done false
33887 9219 Acked Read 8 Sent Done Done false
33888 9220 Acked Flush 0 Sent Done Done false
33889 9221 Acked Write 6 Sent Done Done false
33890 9222 Acked Write 4 Sent Done Done false
33891 9223 Acked Read 4 Sent Done Done false
33892 9224 Acked Read 10 Sent Done Done false
33893 9225 Acked Read 6 Sent Done Done false
33894 9226 Acked Read 8 Sent Done Done false
33895 9227 Acked Write 5 Sent Done Done false
33896 9228 Acked Read 5 Sent Done Done false
33897 9229 Acked Write 3 Sent Done Done false
33898 9230 Acked Read 5 Sent Done Done false
33899 9231 Acked Read 4 Sent Done Done false
33900 9232 Acked Flush 0 Sent Done Done false
33901 9233 Acked Write 1 Sent Done Done false
33902 9234 Acked Write 1 Sent Done Done false
33903 9235 Acked Write 5 Sent Done Done false
33904 9236 Acked Read 8 Sent Done Done false
33905 9237 Acked Flush 0 Sent Done Done false
33906 9238 Acked Read 3 Sent Done Done false
33907 9239 Acked Write 2 Sent Done Done false
33908 9240 Acked Write 9 Sent Done Done false
33909 9241 Acked Read 2 Sent Done Done false
33910 9242 Acked Read 1 Sent Done Done false
33911 9243 Acked Write 9 Sent Done Done false
33912 9244 Acked Read 2 Sent Done Done false
33913 9245 Acked Read 8 Sent Done Done false
33914 9246 Acked Read 7 Sent Done Done false
33915 9247 Acked Write 10 Sent Done Done false
33916 9248 Acked Read 7 Sent Done Done false
33917 9249 Acked Read 1 Sent Done Done false
33918 9250 Acked Write 10 Sent Done Done false
33919 9251 Acked Read 10 Sent Done Done false
33920 9252 Acked Read 8 Sent Done Done false
33921 9253 Acked Read 5 Sent Done Done false
33922 9254 Acked Write 5 Sent Done Done false
33923 9255 Acked Read 5 Sent Done Done false
33924 9256 Acked Read 7 Sent Done Done false
33925 9257 Acked Read 6 Sent Done Done false
33926 9258 Acked Write 4 Sent Done Done false
33927 9259 Acked Write 2 Sent Done Done false
33928 9260 Acked Write 5 Sent Done Done false
33929 9261 Acked Write 8 Sent Done Done false
33930 9262 Acked Write 7 Sent Done Done false
33931 9263 Acked Read 5 Sent Done Done false
33932 9264 Acked Write 1 Sent Done Done false
33933 9265 Acked Read 8 Sent Done Done false
33934 9266 Acked Read 10 Sent Done Done false
33935 9267 Acked Read 1 Sent Done Done false
33936 9268 Acked Read 8 Sent Done Done false
33937 9269 Acked Read 9 Sent Done Done false
33938 9270 Acked Read 7 Sent Done Done false
33939 9271 Acked Read 8 Sent Done Done false
33940 9272 Acked Write 10 Sent Done Done false
33941 9273 Acked Write 1 Sent Done Done false
33942 9274 Acked Read 3 Sent Done Done false
33943 9275 Acked Read 1 Sent Done Done false
33944 9276 Acked Read 5 Sent Done Done false
33945 9277 Acked Flush 0 Sent Done Done false
33946 9278 Acked Write 6 Sent Done Done false
33947 9279 Acked Read 9 Sent Done Done false
33948 9280 Acked Read 4 Sent Done Done false
33949 9281 Acked Read 10 Sent Done Done false
33950 9282 Acked Read 1 Sent Done Done false
33951 9283 Acked Write 5 Sent Done Done false
33952 9284 Acked Read 10 Sent Done Done false
33953 9285 Acked Read 7 Sent Done Done false
33954 9286 Acked Write 1 Sent Done Done false
33955 9287 Acked Read 8 Sent Done Done false
33956 9288 Acked Write 3 Sent Done Done false
33957 9289 Acked Write 3 Sent Done Done false
33958 9290 Acked Write 3 Sent Done Done false
33959 9291 Acked Flush 0 Sent Done Done false
33960 9292 Acked Write 1 Sent Done Done false
33961 9293 Acked Write 4 Sent Done Done false
33962 9294 Acked Read 8 Sent Done Done false
33963 9295 Acked Write 5 Sent Done Done false
33964 9296 Acked Write 1 Sent Done Done false
33965 9297 Acked Write 8 Sent Done Done false
33966 9298 Acked Write 8 Sent Done Done false
33967 9299 Acked Read 1 Sent Done Done false
33968 9300 Acked Flush 0 Sent Done Done false
33969 9301 Acked Write 7 Sent Done Done false
33970 9302 Acked Read 3 Sent Done Done false
33971 9303 Acked Write 4 Sent Done Done false
33972 9304 Acked Read 4 Sent Done Done false
33973 9305 Acked Write 10 Sent Done Done false
33974 9306 Acked Write 6 Sent Done Done false
33975 9307 Acked Write 5 Sent Done Done false
33976 9308 Acked Write 10 Sent Done Done false
33977 9309 Acked Write 4 Sent Done Done false
33978 9310 Acked Read 2 Sent Done Done false
33979 9311 Acked Write 9 Sent Done Done false
33980 9312 Acked Write 5 Sent Done Done false
33981 9313 Acked Write 2 Sent Done Done false
33982 9314 Acked Read 2 Sent Done Done false
33983 9315 Acked Read 7 Sent Done Done false
33984 9316 Acked Read 1 Sent Done Done false
33985 9317 Acked Read 8 Sent Done Done false
33986 9318 Acked Read 9 Sent Done Done false
33987 9319 Acked Read 7 Sent Done Done false
33988 9320 Acked Write 8 Sent Done Done false
33989 9321 Acked Read 6 Sent Done Done false
33990 9322 Acked Write 5 Sent Done Done false
33991 9323 Acked Write 10 Sent Done Done false
33992 9324 Acked Write 2 Sent Done Done false
33993 9325 Acked Read 3 Sent Done Done false
33994 9326 Acked Read 8 Sent Done Done false
33995 9327 Acked Write 6 Sent Done Done false
33996 9328 Acked Read 3 Sent Done Done false
33997 9329 Acked Write 6 Sent Done Done false
33998 9330 Acked Write 6 Sent Done Done false
33999 9331 Acked Write 5 Sent Done Done false
34000 9332 Acked Write 1 Sent Done Done false
34001 9333 Acked Write 7 Sent Done Done false
34002 9334 Acked Read 4 Sent Done Done false
34003 9335 Acked Write 1 Sent Done Done false
34004 9336 Acked Write 3 Sent Done Done false
34005 9337 Acked Write 7 Sent Done Done false
34006 9338 Acked Flush 0 Sent Done Done false
34007 9339 Acked Write 5 Sent Done Done false
34008 9340 Acked Read 7 Sent Done Done false
34009 9341 Acked Read 9 Sent Done Done false
34010 9342 Acked Read 1 Sent Done Done false
34011 9343 Acked Read 6 Sent Done Done false
34012 9344 Acked Write 10 Sent Done Done false
34013 9345 Acked Read 7 Sent Done Done false
34014 9346 Acked Write 1 Sent Done Done false
34015 9347 Acked Read 5 Sent Done Done false
34016 9348 Acked Read 5 Sent Done Done false
34017 9349 Acked Flush 0 Sent Done Done false
34018 9350 Acked Read 2 Sent Done Done false
34019 9351 Acked Read 4 Sent Done Done false
34020 9352 Acked Write 6 Sent Done Done false
34021 9353 Acked Flush 0 Sent Done Done false
34022 9354 Acked Write 5 Sent Done Done false
34023 9355 Acked Read 3 Sent Done Done false
34024 9356 Acked Read 2 Sent Done Done false
34025 9357 Acked Read 8 Sent Done Done false
34026 9358 Acked Read 9 Sent Done Done false
34027 9359 Acked Read 7 Sent Done Done false
34028 9360 Acked Write 10 Sent Done Done false
34029 9361 Acked Write 6 Sent Done Done false
34030 9362 Acked Read 4 Sent Done Done false
34031 9363 Acked Flush 0 Sent Done Done false
34032 9364 Acked Write 10 Sent Done Done false
34033 9365 Acked Read 1 Sent Done Done false
34034 9366 Acked Write 5 Sent Done Done false
34035 9367 Acked Read 4 Sent Done Done false
34036 9368 Acked Read 6 Sent Done Done false
34037 9369 Acked Write 8 Sent Done Done false
34038 9370 Acked Write 9 Sent Done Done false
34039 9371 Acked Write 7 Sent Done Done false
34040 9372 Acked Read 5 Sent Done Done false
34041 9373 Acked Read 7 Sent Done Done false
34042 9374 Acked Read 7 Sent Done Done false
34043 9375 Acked Read 2 Sent Done Done false
34044 9376 Acked Read 10 Sent Done Done false
34045 9377 Acked Read 1 Sent Done Done false
34046 9378 Acked Write 1 Sent Done Done false
34047 9379 Acked Write 3 Sent Done Done false
34048 9380 Acked Write 9 Sent Done Done false
34049 9381 Acked Read 4 Sent Done Done false
34050 9382 Acked Write 6 Sent Done Done false
34051 9383 Acked Read 3 Sent Done Done false
34052 9384 Acked Read 9 Sent Done Done false
34053 9385 Acked Write 8 Sent Done Done false
34054 9386 Acked Read 7 Sent Done Done false
34055 9387 Acked Write 3 Sent Done Done false
34056 9388 Acked Write 6 Sent Done Done false
34057 9389 Acked Flush 0 Sent Done Done false
34058 9390 Acked Read 7 Sent Done Done false
34059 9391 Acked Write 6 Sent Done Done false
34060 9392 Acked Read 2 Sent Done Done false
34061 9393 Acked Read 9 Sent Done Done false
34062 9394 Acked Read 9 Sent Done Done false
34063 9395 Acked Read 1 Sent Done Done false
34064 9396 Acked Flush 0 Sent Done Done false
34065 9397 Acked Read 1 Sent Done Done false
34066 9398 Acked Read 2 Sent Done Done false
34067 9399 Acked Read 4 Sent Done Done false
34068 9400 Acked Read 4 Sent Done Done false
34069 9401 Acked Write 5 Sent Done Done false
34070 9402 Acked Read 4 Sent Done Done false
34071 9403 Acked Write 1 Sent Done Done false
34072 9404 Acked Read 2 Sent Done Done false
34073 9405 Acked Write 2 Sent Done Done false
34074 9406 Acked Write 1 Sent Done Done false
34075 9407 Acked Write 10 Sent Done Done false
34076 9408 Acked Read 4 Sent Done Done false
34077 9409 Acked Read 10 Sent Done Done false
34078 9410 Acked Read 1 Sent Done Done false
34079 9411 Acked Read 6 Sent Done Done false
34080 9412 Acked Write 4 Sent Done Done false
34081 9413 Acked Read 3 Sent Done Done false
34082 9414 Acked Write 6 Sent Done Done false
34083 9415 Acked Write 6 Sent Done Done false
34084 9416 Acked Read 2 Sent Done Done false
34085 9417 Acked Read 6 Sent Done Done false
34086 9418 Acked Write 3 Sent Done Done false
34087 9419 Acked Write 2 Sent Done Done false
34088 9420 Acked Read 2 Sent Done Done false
34089 9421 Acked Write 3 Sent Done Done false
34090 9422 Acked Write 7 Sent Done Done false
34091 9423 Acked Write 1 Sent Done Done false
34092 9424 Acked Read 3 Sent Done Done false
34093 9425 Acked Read 8 Sent Done Done false
34094 9426 Acked Read 1 Sent Done Done false
34095 9427 Acked Write 8 Sent Done Done false
34096 9428 Acked Write 10 Sent Done Done false
34097 9429 Acked Read 5 Sent Done Done false
34098 9430 Acked Read 5 Sent Done Done false
34099 9431 Acked Flush 0 Sent Done Done false
34100 9432 Acked Write 3 Sent Done Done false
34101 9433 Acked Read 2 Sent Done Done false
34102 9434 Acked Read 2 Sent Done Done false
34103 9435 Acked Read 9 Sent Done Done false
34104 9436 Acked Read 4 Sent Done Done false
34105 9437 Acked Read 2 Sent Done Done false
34106 9438 Acked Write 10 Sent Done Done false
34107 9439 Acked Read 8 Sent Done Done false
34108 9440 Acked Write 4 Sent Done Done false
34109 9441 Acked Read 3 Sent Done Done false
34110 9442 Acked Read 9 Sent Done Done false
34111 9443 Acked Write 9 Sent Done Done false
34112 9444 Acked Write 6 Sent Done Done false
34113 9445 Acked Read 1 Sent Done Done false
34114 9446 Acked Read 8 Sent Done Done false
34115 9447 Acked Read 5 Sent Done Done false
34116 9448 Acked Read 7 Sent Done Done false
34117 9449 Acked Write 1 Sent Done Done false
34118 9450 Acked Read 7 Sent Done Done false
34119 9451 Acked Flush 0 Sent Done Done false
34120 STATES DS:0 DS:1 DS:2 TOTAL
34121 Sent 301 0 0 301
34122 Done 0 301 301 602
34123 Skipped 0 0 0 0
34124 Error 0 0 0 0
34125 Last Flush: Some(JobId(9150)) Some(JobId(9451)) Some(JobId(9451))
34126 Downstairs last five completed: 9150 9149 9148 9147 9146
34127 Upstairs last five completed: 9451 9450 9449 9448 9447
34128 CLIENT: Up:0 ds:301 act:2
341292025-03-17T22:23:55.595ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
341302025-03-17T22:23:56.313ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
341312025-03-17T22:23:57.314ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
341322025-03-17T22:23:58.100ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
341332025-03-17T22:23:58.100ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:46946) = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
341342025-03-17T22:23:58.101ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
341352025-03-17T22:23:58.102ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
341362025-03-17T22:23:58.102ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
341372025-03-17T22:23:58.102ZINFOcrucible: send last flush ID to this DS: Some(JobId(9150)) = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
341382025-03-17T22:23:58.102ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
341392025-03-17T22:23:58.102ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(9150)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
341402025-03-17T22:23:58.102ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(9150)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
34141 ----------------------------------------------------------------
34142 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
34143 Downstairs last five completed: 9451 9450 9449 9448 9447
34144 Upstairs last five completed: 9451 9450 9449 9448 9447
34145 CLIENT: Up:0 ds:0 act:3
34146 Replay: All jobs finished
34147 Main task has work to do, go find it
34148 got dsc Stop(1)
34149 stop 1
34150 Mar 17 22:23:58.358 INFO request completed, latency_us: 388, response_code: 204, uri: /stop/cid/1, method: POST, req_id: ab4aaa69-0450-4c09-81d1-310ffc6ed078, remote_addr: 127.0.0.1:60423[8820] Got stop action so:false kr:false
34151 , local_addr: 127.0.0.1:9998
34152 Mar 17 22:23:58.359 INFO request completed, latency_us: 113, response_code: 200, uri: /state/cid/1, method: GET, req_id: 2e02953c-0ba3-46ef-a746-0013164558f0, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:23:58.359488876Z","hostname":"w-01JPK2S810SFAA9787XEG8GFX5","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"0f8310d7-099e-4ead-8484-91ddc162d307"}
34153 
341542025-03-17T22:23:58.359ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
341552025-03-17T22:23:58.359ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
341562025-03-17T22:23:58.359ZINFOcrucible: client task is exiting = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
34157 [8820] Exited with: ExitStatus(unix_wait_status(9))
34158 [8820][1] reports Exit
34159 Mar 17 22:24:02.361 INFO request completed, latency_us: 226, response_code: 200, uri: /state/cid/1, method: GET, req_id: 6c2339ad-ce5f-4bcd-9261-d0bd8e22b027, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
34160 001/300 Read block 1050 len 8192
34161 002/300 Read block 654 len 4096
34162 003/300 Write block 1506 len 16384 data: 4 4 4 4
34163 004/300 Flush
34164 005/300 Write block 1857 len 40960 data: 6 7 6 7 4 4 5 6 5 4
34165 006/300 Write block 4661 len 8192 data: 6 6
34166 007/300 Read block 2440 len 40960
34167 008/300 Read block 1724 len 40960
34168 009/300 Read block 3563 len 28672
34169 010/300 Read block 2273 len 12288
34170 011/300 Write block 2921 len 20480 data: 9 9 6 5 5
34171 012/300 Write block 1411 len 16384 data: 7 6 8 7
34172 013/300 Write block 1456 len 36864 data: 6 6 7 8 7 6 5 5 4
34173 014/300 Write block 2325 len 40960 data: 5 8 7 7 5 5 6 6 4 5
34174 015/300 Read block 764 len 4096
34175 016/300 Read block 3417 len 16384
34176 017/300 Flush
34177 018/300 Read block 2431 len 20480
34178 019/300 Flush
34179 020/300 Read block 1067 len 24576
34180 021/300 Write block 4078 len 20480 data: 3 5 5 4 4
34181 022/300 Read block 3719 len 16384
34182 023/300 Write block 2954 len 16384 data: 6 6 10 10
34183 024/300 Write block 4505 len 12288 data: 6 6 5
34184 025/300 Write block 2235 len 12288 data: 4 4 3
34185 026/300 Read block 2044 len 28672
34186 027/300 Write block 3341 len 8192 data: 4 3
34187 028/300 Read block 1326 len 16384
34188 029/300 Write block 2783 len 24576 data: 5 9 9 7 7 7
34189 030/300 Flush
34190 031/300 Read block 515 len 20480
34191 032/300 Read block 4073 len 4096
34192 033/300 Write block 296 len 12288 data: 4 6 6
34193 034/300 Flush
34194 035/300 Read block 4258 len 8192
34195 036/300 Read block 2316 len 36864
34196 037/300 Write block 4022 len 28672 data: 5 5 3 2 2 3 3
34197 038/300 Read block 4154 len 16384
34198 039/300 Read block 3767 len 24576
34199 040/300 Read block 2035 len 20480
34200 041/300 Write block 4451 len 8192 data: 7 7
34201 042/300 Read block 4260 len 20480
34202 043/300 Flush
34203 044/300 Write block 4850 len 36864 data: 8 7 6 5 4 4 6 9 4
34204 045/300 Write block 4690 len 36864 data: 8 6 6 6 4 4 4 3 2
34205 046/300 Read block 158 len 40960
34206 047/300 Write block 3579 len 16384 data: 5 4 5 6
34207 048/300 Write block 3593 len 4096 data: 6
34208 049/300 Write block 3591 len 16384 data: 5 5 7 4
34209 050/300 Write block 1614 len 36864 data: 8 8 8 8 5 4 4 5 6
34210 051/300 Read block 4074 len 28672
34211 052/300 Read block 1611 len 8192
34212 053/300 Read block 324 len 40960
34213 054/300 Read block 279 len 8192
34214 055/300 Write block 1605 len 12288 data: 3 4 4
34215 056/300 Flush
34216 057/300 Read block 4389 len 32768
34217 058/300 Flush
34218 059/300 Write block 1201 len 8192 data: 6 5
34219 060/300 Write block 3788 len 40960 data: 3 4 4 4 3 4 5 6 6 6
34220 061/300 Read block 953 len 24576
34221 062/300 Read block 367 len 32768
34222 063/300 Write block 4987 len 28672 data: 7 6 5 7 7 6 5
34223 064/300 Read block 1434 len 40960
34224 065/300 Write block 3745 len 28672 data: 8 8 8 7 8 5 3
34225 066/300 Read block 1333 len 40960
34226 067/300 Read block 4715 len 40960
34227 068/300 Write block 3291 len 36864 data: 7 7 7 6 6 6 6 5 6
34228 069/300 Write block 3037 len 20480 data: 6 6 7 5 5
34229 070/300 Read block 638 len 32768
34230 071/300 Read block 4382 len 28672
34231 072/300 Write block 4037 len 8192 data: 5 5
34232 073/300 Write block 3968 len 28672 data: 5 4 5 5 4 4 5
34233 074/300 Flush
34234 075/300 Write block 4473 len 24576 data: 8 5 4 4 5 5
34235 076/300 Write block 2838 len 20480 data: 8 7 7 5 4
34236 077/300 Write block 3502 len 24576 data: 3 3 3 4 4 6
34237 078/300 Read block 3887 len 36864
34238 079/300 Flush
34239 080/300 Write block 2156 len 40960 data: 5 5 4 5 5 4 5 5 4 5
34240 081/300 Write block 1717 len 36864 data: 7 7 7 6 4 5 5 4 4
34241 082/300 Read block 3474 len 16384
34242 083/300 Read block 4566 len 8192
34243 084/300 Write block 4500 len 32768 data: 7 6 7 7 5 7 7 6
34244 085/300 Read block 2746 len 12288
34245 086/300 Read block 4605 len 12288
34246 087/300 Read block 4396 len 12288
34247 088/300 Write block 2765 len 4096 data: 5
34248 089/300 Read block 3614 len 28672
34249 090/300 Read block 13 len 12288
34250 091/300 Read block 3298 len 36864
34251 092/300 Read block 1765 len 8192
34252 093/300 Read block 3679 len 8192
34253 094/300 Write block 798 len 24576 data: 8 7 7 7 5 4
34254 095/300 Write block 4084 len 8192 data: 8 9
34255 096/300 Read block 2310 len 32768
34256 097/300 Write block 1925 len 32768 data: 7 10 10 11 8 7 6 6
34257 098/300 Write block 4212 len 8192 data: 7 5
34258 099/300 Read block 1331 len 32768
34259 100/300 Read block 4471 len 16384
34260 101/300 Write block 2617 len 24576 data: 7 7 8 8 8 7
34261 102/300 Read block 4640 len 36864
34262 103/300 Read block 1322 len 8192
34263 104/300 Read block 3753 len 8192
34264 105/300 Flush
34265 106/300 Read block 1035 len 40960
34266 107/300 Write block 1301 len 24576 data: 6 5 4 3 5 5
34267 108/300 Read block 344 len 16384
34268 109/300 Read block 4384 len 12288
34269 110/300 Write block 661 len 24576 data: 4 4 3 3 5 4
34270 111/300 Read block 1183 len 8192
34271 112/300 Write block 948 len 8192 data: 6 6
34272 113/300 Write block 2374 len 32768 data: 3 3 4 6 6 6 7 6
34273 114/300 Write block 711 len 36864 data: 4 4 4 6 7 6 5 5 6
34274 115/300 Write block 1345 len 36864 data: 3 3 2 2 2 3 4 4 4
34275 116/300 Read block 2254 len 40960
34276 117/300 Read block 1054 len 36864
34277 118/300 Read block 2178 len 12288
34278 119/300 Flush
34279 120/300 Write block 68 len 20480 data: 7 7 6 6 6
34280 121/300 Flush
34281 122/300 Read block 1800 len 4096
34282 123/300 Read block 592 len 8192
34283 124/300 Read block 2121 len 12288
34284 125/300 Write block 368 len 40960 data: 4 4 4 4 6 8 8 9 8 8
34285 126/300 Flush
34286 127/300 Read block 2129 len 20480
34287 128/300 Read block 4622 len 24576
34288 129/300 Read block 2280 len 20480
34289 130/300 Write block 144 len 36864 data: 6 5 3 4 4 4 3 3 2
34290 131/300 Write block 559 len 36864 data: 12 12 12 11 9 8 8 7 4
34291 132/300 Flush
34292 133/300 Write block 4269 len 12288 data: 6 7 7
34293 134/300 Read block 2877 len 32768
34294 135/300 Read block 4310 len 4096
34295 136/300 Write block 4290 len 16384 data: 4 3 4 5
34296 137/300 Read block 1304 len 36864
34297 138/300 Read block 1148 len 36864
34298 139/300 Flush
34299 140/300 Write block 441 len 28672 data: 4 3 4 4 5 5 6
34300 141/300 Write block 2393 len 4096 data: 6
34301 142/300 Write block 530 len 12288 data: 3 4 3
34302 143/300 Read block 3972 len 4096
34303 144/300 Flush
34304 145/300 Write block 4537 len 16384 data: 8 8 5 5
34305 146/300 Read block 4233 len 32768
34306 147/300 Write block 61 len 32768 data: 5 6 6 8 8 7 6 8
34307 148/300 Read block 2560 len 36864
34308 149/300 Write block 3446 len 28672 data: 7 6 6 5 5 4 2
34309 150/300 Write block 4594 len 16384 data: 4 2 2 2
34310 151/300 Read block 1975 len 4096
34311 152/300 Write block 1480 len 40960 data: 7 10 9 10 9 10 8 7 8 8
34312 153/300 Read block 442 len 20480
34313 154/300 Read block 1950 len 12288
34314 155/300 Write block 1623 len 32768 data: 5 6 5 5 5 5 5 3
34315 156/300 Write block 4336 len 28672 data: 8 14 13 13 10 6 8
34316 157/300 Write block 4053 len 36864 data: 7 8 7 10 10 10 9 11 11
34317 158/300 Read block 4598 len 24576
34318 159/300 Read block 643 len 28672
34319 160/300 Read block 2549 len 32768
34320 161/300 Write block 821 len 12288 data: 4 4 3
34321 162/300 Read block 217 len 28672
34322 163/300 Read block 3579 len 32768
34323 164/300 Flush
34324 165/300 Read block 1310 len 20480
34325 166/300 Write block 1765 len 32768 data: 4 5 5 5 6 6 6 7
34326 167/300 Write block 3701 len 4096 data: 6
34327 168/300 Read block 1905 len 20480
34328 169/300 Read block 4769 len 36864
34329 170/300 Write block 2293 len 40960 data: 7 5 7 7 7 8 8 9 8 8
34330 171/300 Write block 1352 len 40960 data: 5 5 5 5 10 11 12 11 11 9
34331 172/300 Write block 1578 len 12288 data: 5 5 8
34332 173/300 Read block 2711 len 24576
34333 174/300 Write block 190 len 36864 data: 7 6 5 5 5 5 4 4 4
34334 175/300 Read block 405 len 28672
34335 176/300 Write block 137 len 8192 data: 4 6
34336 177/300 Write block 1342 len 32768 data: 6 4 4 4 4 3 3 3
34337 178/300 Read block 1029 len 28672
34338 179/300 Write block 2017 len 8192 data: 7 5
34339 180/300 Flush
34340 181/300 Read block 3761 len 16384
34341 182/300 Read block 880 len 16384
34342 183/300 Read block 497 len 24576
34343 184/300 Read block 4206 len 16384
34344 185/300 Read block 16 len 40960
34345 186/300 Read block 3411 len 20480
34346 187/300 Write block 1317 len 8192 data: 5 5
34347 188/300 Write block 3164 len 32768 data: 7 7 7 9 11 10 9 8
34348 189/300 Write block 4484 len 28672 data: 5 6 8 7 6 6 5
34349 190/300 Write block 1089 len 4096 data: 5
34350 191/300 Flush
34351 192/300 Read block 4302 len 8192
34352 193/300 Read block 2076 len 8192
34353 194/300 Write block 826 len 40960 data: 3 5 4 7 6 6 6 6 6 5
34354 195/300 Write block 1409 len 24576 data: 6 7 8 7 9 8
34355 196/300 Read block 2922 len 4096
34356 197/300 Read block 1045 len 8192
34357 198/300 Flush
34358 199/300 Write block 1428 len 16384 data: 5 4 5 4
34359 200/300 Write block 4718 len 28672 data: 5 5 6 7 7 8 9
34360 201/300 Write block 3666 len 12288 data: 3 3 2
34361 202/300 Write block 3954 len 28672 data: 6 5 5 6 4 4 5
34362 203/300 Write block 4933 len 24576 data: 3 3 3 3 3 5
34363 204/300 Write block 4295 len 28672 data: 6 7 9 10 11 10 12
34364 205/300 Read block 3518 len 40960
34365 206/300 Write block 3951 len 12288 data: 6 7 8
34366 207/300 Write block 4965 len 20480 data: 4 4 3 3 3
34367 208/300 Read block 1614 len 24576
34368 209/300 Flush
34369 210/300 Read block 2036 len 28672
34370 211/300 Write block 1440 len 12288 data: 3 3 2
34371 212/300 Write block 4524 len 32768 data: 4 4 4 5 6 6 8 8
34372 213/300 Write block 2345 len 20480 data: 7 5 5 4 2
34373 214/300 Read block 3658 len 32768
34374 215/300 Read block 2778 len 20480
34375 216/300 Write block 2185 len 24576 data: 7 7 6 3 3 3
34376 217/300 Read block 4430 len 20480
34377 218/300 Read block 1074 len 8192
34378 219/300 Write block 2988 len 32768 data: 8 9 9 8 8 8 7 6
34379 220/300 Read block 1699 len 16384
34380 221/300 Write block 2749 len 8192 data: 4 4
34381 222/300 Read block 867 len 8192
34382 223/300 Read block 2832 len 40960
34383 224/300 Write block 3739 len 24576 data: 6 7 8 9 7 5
34384 225/300 Read block 6 len 4096
34385 226/300 Write block 2700 len 12288 data: 8 6 6
34386 227/300 Read block 3712 len 20480
34387 228/300 Write block 1926 len 12288 data: 11 11 12
34388 229/300 Read block 1702 len 32768
34389 230/300 Write block 3968 len 8192 data: 6 5
34390 231/300 Flush
34391 232/300 Read block 3355 len 8192
34392 233/300 Read block 4226 len 24576
34393 234/300 Write block 2026 len 24576 data: 4 3 4 4 4 5
34394 235/300 Write block 3201 len 28672 data: 7 7 7 6 5 5 6
34395 236/300 Read block 1938 len 32768
34396 237/300 Read block 1463 len 20480
34397 238/300 Read block 3011 len 40960
34398 239/300 Read block 380 len 4096
34399 240/300 Write block 3159 len 16384 data: 5 5 4 6
34400 241/300 Read block 30 len 28672
34401 242/300 Read block 4569 len 4096
34402 243/300 Write block 4630 len 8192 data: 9 9
34403 244/300 Read block 2795 len 12288
34404 245/300 Flush
34405 246/300 Read block 143 len 24576
34406 247/300 Read block 367 len 28672
34407 248/300 Read block 1473 len 40960
34408 249/300 Flush
34409 250/300 Write block 3331 len 40960 data: 8 6 5 5 5 7 5 6 6 5
34410 251/300 Read block 4578 len 32768
34411 252/300 Flush
34412 253/300 Read block 3012 len 16384
34413 254/300 Read block 3039 len 36864
34414 255/300 Read block 2770 len 36864
34415 256/300 Read block 1734 len 24576
34416 257/300 Write block 114 len 20480 data: 5 6 6 8 7
34417 258/300 Write block 2055 len 40960 data: 7 5 4 8 9 9 9 7 7 7
34418 259/300 Flush
34419 260/300 Write block 2022 len 32768 data: 6 6 5 5 5 4 5 5
34420 261/300 Flush
34421 262/300 Write block 2953 len 28672 data: 5 7 7 11 11 10 11
34422 263/300 Read block 387 len 28672
34423 264/300 Read block 2523 len 16384
34424 265/300 Read block 1432 len 28672
34425 266/300 Write block 4023 len 12288 data: 6 4 3
34426 267/300 Read block 4805 len 36864
34427 268/300 Read block 439 len 8192
34428 269/300 Write block 1537 len 16384 data: 4 4 2 2
34429 270/300 Write block 3428 len 32768 data: 6 6 5 5 6 9 10 9
34430 271/300 Read block 3658 len 32768
34431 272/300 Write block 2807 len 24576 data: 6 7 5 3 3 4
34432 273/300 Flush
34433 274/300 Write block 3139 len 20480 data: 6 6 5 6 7
34434 275/300 Write block 2049 len 12288 data: 7 6 5
34435 276/300 Read block 4555 len 20480
34436 277/300 Write block 58 len 32768 data: 7 5 6 6 7 7 9 9
34437 278/300 Read block 3703 len 24576
34438 279/300 Flush
34439 280/300 Read block 1119 len 36864
34440 281/300 Read block 4540 len 36864
34441 282/300 Read block 2625 len 20480
34442 283/300 Write block 1712 len 24576 data: 5 5 6 6 6 8
34443 284/300 Flush
34444 285/300 Write block 2373 len 36864 data: 4 4 4 5 7 7 7 8 7
34445 286/300 Read block 1611 len 8192
34446 287/300 Read block 4561 len 12288
34447 288/300 Read block 1779 len 8192
34448 289/300 Flush
34449 290/300 Read block 1167 len 8192
34450 291/300 Read block 351 len 16384
34451 292/300 Read block 1979 len 20480
34452 293/300 Read block 3489 len 24576
34453 294/300 Write block 1272 len 40960 data: 4 5 7 8 7 7 7 6 5 5
34454 295/300 Read block 2569 len 16384
34455 296/300 Read block 4171 len 28672
34456 297/300 Write block 2428 len 8192 data: 5 6
34457 298/300 Flush
34458 299/300 Write block 1019 len 8192 data: 5 6
34459 300/300 Write block 2454 len 32768 data: 4 4 4 5 6 5 5 3
34460 301/300 Read block 370 len 20480
34461 Main task has work to do, go find it
34462 got dsc Start(1)
34463 start 1
34464 Mar 17 22:24:02.606 INFO request completed, latency_us: 215, response_code: 204, uri: /start/cid/1, method: POST, req_id: 894f74e3-a7b4-4745-96f7-63b36c300b06, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
34465 Replay: started 1, returned:Ok(())
34466 ----------------------------------------------------------------
34467 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
34468 States: Active Offline Active
34469 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34470 9452 Acked Read 2 Done Sent Done false
34471 9453 Acked Read 1 Done Sent Done false
34472 9454 Acked Write 4 Done Sent Done false
34473 9455 Acked Flush 0 Done Sent Done false
34474 9456 Acked Write 10 Done Sent Done false
34475 9457 Acked Write 2 Done Sent Done false
34476 9458 Acked Read 10 Done Sent Done false
34477 9459 Acked Read 10 Done Sent Done false
34478 9460 Acked Read 7 Done Sent Done false
34479 9461 Acked Read 3 Done Sent Done false
34480 9462 Acked Write 5 Done Sent Done false
34481 9463 Acked Write 4 Done Sent Done false
34482 9464 Acked Write 9 Done Sent Done false
34483 9465 Acked Write 10 Done Sent Done false
34484 9466 Acked Read 1 Done Sent Done false
34485 9467 Acked Read 4 Done Sent Done false
34486 9468 Acked Flush 0 Done Sent Done false
34487 9469 Acked Read 5 Done Sent Done false
34488 9470 Acked Flush 0 Done Sent Done false
34489 9471 Acked Read 6 Done Sent Done false
34490 9472 Acked Write 5 Done Sent Done false
34491 9473 Acked Read 4 Done Sent Done false
34492 9474 Acked Write 4 Done Sent Done false
34493 9475 Acked Write 3 Done Sent Done false
34494 9476 Acked Write 3 Done Sent Done false
34495 9477 Acked Read 7 Done Sent Done false
34496 9478 Acked Write 2 Done Sent Done false
34497 9479 Acked Read 4 Done Sent Done false
34498 9480 Acked Write 6 Done Sent Done false
34499 9481 Acked Flush 0 Done Sent Done false
34500 9482 Acked Read 5 Done Sent Done false
34501 9483 Acked Read 1 Done Sent Done false
34502 9484 Acked Write 3 Done Sent Done false
34503 9485 Acked Flush 0 Done Sent Done false
34504 9486 Acked Read 2 Done Sent Done false
34505 9487 Acked Read 9 Done Sent Done false
34506 9488 Acked Write 7 Done Sent Done false
34507 9489 Acked Read 4 Done Sent Done false
34508 9490 Acked Read 6 Done Sent Done false
34509 9491 Acked Read 5 Done Sent Done false
34510 9492 Acked Write 2 Done Sent Done false
34511 9493 Acked Read 5 Done Sent Done false
34512 9494 Acked Flush 0 Done Sent Done false
34513 9495 Acked Write 9 Done Sent Done false
34514 9496 Acked Write 9 Done Sent Done false
34515 9497 Acked Read 10 Done Sent Done false
34516 9498 Acked Write 4 Done Sent Done false
34517 9499 Acked Write 1 Done Sent Done false
34518 9500 Acked Write 4 Done Sent Done false
34519 9501 Acked Write 9 Done Sent Done false
34520 9502 Acked Read 7 Done Sent Done false
34521 9503 Acked Read 2 Done Sent Done false
34522 9504 Acked Read 10 Done Sent Done false
34523 9505 Acked Read 2 Done Sent Done false
34524 9506 Acked Write 3 Done Sent Done false
34525 9507 Acked Flush 0 Done Sent Done false
34526 9508 Acked Read 8 Done Sent Done false
34527 9509 Acked Flush 0 Done Sent Done false
34528 9510 Acked Write 2 Done Sent Done false
34529 9511 Acked Write 10 Done Sent Done false
34530 9512 Acked Read 6 Done Sent Done false
34531 9513 Acked Read 8 Done Sent Done false
34532 9514 Acked Write 7 Done Sent Done false
34533 9515 Acked Read 10 Done Sent Done false
34534 9516 Acked Write 7 Done Sent Done false
34535 9517 Acked Read 10 Done Sent Done false
34536 9518 Acked Read 10 Done Sent Done false
34537 9519 Acked Write 9 Done Sent Done false
34538 9520 Acked Write 5 Done Sent Done false
34539 9521 Acked Read 8 Done Sent Done false
34540 9522 Acked Read 7 Done Sent Done false
34541 9523 Acked Write 2 Done Sent Done false
34542 9524 Acked Write 7 Done Sent Done false
34543 9525 Acked Flush 0 Done Sent Done false
34544 9526 Acked Write 6 Done Sent Done false
34545 9527 Acked Write 5 Done Sent Done false
34546 9528 Acked Write 6 Done Sent Done false
34547 9529 Acked Read 9 Done Sent Done false
34548 9530 Acked Flush 0 Done Sent Done false
34549 9531 Acked Write 10 Done Sent Done false
34550 9532 Acked Write 9 Done Sent Done false
34551 9533 Acked Read 4 Done Sent Done false
34552 9534 Acked Read 2 Done Sent Done false
34553 9535 Acked Write 8 Done Sent Done false
34554 9536 Acked Read 3 Done Sent Done false
34555 9537 Acked Read 3 Done Sent Done false
34556 9538 Acked Read 3 Done Sent Done false
34557 9539 Acked Write 1 Done Sent Done false
34558 9540 Acked Read 7 Done Sent Done false
34559 9541 Acked Read 3 Done Sent Done false
34560 9542 Acked Read 9 Done Sent Done false
34561 9543 Acked Read 2 Done Sent Done false
34562 9544 Acked Read 2 Done Sent Done false
34563 9545 Acked Write 6 Done Sent Done false
34564 9546 Acked Write 2 Done Sent Done false
34565 9547 Acked Read 8 Done Sent Done false
34566 9548 Acked Write 8 Done Sent Done false
34567 9549 Acked Write 2 Done Sent Done false
34568 9550 Acked Read 8 Done Sent Done false
34569 9551 Acked Read 4 Done Sent Done false
34570 9552 Acked Write 6 Done Sent Done false
34571 9553 Acked Read 9 Done Sent Done false
34572 9554 Acked Read 2 Done Sent Done false
34573 9555 Acked Read 2 Done Sent Done false
34574 9556 Acked Flush 0 Done Sent Done false
34575 9557 Acked Read 10 Done Sent Done false
34576 9558 Acked Write 6 Done Sent Done false
34577 9559 Acked Read 4 Done Sent Done false
34578 9560 Acked Read 3 Done Sent Done false
34579 9561 Acked Write 6 Done Sent Done false
34580 9562 Acked Read 2 Done Sent Done false
34581 9563 Acked Write 2 Done Sent Done false
34582 9564 Acked Write 8 Done Sent Done false
34583 9565 Acked Write 9 Done Sent Done false
34584 9566 Acked Write 9 Done Sent Done false
34585 9567 Acked Read 10 Done Sent Done false
34586 9568 Acked Read 9 Done Sent Done false
34587 9569 Acked Read 3 Done Sent Done false
34588 9570 Acked Flush 0 Done Sent Done false
34589 9571 Acked Write 5 Done Sent Done false
34590 9572 Acked Flush 0 Done Sent Done false
34591 9573 Acked Read 1 Done Sent Done false
34592 9574 Acked Read 2 Done Sent Done false
34593 9575 Acked Read 3 Done Sent Done false
34594 9576 Acked Write 10 Done Sent Done false
34595 9577 Acked Flush 0 Done Sent Done false
34596 9578 Acked Read 5 Done Sent Done false
34597 9579 Acked Read 6 Done Sent Done false
34598 9580 Acked Read 5 Done Sent Done false
34599 9581 Acked Write 9 Done Sent Done false
34600 9582 Acked Write 9 Done Sent Done false
34601 9583 Acked Flush 0 Done Sent Done false
34602 9584 Acked Write 3 Done Sent Done false
34603 9585 Acked Read 8 Done Sent Done false
34604 9586 Acked Read 1 Done Sent Done false
34605 9587 Acked Write 4 Done Sent Done false
34606 9588 Acked Read 9 Done Sent Done false
34607 9589 Acked Read 9 Done Sent Done false
34608 9590 Acked Flush 0 Done Sent Done false
34609 9591 Acked Write 7 Done Sent Done false
34610 9592 Acked Write 1 Done Sent Done false
34611 9593 Acked Write 3 Done Sent Done false
34612 9594 Acked Read 1 Done Sent Done false
34613 9595 Acked Flush 0 Done Sent Done false
34614 9596 Acked Write 4 Done Sent Done false
34615 9597 Acked Read 8 Done Sent Done false
34616 9598 Acked Write 8 Done Sent Done false
34617 9599 Acked Read 9 Done Sent Done false
34618 9600 Acked Write 7 Done Sent Done false
34619 9601 Acked Write 4 Done Sent Done false
34620 9602 Acked Read 1 Done Sent Done false
34621 9603 Acked Write 10 Done Sent Done false
34622 9604 Acked Read 5 Done Sent Done false
34623 9605 Acked Read 3 Done Sent Done false
34624 9606 Acked Write 8 Done Sent Done false
34625 9607 Acked Write 7 Done Sent Done false
34626 9608 Acked Write 9 Done Sent Done false
34627 9609 Acked Read 6 Done Sent Done false
34628 9610 Acked Read 7 Done Sent Done false
34629 9611 Acked Read 8 Done Sent Done false
34630 9612 Acked Write 3 Done Sent Done false
34631 9613 Acked Read 7 Done Sent Done false
34632 9614 Acked Read 8 Done Sent Done false
34633 9615 Acked Flush 0 Done Sent Done false
34634 9616 Acked Read 5 Done Sent Done false
34635 9617 Acked Write 8 Done Sent Done false
34636 9618 Acked Write 1 Done Sent Done false
34637 9619 Acked Read 5 Done Sent Done false
34638 9620 Acked Read 9 Done Sent Done false
34639 9621 Acked Write 10 Done Sent Done false
34640 9622 Acked Write 10 Done Sent Done false
34641 9623 Acked Write 3 Done Sent Done false
34642 9624 Acked Read 6 Done Sent Done false
34643 9625 Acked Write 9 Done Sent Done false
34644 9626 Acked Read 7 Done Sent Done false
34645 9627 Acked Write 2 Done Sent Done false
34646 9628 Acked Write 8 Done Sent Done false
34647 9629 Acked Read 7 Done Sent Done false
34648 9630 Acked Write 2 Done Sent Done false
34649 9631 Acked Flush 0 Done Sent Done false
34650 9632 Acked Read 4 Done Sent Done false
34651 9633 Acked Read 4 Done Sent Done false
34652 9634 Acked Read 6 Done Sent Done false
34653 9635 Acked Read 4 Done Sent Done false
34654 9636 Acked Read 10 Done Sent Done false
34655 9637 Acked Read 5 Done Sent Done false
34656 9638 Acked Write 2 Done Sent Done false
34657 9639 Acked Write 8 Done Sent Done false
34658 9640 Acked Write 7 Done Sent Done false
34659 9641 Acked Write 1 Done Sent Done false
34660 9642 Acked Flush 0 Done Sent Done false
34661 9643 Acked Read 2 Done Sent Done false
34662 9644 Acked Read 2 Done Sent Done false
34663 9645 Acked Write 10 Done Sent Done false
34664 9646 Acked Write 6 Done Sent Done false
34665 9647 Acked Read 1 Done Sent Done false
34666 9648 Acked Read 2 Done Sent Done false
34667 9649 Acked Flush 0 Done Sent Done false
34668 9650 Acked Write 4 Done Sent Done false
34669 9651 Acked Write 7 Done Sent Done false
34670 9652 Acked Write 3 Done Sent Done false
34671 9653 Acked Write 7 Done Sent Done false
34672 9654 Acked Write 6 Done Sent Done false
34673 9655 Acked Write 7 Done Sent Done false
34674 9656 Acked Read 10 Done Sent Done false
34675 9657 Acked Write 3 Done Sent Done false
34676 9658 Acked Write 5 Done Sent Done false
34677 9659 Acked Read 6 Done Sent Done false
34678 9660 Acked Flush 0 Done Sent Done false
34679 9661 Acked Read 7 Done Sent Done false
34680 9662 Acked Write 3 Done Sent Done false
34681 9663 Acked Write 8 Done Sent Done false
34682 9664 Acked Write 5 Done Sent Done false
34683 9665 Acked Read 8 Done Sent Done false
34684 9666 Acked Read 5 Done Sent Done false
34685 9667 Acked Write 6 Done Sent Done false
34686 9668 Acked Read 5 Done Sent Done false
34687 9669 Acked Read 2 Done Sent Done false
34688 9670 Acked Write 8 Done Sent Done false
34689 9671 Acked Read 4 Done Sent Done false
34690 9672 Acked Write 2 Done Sent Done false
34691 9673 Acked Read 2 Done Sent Done false
34692 9674 Acked Read 10 Done Sent Done false
34693 9675 Acked Write 6 Done Sent Done false
34694 9676 Acked Read 1 Done Sent Done false
34695 9677 Acked Write 3 Done Sent Done false
34696 9678 Acked Read 5 Done Sent Done false
34697 9679 Acked Write 3 Done Sent Done false
34698 9680 Acked Read 8 Done Sent Done false
34699 9681 Acked Write 2 Done Sent Done false
34700 9682 Acked Flush 0 Done Sent Done false
34701 9683 Acked Read 2 Done Sent Done false
34702 9684 Acked Read 6 Done Sent Done false
34703 9685 Acked Write 6 Done Sent Done false
34704 9686 Acked Write 7 Done Sent Done false
34705 9687 Acked Read 8 Done Sent Done false
34706 9688 Acked Read 5 Done Sent Done false
34707 9689 Acked Read 10 Done Sent Done false
34708 9690 Acked Read 1 Done Sent Done false
34709 9691 Acked Write 4 Done Sent Done false
34710 9692 Acked Read 7 Done Sent Done false
34711 9693 Acked Read 1 Done Sent Done false
34712 9694 Acked Write 2 Done Sent Done false
34713 9695 Acked Read 3 Done Sent Done false
34714 9696 Acked Flush 0 Done Sent Done false
34715 9697 Acked Read 6 Done Sent Done false
34716 9698 Acked Read 7 Done Sent Done false
34717 9699 Acked Read 10 Done Sent Done false
34718 9700 Acked Flush 0 Done Sent Done false
34719 9701 Acked Write 10 Done Sent Done false
34720 9702 Acked Read 8 Done Sent Done false
34721 9703 Acked Flush 0 Done Sent Done false
34722 9704 Acked Read 4 Done Sent Done false
34723 9705 Acked Read 9 Done Sent Done false
34724 9706 Acked Read 9 Done Sent Done false
34725 9707 Acked Read 6 Done Sent Done false
34726 9708 Acked Write 5 Done Sent Done false
34727 9709 Acked Write 10 Done Sent Done false
34728 9710 Acked Flush 0 Done Sent Done false
34729 9711 Acked Write 8 Done Sent Done false
34730 9712 Acked Flush 0 Done Sent Done false
34731 9713 Acked Write 7 Done Sent Done false
34732 9714 Acked Read 7 Done Sent Done false
34733 9715 Acked Read 4 Done Sent Done false
34734 9716 Acked Read 7 Done Sent Done false
34735 9717 Acked Write 3 Done Sent Done false
34736 9718 Acked Read 9 Done Sent Done false
34737 9719 Acked Read 2 Done Sent Done false
34738 9720 Acked Write 4 Done Sent Done false
34739 9721 Acked Write 8 Done Sent Done false
34740 9722 Acked Read 8 Done Sent Done false
34741 9723 Acked Write 6 Done Sent Done false
34742 9724 Acked Flush 0 Done Sent Done false
34743 9725 Acked Write 5 Done Sent Done false
34744 9726 Acked Write 3 Done Sent Done false
34745 9727 Acked Read 5 Done Sent Done false
34746 9728 Acked Write 8 Done Sent Done false
34747 9729 Acked Read 6 Done Sent Done false
34748 9730 Acked Flush 0 Done Sent Done false
34749 9731 Acked Read 9 Done Sent Done false
34750 9732 Acked Read 9 Done Sent Done false
34751 9733 Acked Read 5 Done Sent Done false
34752 9734 Acked Write 6 Done Sent Done false
34753 9735 Acked Flush 0 Done Sent Done false
34754 9736 Acked Write 9 Done Sent Done false
34755 9737 Acked Read 2 Done Sent Done false
34756 9738 Acked Read 3 Done Sent Done false
34757 9739 Acked Read 2 Done Sent Done false
34758 9740 Acked Flush 0 Done Sent Done false
34759 9741 Acked Read 2 Done Sent Done false
34760 9742 Acked Read 4 Done Sent Done false
34761 9743 Acked Read 5 Done Sent Done false
34762 9744 Acked Read 6 Done Sent Done false
34763 9745 Acked Write 10 Done Sent Done false
34764 9746 Acked Read 4 Done Sent Done false
34765 9747 Acked Read 7 Done Sent Done false
34766 9748 Acked Write 2 Done Sent Done false
34767 9749 Acked Flush 0 Done Sent Done false
34768 9750 Acked Write 2 Done Sent Done false
34769 9751 Acked Write 8 Done Sent Done false
34770 9752 Acked Read 5 Done Sent Done false
34771 STATES DS:0 DS:1 DS:2 TOTAL
34772 Sent 0 301 0 301
34773 Done 301 0 301 602
34774 Skipped 0 0 0 0
34775 Error 0 0 0 0
34776 Last Flush: Some(JobId(9749)) Some(JobId(9451)) Some(JobId(9749))
34777 Downstairs last five completed: 9451 9450 9449 9448 9447
34778 Upstairs last five completed: 9752 9751 9750 9749 9748
34779 CLIENT: Up:0 ds:301 act:2
347802025-03-17T22:24:03.604ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
347812025-03-17T22:24:04.325ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
347822025-03-17T22:24:04.825ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
347832025-03-17T22:24:05.599ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
347842025-03-17T22:24:06.329ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
34785 [8820] got start action
34786 [8820] I am going to restart
34787 Starting downstairs at port 8820
34788 Make output file at "/tmp/dsc/downstairs-8820.txt"
34789 [8820][1] reports Starting
34790 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1235)
34791 [8820][1] reports Running
34792 ----------------------------------------------------------------
34793 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
34794 States: Active Offline Active
34795 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
34796 9452 Acked Read 2 Done Sent Done false
34797 9453 Acked Read 1 Done Sent Done false
34798 9454 Acked Write 4 Done Sent Done false
34799 9455 Acked Flush 0 Done Sent Done false
34800 9456 Acked Write 10 Done Sent Done false
34801 9457 Acked Write 2 Done Sent Done false
34802 9458 Acked Read 10 Done Sent Done false
34803 9459 Acked Read 10 Done Sent Done false
34804 9460 Acked Read 7 Done Sent Done false
34805 9461 Acked Read 3 Done Sent Done false
34806 9462 Acked Write 5 Done Sent Done false
34807 9463 Acked Write 4 Done Sent Done false
34808 9464 Acked Write 9 Done Sent Done false
34809 9465 Acked Write 10 Done Sent Done false
34810 9466 Acked Read 1 Done Sent Done false
34811 9467 Acked Read 4 Done Sent Done false
34812 9468 Acked Flush 0 Done Sent Done false
34813 9469 Acked Read 5 Done Sent Done false
34814 9470 Acked Flush 0 Done Sent Done false
34815 9471 Acked Read 6 Done Sent Done false
34816 9472 Acked Write 5 Done Sent Done false
34817 9473 Acked Read 4 Done Sent Done false
34818 9474 Acked Write 4 Done Sent Done false
34819 9475 Acked Write 3 Done Sent Done false
34820 9476 Acked Write 3 Done Sent Done false
34821 9477 Acked Read 7 Done Sent Done false
34822 9478 Acked Write 2 Done Sent Done false
34823 9479 Acked Read 4 Done Sent Done false
34824 9480 Acked Write 6 Done Sent Done false
34825 9481 Acked Flush 0 Done Sent Done false
34826 9482 Acked Read 5 Done Sent Done false
34827 9483 Acked Read 1 Done Sent Done false
34828 9484 Acked Write 3 Done Sent Done false
34829 9485 Acked Flush 0 Done Sent Done false
34830 9486 Acked Read 2 Done Sent Done false
34831 9487 Acked Read 9 Done Sent Done false
34832 9488 Acked Write 7 Done Sent Done false
34833 9489 Acked Read 4 Done Sent Done false
34834 9490 Acked Read 6 Done Sent Done false
34835 9491 Acked Read 5 Done Sent Done false
34836 9492 Acked Write 2 Done Sent Done false
34837 9493 Acked Read 5 Done Sent Done false
34838 9494 Acked Flush 0 Done Sent Done false
34839 9495 Acked Write 9 Done Sent Done false
34840 9496 Acked Write 9 Done Sent Done false
34841 9497 Acked Read 10 Done Sent Done false
34842 9498 Acked Write 4 Done Sent Done false
34843 9499 Acked Write 1 Done Sent Done false
34844 9500 Acked Write 4 Done Sent Done false
34845 9501 Acked Write 9 Done Sent Done false
34846 9502 Acked Read 7 Done Sent Done false
34847 9503 Acked Read 2 Done Sent Done false
34848 9504 Acked Read 10 Done Sent Done false
34849 9505 Acked Read 2 Done Sent Done false
34850 9506 Acked Write 3 Done Sent Done false
34851 9507 Acked Flush 0 Done Sent Done false
34852 9508 Acked Read 8 Done Sent Done false
34853 9509 Acked Flush 0 Done Sent Done false
34854 9510 Acked Write 2 Done Sent Done false
34855 9511 Acked Write 10 Done Sent Done false
34856 9512 Acked Read 6 Done Sent Done false
34857 9513 Acked Read 8 Done Sent Done false
34858 9514 Acked Write 7 Done Sent Done false
34859 9515 Acked Read 10 Done Sent Done false
34860 9516 Acked Write 7 Done Sent Done false
34861 9517 Acked Read 10 Done Sent Done false
34862 9518 Acked Read 10 Done Sent Done false
34863 9519 Acked Write 9 Done Sent Done false
34864 9520 Acked Write 5 Done Sent Done false
34865 9521 Acked Read 8 Done Sent Done false
34866 9522 Acked Read 7 Done Sent Done false
34867 9523 Acked Write 2 Done Sent Done false
34868 9524 Acked Write 7 Done Sent Done false
34869 9525 Acked Flush 0 Done Sent Done false
34870 9526 Acked Write 6 Done Sent Done false
34871 9527 Acked Write 5 Done Sent Done false
34872 9528 Acked Write 6 Done Sent Done false
34873 9529 Acked Read 9 Done Sent Done false
34874 9530 Acked Flush 0 Done Sent Done false
34875 9531 Acked Write 10 Done Sent Done false
34876 9532 Acked Write 9 Done Sent Done false
34877 9533 Acked Read 4 Done Sent Done false
34878 9534 Acked Read 2 Done Sent Done false
34879 9535 Acked Write 8 Done Sent Done false
34880 9536 Acked Read 3 Done Sent Done false
34881 9537 Acked Read 3 Done Sent Done false
34882 9538 Acked Read 3 Done Sent Done false
34883 9539 Acked Write 1 Done Sent Done false
34884 9540 Acked Read 7 Done Sent Done false
34885 9541 Acked Read 3 Done Sent Done false
34886 9542 Acked Read 9 Done Sent Done false
34887 9543 Acked Read 2 Done Sent Done false
34888 9544 Acked Read 2 Done Sent Done false
34889 9545 Acked Write 6 Done Sent Done false
34890 9546 Acked Write 2 Done Sent Done false
34891 9547 Acked Read 8 Done Sent Done false
34892 9548 Acked Write 8 Done Sent Done false
34893 9549 Acked Write 2 Done Sent Done false
34894 9550 Acked Read 8 Done Sent Done false
34895 9551 Acked Read 4 Done Sent Done false
34896 9552 Acked Write 6 Done Sent Done false
34897 9553 Acked Read 9 Done Sent Done false
34898 9554 Acked Read 2 Done Sent Done false
34899 9555 Acked Read 2 Done Sent Done false
34900 9556 Acked Flush 0 Done Sent Done false
34901 9557 Acked Read 10 Done Sent Done false
34902 9558 Acked Write 6 Done Sent Done false
34903 9559 Acked Read 4 Done Sent Done false
34904 9560 Acked Read 3 Done Sent Done false
34905 9561 Acked Write 6 Done Sent Done false
34906 9562 Acked Read 2 Done Sent Done false
34907 9563 Acked Write 2 Done Sent Done false
34908 9564 Acked Write 8 Done Sent Done false
34909 9565 Acked Write 9 Done Sent Done false
34910 9566 Acked Write 9 Done Sent Done false
34911 9567 Acked Read 10 Done Sent Done false
34912 9568 Acked Read 9 Done Sent Done false
34913 9569 Acked Read 3 Done Sent Done false
34914 9570 Acked Flush 0 Done Sent Done false
34915 9571 Acked Write 5 Done Sent Done false
34916 9572 Acked Flush 0 Done Sent Done false
34917 9573 Acked Read 1 Done Sent Done false
34918 9574 Acked Read 2 Done Sent Done false
34919 9575 Acked Read 3 Done Sent Done false
34920 9576 Acked Write 10 Done Sent Done false
34921 9577 Acked Flush 0 Done Sent Done false
34922 9578 Acked Read 5 Done Sent Done false
34923 9579 Acked Read 6 Done Sent Done false
34924 9580 Acked Read 5 Done Sent Done false
34925 9581 Acked Write 9 Done Sent Done false
34926 9582 Acked Write 9 Done Sent Done false
34927 9583 Acked Flush 0 Done Sent Done false
34928 9584 Acked Write 3 Done Sent Done false
34929 9585 Acked Read 8 Done Sent Done false
34930 9586 Acked Read 1 Done Sent Done false
34931 9587 Acked Write 4 Done Sent Done false
34932 9588 Acked Read 9 Done Sent Done false
34933 9589 Acked Read 9 Done Sent Done false
34934 9590 Acked Flush 0 Done Sent Done false
34935 9591 Acked Write 7 Done Sent Done false
34936 9592 Acked Write 1 Done Sent Done false
34937 9593 Acked Write 3 Done Sent Done false
34938 9594 Acked Read 1 Done Sent Done false
34939 9595 Acked Flush 0 Done Sent Done false
34940 9596 Acked Write 4 Done Sent Done false
34941 9597 Acked Read 8 Done Sent Done false
34942 9598 Acked Write 8 Done Sent Done false
34943 9599 Acked Read 9 Done Sent Done false
34944 9600 Acked Write 7 Done Sent Done false
34945 9601 Acked Write 4 Done Sent Done false
34946 9602 Acked Read 1 Done Sent Done false
34947 9603 Acked Write 10 Done Sent Done false
34948 9604 Acked Read 5 Done Sent Done false
34949 9605 Acked Read 3 Done Sent Done false
34950 9606 Acked Write 8 Done Sent Done false
34951 9607 Acked Write 7 Done Sent Done false
34952 9608 Acked Write 9 Done Sent Done false
34953 9609 Acked Read 6 Done Sent Done false
34954 9610 Acked Read 7 Done Sent Done false
34955 9611 Acked Read 8 Done Sent Done false
34956 9612 Acked Write 3 Done Sent Done false
34957 9613 Acked Read 7 Done Sent Done false
34958 9614 Acked Read 8 Done Sent Done false
34959 9615 Acked Flush 0 Done Sent Done false
34960 9616 Acked Read 5 Done Sent Done false
34961 9617 Acked Write 8 Done Sent Done false
34962 9618 Acked Write 1 Done Sent Done false
34963 9619 Acked Read 5 Done Sent Done false
34964 9620 Acked Read 9 Done Sent Done false
34965 9621 Acked Write 10 Done Sent Done false
34966 9622 Acked Write 10 Done Sent Done false
34967 9623 Acked Write 3 Done Sent Done false
34968 9624 Acked Read 6 Done Sent Done false
34969 9625 Acked Write 9 Done Sent Done false
34970 9626 Acked Read 7 Done Sent Done false
34971 9627 Acked Write 2 Done Sent Done false
34972 9628 Acked Write 8 Done Sent Done false
34973 9629 Acked Read 7 Done Sent Done false
34974 9630 Acked Write 2 Done Sent Done false
34975 9631 Acked Flush 0 Done Sent Done false
34976 9632 Acked Read 4 Done Sent Done false
34977 9633 Acked Read 4 Done Sent Done false
34978 9634 Acked Read 6 Done Sent Done false
34979 9635 Acked Read 4 Done Sent Done false
34980 9636 Acked Read 10 Done Sent Done false
34981 9637 Acked Read 5 Done Sent Done false
34982 9638 Acked Write 2 Done Sent Done false
34983 9639 Acked Write 8 Done Sent Done false
34984 9640 Acked Write 7 Done Sent Done false
34985 9641 Acked Write 1 Done Sent Done false
34986 9642 Acked Flush 0 Done Sent Done false
34987 9643 Acked Read 2 Done Sent Done false
34988 9644 Acked Read 2 Done Sent Done false
34989 9645 Acked Write 10 Done Sent Done false
34990 9646 Acked Write 6 Done Sent Done false
34991 9647 Acked Read 1 Done Sent Done false
34992 9648 Acked Read 2 Done Sent Done false
34993 9649 Acked Flush 0 Done Sent Done false
34994 9650 Acked Write 4 Done Sent Done false
34995 9651 Acked Write 7 Done Sent Done false
34996 9652 Acked Write 3 Done Sent Done false
34997 9653 Acked Write 7 Done Sent Done false
34998 9654 Acked Write 6 Done Sent Done false
34999 9655 Acked Write 7 Done Sent Done false
35000 9656 Acked Read 10 Done Sent Done false
35001 9657 Acked Write 3 Done Sent Done false
35002 9658 Acked Write 5 Done Sent Done false
35003 9659 Acked Read 6 Done Sent Done false
35004 9660 Acked Flush 0 Done Sent Done false
35005 9661 Acked Read 7 Done Sent Done false
35006 9662 Acked Write 3 Done Sent Done false
35007 9663 Acked Write 8 Done Sent Done false
35008 9664 Acked Write 5 Done Sent Done false
35009 9665 Acked Read 8 Done Sent Done false
35010 9666 Acked Read 5 Done Sent Done false
35011 9667 Acked Write 6 Done Sent Done false
35012 9668 Acked Read 5 Done Sent Done false
35013 9669 Acked Read 2 Done Sent Done false
35014 9670 Acked Write 8 Done Sent Done false
35015 9671 Acked Read 4 Done Sent Done false
35016 9672 Acked Write 2 Done Sent Done false
35017 9673 Acked Read 2 Done Sent Done false
35018 9674 Acked Read 10 Done Sent Done false
35019 9675 Acked Write 6 Done Sent Done false
35020 9676 Acked Read 1 Done Sent Done false
35021 9677 Acked Write 3 Done Sent Done false
35022 9678 Acked Read 5 Done Sent Done false
35023 9679 Acked Write 3 Done Sent Done false
35024 9680 Acked Read 8 Done Sent Done false
35025 9681 Acked Write 2 Done Sent Done false
35026 9682 Acked Flush 0 Done Sent Done false
35027 9683 Acked Read 2 Done Sent Done false
35028 9684 Acked Read 6 Done Sent Done false
35029 9685 Acked Write 6 Done Sent Done false
35030 9686 Acked Write 7 Done Sent Done false
35031 9687 Acked Read 8 Done Sent Done false
35032 9688 Acked Read 5 Done Sent Done false
35033 9689 Acked Read 10 Done Sent Done false
35034 9690 Acked Read 1 Done Sent Done false
35035 9691 Acked Write 4 Done Sent Done false
35036 9692 Acked Read 7 Done Sent Done false
35037 9693 Acked Read 1 Done Sent Done false
35038 9694 Acked Write 2 Done Sent Done false
35039 9695 Acked Read 3 Done Sent Done false
35040 9696 Acked Flush 0 Done Sent Done false
35041 9697 Acked Read 6 Done Sent Done false
35042 9698 Acked Read 7 Done Sent Done false
35043 9699 Acked Read 10 Done Sent Done false
35044 9700 Acked Flush 0 Done Sent Done false
35045 9701 Acked Write 10 Done Sent Done false
35046 9702 Acked Read 8 Done Sent Done false
35047 9703 Acked Flush 0 Done Sent Done false
35048 9704 Acked Read 4 Done Sent Done false
35049 9705 Acked Read 9 Done Sent Done false
35050 9706 Acked Read 9 Done Sent Done false
35051 9707 Acked Read 6 Done Sent Done false
35052 9708 Acked Write 5 Done Sent Done false
35053 9709 Acked Write 10 Done Sent Done false
35054 9710 Acked Flush 0 Done Sent Done false
35055 9711 Acked Write 8 Done Sent Done false
35056 9712 Acked Flush 0 Done Sent Done false
35057 9713 Acked Write 7 Done Sent Done false
35058 9714 Acked Read 7 Done Sent Done false
35059 9715 Acked Read 4 Done Sent Done false
35060 9716 Acked Read 7 Done Sent Done false
35061 9717 Acked Write 3 Done Sent Done false
35062 9718 Acked Read 9 Done Sent Done false
35063 9719 Acked Read 2 Done Sent Done false
35064 9720 Acked Write 4 Done Sent Done false
35065 9721 Acked Write 8 Done Sent Done false
35066 9722 Acked Read 8 Done Sent Done false
35067 9723 Acked Write 6 Done Sent Done false
35068 9724 Acked Flush 0 Done Sent Done false
35069 9725 Acked Write 5 Done Sent Done false
35070 9726 Acked Write 3 Done Sent Done false
35071 9727 Acked Read 5 Done Sent Done false
35072 9728 Acked Write 8 Done Sent Done false
35073 9729 Acked Read 6 Done Sent Done false
35074 9730 Acked Flush 0 Done Sent Done false
35075 9731 Acked Read 9 Done Sent Done false
35076 9732 Acked Read 9 Done Sent Done false
35077 9733 Acked Read 5 Done Sent Done false
35078 9734 Acked Write 6 Done Sent Done false
35079 9735 Acked Flush 0 Done Sent Done false
35080 9736 Acked Write 9 Done Sent Done false
35081 9737 Acked Read 2 Done Sent Done false
35082 9738 Acked Read 3 Done Sent Done false
35083 9739 Acked Read 2 Done Sent Done false
35084 9740 Acked Flush 0 Done Sent Done false
35085 9741 Acked Read 2 Done Sent Done false
35086 9742 Acked Read 4 Done Sent Done false
35087 9743 Acked Read 5 Done Sent Done false
35088 9744 Acked Read 6 Done Sent Done false
35089 9745 Acked Write 10 Done Sent Done false
35090 9746 Acked Read 4 Done Sent Done false
35091 9747 Acked Read 7 Done Sent Done false
35092 9748 Acked Write 2 Done Sent Done false
35093 9749 Acked Flush 0 Done Sent Done false
35094 9750 Acked Write 2 Done Sent Done false
35095 9751 Acked Write 8 Done Sent Done false
35096 9752 Acked Read 5 Done Sent Done false
35097 STATES DS:0 DS:1 DS:2 TOTAL
35098 Sent 0 301 0 301
35099 Done 301 0 301 602
35100 Skipped 0 0 0 0
35101 Error 0 0 0 0
35102 Last Flush: Some(JobId(9749)) Some(JobId(9451)) Some(JobId(9749))
35103 Downstairs last five completed: 9451 9450 9449 9448 9447
35104 Upstairs last five completed: 9752 9751 9750 9749 9748
35105 CLIENT: Up:0 ds:301 act:2
351062025-03-17T22:24:07.330ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
351072025-03-17T22:24:08.361ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
351082025-03-17T22:24:08.361ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:43147) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
351092025-03-17T22:24:08.362ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
351102025-03-17T22:24:08.363ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
351112025-03-17T22:24:08.363ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
351122025-03-17T22:24:08.363ZINFOcrucible: send last flush ID to this DS: Some(JobId(9451)) = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
351132025-03-17T22:24:08.363ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
351142025-03-17T22:24:08.363ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(9451)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
351152025-03-17T22:24:08.363ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(9451)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
35116 ----------------------------------------------------------------
35117 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
35118 Downstairs last five completed: 9753 9752 9751 9750 9749
35119 Upstairs last five completed: 9753 9752 9751 9750 9749
35120 CLIENT: Up:0 ds:0 act:3
35121 Replay: All jobs finished
35122 Main task has work to do, go find it
35123 got dsc Stop(0)
35124 stop 0
35125 Mar 17 22:24:10.612 INFO request completed, latency_us: 384, response_code: 204, uri: /stop/cid/0, method: POST, req_id: d8439aac-7197-4e76-bbbe-c2f398a99a98, remote_addr: 127.0.0.1:60423, local_addr[8810] Got stop action so:false kr:false
35126 : 127.0.0.1:9998
35127 Mar 17 22:24:10.613 INFO request completed, latency_us: 130, response_code: 200, uri: /state/cid/0, method: GET, req_id:{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:24:10.613571892Z","hostname":"w-01JPK2S810SFAA9787XEG8GFX5","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"0f8310d7-099e-4ead-8484-91ddc162d307"}
35128 c363fea1-71e8-4130-bdb6-f1b300926411, remote_addr: 127.0.0.1:60423, local_addr: 127{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:24:10.613648416Z","hostname":"w-01JPK2S810SFAA9787XEG8GFX5","pid":1206,"":"io task","client":"0","":"downstairs","session_id":"0f8310d7-099e-4ead-8484-91ddc162d307"}
35129 .0.0.1:9998
351302025-03-17T22:24:10.613ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
351312025-03-17T22:24:10.613ZINFOcrucible: client task is exiting = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
35132 [8810] Exited with: ExitStatus(unix_wait_status(9))
35133 [8810][0] reports Exit
35134 Mar 17 22:24:14.615 INFO request completed, latency_us: 233, response_code: 200, uri: /state/cid/0, method: GET, req_id: c9ce5c58-9e36-421c-9e0c-5cc1f848776f, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
35135 001/300 Read block 2759 len 16384
35136 002/300 Read block 4583 len 8192
35137 003/300 Write block 4760 len 16384 data: 5 5 4 6
35138 004/300 Write block 1207 len 40960 data: 3 4 4 5 5 7 7 7 8 8
35139 005/300 Read block 433 len 16384
35140 006/300 Write block 1379 len 16384 data: 4 4 5 5
35141 007/300 Write block 4595 len 20480 data: 3 3 3 4 4
35142 008/300 Read block 1897 len 28672
35143 009/300 Read block 3329 len 4096
35144 010/300 Write block 4883 len 40960 data: 9 10 7 9 8 6 4 3 4 4
35145 011/300 Write block 1323 len 12288 data: 6 6 5
35146 012/300 Write block 1476 len 12288 data: 6 7 8
35147 013/300 Write block 4014 len 36864 data: 3 5 6 6 6 9 7 7 6
35148 014/300 Write block 1514 len 32768 data: 4 4 4 3 4 3 3 4
35149 015/300 Read block 2552 len 20480
35150 016/300 Read block 1699 len 24576
35151 017/300 Write block 2179 len 28672 data: 9 8 9 8 8 7 8
35152 018/300 Read block 2962 len 12288
35153 019/300 Flush
35154 020/300 Read block 4734 len 8192
35155 021/300 Read block 4266 len 16384
35156 022/300 Write block 1676 len 24576 data: 6 5 4 5 7 7
35157 023/300 Read block 2192 len 8192
35158 024/300 Write block 3645 len 20480 data: 11 9 9 8 9
35159 025/300 Write block 2318 len 28672 data: 4 4 5 4 4 5 5
35160 026/300 Read block 4916 len 4096
35161 027/300 Read block 2454 len 16384
35162 028/300 Flush
35163 029/300 Read block 4227 len 4096
35164 030/300 Read block 894 len 4096
35165 031/300 Write block 3608 len 36864 data: 5 5 6 7 7 8 7 7 7
35166 032/300 Write block 1830 len 12288 data: 7 9 7
35167 033/300 Write block 3447 len 36864 data: 7 7 6 6 5 3 3 3 2
35168 034/300 Read block 3490 len 8192
35169 035/300 Flush
35170 036/300 Write block 1400 len 4096 data: 9
35171 037/300 Read block 2336 len 36864
35172 038/300 Write block 1008 len 40960 data: 6 6 7 8 7 6 7 7 6 7
35173 039/300 Read block 1879 len 20480
35174 040/300 Write block 1475 len 4096 data: 6
35175 041/300 Read block 921 len 32768
35176 042/300 Write block 430 len 20480 data: 5 5 5 4 4
35177 043/300 Write block 1300 len 36864 data: 6 7 6 5 4 6 6 7 9
35178 044/300 Write block 1591 len 36864 data: 2 2 2 2 4 4 8 9 9
35179 045/300 Read block 1240 len 28672
35180 046/300 Write block 102 len 32768 data: 5 5 6 5 5 6 5 4
35181 047/300 Read block 3303 len 16384
35182 048/300 Write block 4960 len 8192 data: 3 4
35183 049/300 Read block 2131 len 32768
35184 050/300 Read block 2196 len 36864
35185 051/300 Read block 1346 len 28672
35186 052/300 Read block 3314 len 40960
35187 053/300 Read block 3045 len 28672
35188 054/300 Write block 3169 len 24576 data: 11 10 9 7 9 9
35189 055/300 Read block 2014 len 32768
35190 056/300 Read block 3341 len 32768
35191 057/300 Read block 4546 len 24576
35192 058/300 Write block 128 len 28672 data: 7 7 7 7 4 3 2
35193 059/300 Read block 415 len 36864
35194 060/300 Write block 2380 len 16384 data: 9 8 7 6
35195 061/300 Write block 201 len 32768 data: 6 6 5 3 3 3 3 2
35196 062/300 Read block 1795 len 4096
35197 063/300 Read block 392 len 36864
35198 064/300 Flush
35199 065/300 Read block 1675 len 32768
35200 066/300 Read block 3490 len 20480
35201 067/300 Read block 4396 len 20480
35202 068/300 Read block 3420 len 12288
35203 069/300 Write block 4010 len 8192 data: 2 2
35204 070/300 Write block 612 len 16384 data: 6 8 8 8
35205 071/300 Read block 3362 len 16384
35206 072/300 Flush
35207 073/300 Read block 1114 len 40960
35208 074/300 Read block 2128 len 24576
35209 075/300 Read block 4350 len 28672
35210 076/300 Flush
35211 077/300 Write block 1941 len 40960 data: 6 6 6 6 6 4 4 4 3 3
35212 078/300 Read block 1705 len 40960
35213 079/300 Read block 2003 len 24576
35214 080/300 Read block 2667 len 32768
35215 081/300 Read block 4378 len 36864
35216 082/300 Write block 2277 len 16384 data: 6 8 8 8
35217 083/300 Read block 692 len 24576
35218 084/300 Read block 3043 len 40960
35219 085/300 Read block 4818 len 32768
35220 086/300 Read block 2378 len 24576
35221 087/300 Read block 1562 len 28672
35222 088/300 Read block 4079 len 8192
35223 089/300 Write block 3528 len 12288 data: 4 6 7
35224 090/300 Read block 4640 len 16384
35225 091/300 Read block 1008 len 28672
35226 092/300 Read block 464 len 20480
35227 093/300 Read block 1843 len 20480
35228 094/300 Flush
35229 095/300 Read block 249 len 8192
35230 096/300 Flush
35231 097/300 Read block 4568 len 36864
35232 098/300 Flush
35233 099/300 Write block 4280 len 16384 data: 4 4 5 5
35234 100/300 Read block 4308 len 40960
35235 101/300 Flush
35236 102/300 Write block 1686 len 4096 data: 5
35237 103/300 Write block 1879 len 36864 data: 5 4 6 8 6 8 8 9 10
35238 104/300 Read block 549 len 32768
35239 105/300 Write block 1009 len 40960 data: 7 8 9 8 7 8 8 7 8 6
35240 106/300 Write block 748 len 36864 data: 4 4 5 5 5 5 7 7 6
35241 107/300 Write block 3120 len 8192 data: 6 6
35242 108/300 Read block 1983 len 28672
35243 109/300 Read block 2503 len 12288
35244 110/300 Read block 1203 len 32768
35245 111/300 Read block 1219 len 32768
35246 112/300 Flush
35247 113/300 Read block 1080 len 4096
35248 114/300 Write block 1369 len 36864 data: 4 6 6 5 7 7 7 5 5
35249 115/300 Write block 2509 len 24576 data: 4 4 4 4 4 5
35250 116/300 Flush
35251 117/300 Write block 2377 len 40960 data: 8 8 8 10 9 8 7 6 6 5
35252 118/300 Read block 1700 len 4096
35253 119/300 Read block 4637 len 4096
35254 120/300 Read block 2357 len 4096
35255 121/300 Write block 4436 len 16384 data: 4 5 6 6
35256 122/300 Read block 3314 len 4096
35257 123/300 Read block 1036 len 4096
35258 124/300 Write block 2448 len 8192 data: 4 4
35259 125/300 Read block 1796 len 24576
35260 126/300 Write block 3188 len 32768 data: 5 5 4 4 3 3 3 3
35261 127/300 Write block 1324 len 24576 data: 7 6 8 8 8 5
35262 128/300 Flush
35263 129/300 Read block 2691 len 4096
35264 130/300 Read block 4014 len 16384
35265 131/300 Read block 914 len 36864
35266 132/300 Read block 2442 len 4096
35267 133/300 Flush
35268 134/300 Write block 1122 len 28672 data: 4 4 4 5 6 8 8
35269 135/300 Write block 2062 len 36864 data: 8 8 8 6 7 7 7 8 6
35270 136/300 Read block 4088 len 4096
35271 137/300 Read block 3608 len 32768
35272 138/300 Read block 4515 len 24576
35273 139/300 Read block 3186 len 20480
35274 140/300 Read block 1218 len 8192
35275 141/300 Read block 3187 len 32768
35276 142/300 Read block 1783 len 4096
35277 143/300 Read block 4069 len 24576
35278 144/300 Read block 2975 len 32768
35279 145/300 Write block 1550 len 32768 data: 5 5 4 3 4 4 5 5
35280 146/300 Write block 2027 len 36864 data: 5 6 6 5 6 5 5 5 5
35281 147/300 Write block 139 len 36864 data: 6 7 5 5 5 7 6 4 5
35282 148/300 Write block 1374 len 24576 data: 8 8 6 6 4 5
35283 149/300 Read block 4129 len 32768
35284 150/300 Write block 3073 len 20480 data: 8 7 5 4 4
35285 151/300 Read block 4608 len 28672
35286 152/300 Read block 2121 len 8192
35287 153/300 Read block 2151 len 4096
35288 154/300 Read block 2370 len 32768
35289 155/300 Read block 1296 len 16384
35290 156/300 Read block 4958 len 36864
35291 157/300 Read block 1275 len 28672
35292 158/300 Write block 3006 len 4096 data: 10
35293 159/300 Write block 3668 len 8192 data: 3 4
35294 160/300 Read block 4093 len 8192
35295 161/300 Write block 1055 len 40960 data: 4 4 4 4 4 5 6 5 6 7
35296 162/300 Write block 3198 len 32768 data: 3 6 7 8 8 8 7 6
35297 163/300 Read block 4207 len 4096
35298 164/300 Read block 2582 len 12288
35299 165/300 Read block 3970 len 12288
35300 166/300 Write block 1095 len 12288 data: 7 7 7
35301 167/300 Read block 4192 len 16384
35302 168/300 Read block 1216 len 24576
35303 169/300 Read block 2851 len 24576
35304 170/300 Read block 3098 len 24576
35305 171/300 Write block 2 len 36864 data: 2 4 4 4 5 6 6 5 7
35306 172/300 Write block 629 len 20480 data: 8 8 8 8 7
35307 173/300 Read block 4622 len 16384
35308 174/300 Read block 3771 len 20480
35309 175/300 Write block 1583 len 24576 data: 8 7 7 7 4 3
35310 176/300 Read block 1986 len 4096
35311 177/300 Write block 1977 len 4096 data: 7
35312 178/300 Write block 1891 len 20480 data: 5 4 5 5 6
35313 179/300 Write block 4966 len 16384 data: 5 4 4 4
35314 180/300 Read block 2369 len 4096
35315 181/300 Read block 304 len 24576
35316 182/300 Read block 2905 len 12288
35317 183/300 Read block 722 len 8192
35318 184/300 Write block 3179 len 24576 data: 4 5 4 6 5 4
35319 185/300 Flush
35320 186/300 Write block 688 len 28672 data: 4 4 4 3 2 2 2
35321 187/300 Write block 3067 len 4096 data: 7
35322 188/300 Read block 1809 len 24576
35323 189/300 Read block 3076 len 4096
35324 190/300 Write block 3397 len 40960 data: 4 5 5 5 6 9 11 9 8 7
35325 191/300 Write block 2212 len 36864 data: 6 7 7 8 6 8 8 10 9
35326 192/300 Write block 2672 len 16384 data: 12 12 12 12
35327 193/300 Write block 3349 len 28672 data: 4 5 6 9 10 9 9
35328 194/300 Write block 4216 len 32768 data: 6 8 8 6 6 4 5 5
35329 195/300 Read block 4904 len 36864
35330 196/300 Read block 4887 len 24576
35331 197/300 Read block 3053 len 32768
35332 198/300 Write block 4464 len 12288 data: 7 8 8
35333 199/300 Read block 4453 len 32768
35334 200/300 Write block 3857 len 32768 data: 4 4 5 5 5 5 4 5
35335 201/300 Write block 2933 len 20480 data: 7 8 7 7 7
35336 202/300 Flush
35337 203/300 Flush
35338 204/300 Write block 1073 len 36864 data: 5 7 9 10 10 8 9 8 7
35339 205/300 Write block 220 len 16384 data: 4 4 6 6
35340 206/300 Flush
35341 207/300 Write block 4495 len 24576 data: 4 3 4 4 5 8
35342 208/300 Write block 275 len 20480 data: 6 5 6 5 4
35343 209/300 Flush
35344 210/300 Write block 217 len 4096 data: 3
35345 211/300 Read block 1030 len 4096
35346 212/300 Write block 1736 len 24576 data: 5 5 5 4 3 3
35347 213/300 Write block 2822 len 28672 data: 6 6 9 9 11 12 9
35348 214/300 Write block 1384 len 12288 data: 4 5 5
35349 215/300 Write block 687 len 32768 data: 5 5 5 5 4 3 3 3
35350 216/300 Write block 1773 len 4096 data: 6
35351 217/300 Read block 1586 len 28672
35352 218/300 Write block 2580 len 12288 data: 4 4 3
35353 219/300 Read block 2553 len 40960
35354 220/300 Write block 3693 len 28672 data: 2 2 2 2 3 3 5
35355 221/300 Read block 4947 len 36864
35356 222/300 Write block 4847 len 40960 data: 5 8 8 9 8 7 6 5 5 7
35357 223/300 Flush
35358 224/300 Read block 4055 len 4096
35359 225/300 Read block 202 len 36864
35360 226/300 Flush
35361 227/300 Write block 7 len 40960 data: 7 7 6 8 7 8 9 9 9 7
35362 228/300 Read block 3785 len 24576
35363 229/300 Read block 1667 len 12288
35364 230/300 Flush
35365 231/300 Write block 30 len 36864 data: 7 6 7 8 7 7 7 5 5
35366 232/300 Write block 4264 len 36864 data: 5 6 8 6 5 7 8 8 6
35367 233/300 Read block 2676 len 24576
35368 234/300 Flush
35369 235/300 Write block 4259 len 24576 data: 7 6 8 7 6 6
35370 236/300 Write block 2681 len 12288 data: 4 2 4
35371 237/300 Read block 127 len 28672
35372 238/300 Flush
35373 239/300 Read block 3188 len 36864
35374 240/300 Read block 3700 len 20480
35375 241/300 Read block 187 len 40960
35376 242/300 Read block 2290 len 24576
35377 243/300 Read block 4968 len 24576
35378 244/300 Write block 124 len 24576 data: 10 8 9 7 8 8
35379 245/300 Read block 891 len 20480
35380 246/300 Read block 4940 len 24576
35381 247/300 Read block 3526 len 8192
35382 248/300 Read block 4777 len 20480
35383 249/300 Write block 4187 len 4096 data: 6
35384 250/300 Read block 3942 len 36864
35385 251/300 Read block 339 len 8192
35386 252/300 Write block 1536 len 16384 data: 5 5 5 3
35387 253/300 Read block 3783 len 28672
35388 254/300 Read block 1201 len 28672
35389 255/300 Read block 4806 len 4096
35390 256/300 Read block 1892 len 28672
35391 257/300 Read block 439 len 4096
35392 258/300 Read block 4083 len 32768
35393 259/300 Read block 4622 len 8192
35394 260/300 Write block 2493 len 28672 data: 8 8 10 8 9 10 10
35395 261/300 Read block 246 len 12288
35396 262/300 Write block 175 len 32768 data: 5 5 5 5 4 3 5 5
35397 263/300 Write block 2605 len 8192 data: 7 9
35398 264/300 Read block 2839 len 12288
35399 265/300 Read block 4428 len 12288
35400 266/300 Write block 2448 len 28672 data: 5 5 6 5 4 4 5
35401 267/300 Write block 823 len 12288 data: 4 2 2
35402 268/300 Write block 2174 len 16384 data: 6 6 6 7
35403 269/300 Read block 153 len 40960
35404 270/300 Read block 2461 len 24576
35405 271/300 Read block 2539 len 4096
35406 272/300 Write block 2853 len 36864 data: 8 7 8 7 6 5 4 3 4
35407 273/300 Read block 4726 len 40960
35408 274/300 Read block 3358 len 20480
35409 275/300 Read block 3438 len 4096
35410 276/300 Write block 288 len 32768 data: 4 3 3 3 5 5 5 5
35411 277/300 Read block 3983 len 24576
35412 278/300 Write block 1535 len 36864 data: 6 6 6 6 4 3 2 2 4
35413 279/300 Read block 706 len 16384
35414 280/300 Write block 2782 len 24576 data: 5 6 10 10 8 8
35415 281/300 Write block 2135 len 20480 data: 8 8 8 6 6
35416 282/300 Write block 800 len 36864 data: 8 8 6 5 4 5 4 4 2
35417 283/300 Read block 2064 len 4096
35418 284/300 Write block 1371 len 8192 data: 7 6
35419 285/300 Read block 4029 len 36864
35420 286/300 Write block 572 len 8192 data: 4 4
35421 287/300 Read block 4603 len 20480
35422 288/300 Read block 1686 len 16384
35423 289/300 Write block 4292 len 32768 data: 5 6 5 7 8 10 11 12
35424 290/300 Write block 4925 len 36864 data: 5 5 5 6 6 5 4 4 4
35425 291/300 Write block 4364 len 4096 data: 6
35426 292/300 Read block 4157 len 36864
35427 293/300 Write block 4959 len 20480 data: 3 4 5 4 5
35428 294/300 Write block 1686 len 28672 data: 6 5 5 6 7 7 6
35429 295/300 Read block 644 len 12288
35430 296/300 Read block 2846 len 20480
35431 297/300 Read block 3487 len 36864
35432 298/300 Write block 2760 len 40960 data: 9 7 7 5 5 6 6 8 8 8
35433 299/300 Read block 2699 len 8192
35434 300/300 Read block 4438 len 20480
35435 301/300 Write block 878 len 20480 data: 5 6 6 6 5
35436 Main task has work to do, go find it
35437 got dsc Start(0)
35438 start 0
35439 Mar 17 22:24:14.864 INFO request completed, latency_us: 260, response_code: 204, uri: /start/cid/0, method: POST, req_id: f2683800-63c4-4161-baa3-e5dc0cb596c0, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
35440 Replay: started 0, returned:Ok(())
35441 ----------------------------------------------------------------
35442 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
35443 States: Offline Active Active
35444 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
35445 9754 Acked Read 4 Sent Done Done false
35446 9755 Acked Read 2 Sent Done Done false
35447 9756 Acked Write 4 Sent Done Done false
35448 9757 Acked Write 10 Sent Done Done false
35449 9758 Acked Read 4 Sent Done Done false
35450 9759 Acked Write 4 Sent Done Done false
35451 9760 Acked Write 5 Sent Done Done false
35452 9761 Acked Read 7 Sent Done Done false
35453 9762 Acked Read 1 Sent Done Done false
35454 9763 Acked Write 10 Sent Done Done false
35455 9764 Acked Write 3 Sent Done Done false
35456 9765 Acked Write 3 Sent Done Done false
35457 9766 Acked Write 9 Sent Done Done false
35458 9767 Acked Write 8 Sent Done Done false
35459 9768 Acked Read 5 Sent Done Done false
35460 9769 Acked Read 6 Sent Done Done false
35461 9770 Acked Write 7 Sent Done Done false
35462 9771 Acked Read 3 Sent Done Done false
35463 9772 Acked Flush 0 Sent Done Done false
35464 9773 Acked Read 2 Sent Done Done false
35465 9774 Acked Read 4 Sent Done Done false
35466 9775 Acked Write 6 Sent Done Done false
35467 9776 Acked Read 2 Sent Done Done false
35468 9777 Acked Write 5 Sent Done Done false
35469 9778 Acked Write 7 Sent Done Done false
35470 9779 Acked Read 1 Sent Done Done false
35471 9780 Acked Read 4 Sent Done Done false
35472 9781 Acked Flush 0 Sent Done Done false
35473 9782 Acked Read 1 Sent Done Done false
35474 9783 Acked Read 1 Sent Done Done false
35475 9784 Acked Write 9 Sent Done Done false
35476 9785 Acked Write 3 Sent Done Done false
35477 9786 Acked Write 9 Sent Done Done false
35478 9787 Acked Read 2 Sent Done Done false
35479 9788 Acked Flush 0 Sent Done Done false
35480 9789 Acked Write 1 Sent Done Done false
35481 9790 Acked Read 9 Sent Done Done false
35482 9791 Acked Write 10 Sent Done Done false
35483 9792 Acked Read 5 Sent Done Done false
35484 9793 Acked Write 1 Sent Done Done false
35485 9794 Acked Read 8 Sent Done Done false
35486 9795 Acked Write 5 Sent Done Done false
35487 9796 Acked Write 9 Sent Done Done false
35488 9797 Acked Write 9 Sent Done Done false
35489 9798 Acked Read 7 Sent Done Done false
35490 9799 Acked Write 8 Sent Done Done false
35491 9800 Acked Read 4 Sent Done Done false
35492 9801 Acked Write 2 Sent Done Done false
35493 9802 Acked Read 8 Sent Done Done false
35494 9803 Acked Read 9 Sent Done Done false
35495 9804 Acked Read 7 Sent Done Done false
35496 9805 Acked Read 10 Sent Done Done false
35497 9806 Acked Read 7 Sent Done Done false
35498 9807 Acked Write 6 Sent Done Done false
35499 9808 Acked Read 8 Sent Done Done false
35500 9809 Acked Read 8 Sent Done Done false
35501 9810 Acked Read 6 Sent Done Done false
35502 9811 Acked Write 7 Sent Done Done false
35503 9812 Acked Read 9 Sent Done Done false
35504 9813 Acked Write 4 Sent Done Done false
35505 9814 Acked Write 8 Sent Done Done false
35506 9815 Acked Read 1 Sent Done Done false
35507 9816 Acked Read 9 Sent Done Done false
35508 9817 Acked Flush 0 Sent Done Done false
35509 9818 Acked Read 8 Sent Done Done false
35510 9819 Acked Read 5 Sent Done Done false
35511 9820 Acked Read 5 Sent Done Done false
35512 9821 Acked Read 3 Sent Done Done false
35513 9822 Acked Write 2 Sent Done Done false
35514 9823 Acked Write 4 Sent Done Done false
35515 9824 Acked Read 4 Sent Done Done false
35516 9825 Acked Flush 0 Sent Done Done false
35517 9826 Acked Read 10 Sent Done Done false
35518 9827 Acked Read 6 Sent Done Done false
35519 9828 Acked Read 7 Sent Done Done false
35520 9829 Acked Flush 0 Sent Done Done false
35521 9830 Acked Write 10 Sent Done Done false
35522 9831 Acked Read 10 Sent Done Done false
35523 9832 Acked Read 6 Sent Done Done false
35524 9833 Acked Read 8 Sent Done Done false
35525 9834 Acked Read 9 Sent Done Done false
35526 9835 Acked Write 4 Sent Done Done false
35527 9836 Acked Read 6 Sent Done Done false
35528 9837 Acked Read 10 Sent Done Done false
35529 9838 Acked Read 8 Sent Done Done false
35530 9839 Acked Read 6 Sent Done Done false
35531 9840 Acked Read 7 Sent Done Done false
35532 9841 Acked Read 2 Sent Done Done false
35533 9842 Acked Write 3 Sent Done Done false
35534 9843 Acked Read 4 Sent Done Done false
35535 9844 Acked Read 7 Sent Done Done false
35536 9845 Acked Read 5 Sent Done Done false
35537 9846 Acked Read 5 Sent Done Done false
35538 9847 Acked Flush 0 Sent Done Done false
35539 9848 Acked Read 2 Sent Done Done false
35540 9849 Acked Flush 0 Sent Done Done false
35541 9850 Acked Read 9 Sent Done Done false
35542 9851 Acked Flush 0 Sent Done Done false
35543 9852 Acked Write 4 Sent Done Done false
35544 9853 Acked Read 10 Sent Done Done false
35545 9854 Acked Flush 0 Sent Done Done false
35546 9855 Acked Write 1 Sent Done Done false
35547 9856 Acked Write 9 Sent Done Done false
35548 9857 Acked Read 8 Sent Done Done false
35549 9858 Acked Write 10 Sent Done Done false
35550 9859 Acked Write 9 Sent Done Done false
35551 9860 Acked Write 2 Sent Done Done false
35552 9861 Acked Read 7 Sent Done Done false
35553 9862 Acked Read 3 Sent Done Done false
35554 9863 Acked Read 8 Sent Done Done false
35555 9864 Acked Read 8 Sent Done Done false
35556 9865 Acked Flush 0 Sent Done Done false
35557 9866 Acked Read 1 Sent Done Done false
35558 9867 Acked Write 9 Sent Done Done false
35559 9868 Acked Write 6 Sent Done Done false
35560 9869 Acked Flush 0 Sent Done Done false
35561 9870 Acked Write 10 Sent Done Done false
35562 9871 Acked Read 1 Sent Done Done false
35563 9872 Acked Read 1 Sent Done Done false
35564 9873 Acked Read 1 Sent Done Done false
35565 9874 Acked Write 4 Sent Done Done false
35566 9875 Acked Read 1 Sent Done Done false
35567 9876 Acked Read 1 Sent Done Done false
35568 9877 Acked Write 2 Sent Done Done false
35569 9878 Acked Read 6 Sent Done Done false
35570 9879 Acked Write 8 Sent Done Done false
35571 9880 Acked Write 6 Sent Done Done false
35572 9881 Acked Flush 0 Sent Done Done false
35573 9882 Acked Read 1 Sent Done Done false
35574 9883 Acked Read 4 Sent Done Done false
35575 9884 Acked Read 9 Sent Done Done false
35576 9885 Acked Read 1 Sent Done Done false
35577 9886 Acked Flush 0 Sent Done Done false
35578 9887 Acked Write 7 Sent Done Done false
35579 9888 Acked Write 9 Sent Done Done false
35580 9889 Acked Read 1 Sent Done Done false
35581 9890 Acked Read 8 Sent Done Done false
35582 9891 Acked Read 6 Sent Done Done false
35583 9892 Acked Read 5 Sent Done Done false
35584 9893 Acked Read 2 Sent Done Done false
35585 9894 Acked Read 8 Sent Done Done false
35586 9895 Acked Read 1 Sent Done Done false
35587 9896 Acked Read 6 Sent Done Done false
35588 9897 Acked Read 8 Sent Done Done false
35589 9898 Acked Write 8 Sent Done Done false
35590 9899 Acked Write 9 Sent Done Done false
35591 9900 Acked Write 9 Sent Done Done false
35592 9901 Acked Write 6 Sent Done Done false
35593 9902 Acked Read 8 Sent Done Done false
35594 9903 Acked Write 5 Sent Done Done false
35595 9904 Acked Read 7 Sent Done Done false
35596 9905 Acked Read 2 Sent Done Done false
35597 9906 Acked Read 1 Sent Done Done false
35598 9907 Acked Read 8 Sent Done Done false
35599 9908 Acked Read 4 Sent Done Done false
35600 9909 Acked Read 9 Sent Done Done false
35601 9910 Acked Read 7 Sent Done Done false
35602 9911 Acked Write 1 Sent Done Done false
35603 9912 Acked Write 2 Sent Done Done false
35604 9913 Acked Read 2 Sent Done Done false
35605 9914 Acked Write 10 Sent Done Done false
35606 9915 Acked Write 8 Sent Done Done false
35607 9916 Acked Read 1 Sent Done Done false
35608 9917 Acked Read 3 Sent Done Done false
35609 9918 Acked Read 3 Sent Done Done false
35610 9919 Acked Write 3 Sent Done Done false
35611 9920 Acked Read 4 Sent Done Done false
35612 9921 Acked Read 6 Sent Done Done false
35613 9922 Acked Read 6 Sent Done Done false
35614 9923 Acked Read 6 Sent Done Done false
35615 9924 Acked Write 9 Sent Done Done false
35616 9925 Acked Write 5 Sent Done Done false
35617 9926 Acked Read 4 Sent Done Done false
35618 9927 Acked Read 5 Sent Done Done false
35619 9928 Acked Write 6 Sent Done Done false
35620 9929 Acked Read 1 Sent Done Done false
35621 9930 Acked Write 1 Sent Done Done false
35622 9931 Acked Write 5 Sent Done Done false
35623 9932 Acked Write 4 Sent Done Done false
35624 9933 Acked Read 1 Sent Done Done false
35625 9934 Acked Read 6 Sent Done Done false
35626 9935 Acked Read 3 Sent Done Done false
35627 9936 Acked Read 2 Sent Done Done false
35628 9937 Acked Write 6 Sent Done Done false
35629 9938 Acked Flush 0 Sent Done Done false
35630 9939 Acked Write 7 Sent Done Done false
35631 9940 Acked Write 1 Sent Done Done false
35632 9941 Acked Read 6 Sent Done Done false
35633 9942 Acked Read 1 Sent Done Done false
35634 9943 Acked Write 10 Sent Done Done false
35635 9944 Acked Write 9 Sent Done Done false
35636 9945 Acked Write 4 Sent Done Done false
35637 9946 Acked Write 7 Sent Done Done false
35638 9947 Acked Write 8 Sent Done Done false
35639 9948 Acked Read 9 Sent Done Done false
35640 9949 Acked Read 6 Sent Done Done false
35641 9950 Acked Read 8 Sent Done Done false
35642 9951 Acked Write 3 Sent Done Done false
35643 9952 Acked Read 8 Sent Done Done false
35644 9953 Acked Write 8 Sent Done Done false
35645 9954 Acked Write 5 Sent Done Done false
35646 9955 Acked Flush 0 Sent Done Done false
35647 9956 Acked Flush 0 Sent Done Done false
35648 9957 Acked Write 9 Sent Done Done false
35649 9958 Acked Write 4 Sent Done Done false
35650 9959 Acked Flush 0 Sent Done Done false
35651 9960 Acked Write 6 Sent Done Done false
35652 9961 Acked Write 5 Sent Done Done false
35653 9962 Acked Flush 0 Sent Done Done false
35654 9963 Acked Write 1 Sent Done Done false
35655 9964 Acked Read 1 Sent Done Done false
35656 9965 Acked Write 6 Sent Done Done false
35657 9966 Acked Write 7 Sent Done Done false
35658 9967 Acked Write 3 Sent Done Done false
35659 9968 Acked Write 8 Sent Done Done false
35660 9969 Acked Write 1 Sent Done Done false
35661 9970 Acked Read 7 Sent Done Done false
35662 9971 Acked Write 3 Sent Done Done false
35663 9972 Acked Read 10 Sent Done Done false
35664 9973 Acked Write 7 Sent Done Done false
35665 9974 Acked Read 9 Sent Done Done false
35666 9975 Acked Write 10 Sent Done Done false
35667 9976 Acked Flush 0 Sent Done Done false
35668 9977 Acked Read 1 Sent Done Done false
35669 9978 Acked Read 9 Sent Done Done false
35670 9979 Acked Flush 0 Sent Done Done false
35671 9980 Acked Write 10 Sent Done Done false
35672 9981 Acked Read 6 Sent Done Done false
35673 9982 Acked Read 3 Sent Done Done false
35674 9983 Acked Flush 0 Sent Done Done false
35675 9984 Acked Write 9 Sent Done Done false
35676 9985 Acked Write 9 Sent Done Done false
35677 9986 Acked Read 6 Sent Done Done false
35678 9987 Acked Flush 0 Sent Done Done false
35679 9988 Acked Write 6 Sent Done Done false
35680 9989 Acked Write 3 Sent Done Done false
35681 9990 Acked Read 7 Sent Done Done false
35682 9991 Acked Flush 0 Sent Done Done false
35683 9992 Acked Read 9 Sent Done Done false
35684 9993 Acked Read 5 Sent Done Done false
35685 9994 Acked Read 10 Sent Done Done false
35686 9995 Acked Read 6 Sent Done Done false
35687 9996 Acked Read 6 Sent Done Done false
35688 9997 Acked Write 6 Sent Done Done false
35689 9998 Acked Read 5 Sent Done Done false
35690 9999 Acked Read 6 Sent Done Done false
35691 10000 Acked Read 2 Sent Done Done false
35692 10001 Acked Read 5 Sent Done Done false
35693 10002 Acked Write 1 Sent Done Done false
35694 10003 Acked Read 9 Sent Done Done false
35695 10004 Acked Read 2 Sent Done Done false
35696 10005 Acked Write 4 Sent Done Done false
35697 10006 Acked Read 7 Sent Done Done false
35698 10007 Acked Read 7 Sent Done Done false
35699 10008 Acked Read 1 Sent Done Done false
35700 10009 Acked Read 7 Sent Done Done false
35701 10010 Acked Read 1 Sent Done Done false
35702 10011 Acked Read 8 Sent Done Done false
35703 10012 Acked Read 2 Sent Done Done false
35704 10013 Acked Write 7 Sent Done Done false
35705 10014 Acked Read 3 Sent Done Done false
35706 10015 Acked Write 8 Sent Done Done false
35707 10016 Acked Write 2 Sent Done Done false
35708 10017 Acked Read 3 Sent Done Done false
35709 10018 Acked Read 3 Sent Done Done false
35710 10019 Acked Write 7 Sent Done Done false
35711 10020 Acked Write 3 Sent Done Done false
35712 10021 Acked Write 4 Sent Done Done false
35713 10022 Acked Read 10 Sent Done Done false
35714 10023 Acked Read 6 Sent Done Done false
35715 10024 Acked Read 1 Sent Done Done false
35716 10025 Acked Write 9 Sent Done Done false
35717 10026 Acked Read 10 Sent Done Done false
35718 10027 Acked Read 5 Sent Done Done false
35719 10028 Acked Read 1 Sent Done Done false
35720 10029 Acked Write 8 Sent Done Done false
35721 10030 Acked Read 6 Sent Done Done false
35722 10031 Acked Write 9 Sent Done Done false
35723 10032 Acked Read 4 Sent Done Done false
35724 10033 Acked Write 6 Sent Done Done false
35725 10034 Acked Write 5 Sent Done Done false
35726 10035 Acked Write 9 Sent Done Done false
35727 10036 Acked Read 1 Sent Done Done false
35728 10037 Acked Write 2 Sent Done Done false
35729 10038 Acked Read 9 Sent Done Done false
35730 10039 Acked Write 2 Sent Done Done false
35731 10040 Acked Read 5 Sent Done Done false
35732 10041 Acked Read 4 Sent Done Done false
35733 10042 Acked Write 8 Sent Done Done false
35734 10043 Acked Write 9 Sent Done Done false
35735 10044 Acked Write 1 Sent Done Done false
35736 10045 Acked Read 9 Sent Done Done false
35737 10046 Acked Write 5 Sent Done Done false
35738 10047 Acked Write 7 Sent Done Done false
35739 10048 Acked Read 3 Sent Done Done false
35740 10049 Acked Read 5 Sent Done Done false
35741 10050 Acked Read 9 Sent Done Done false
35742 10051 Acked Write 10 Sent Done Done false
35743 10052 Acked Read 2 Sent Done Done false
35744 10053 Acked Read 5 Sent Done Done false
35745 10054 Acked Write 5 Sent Done Done false
35746 STATES DS:0 DS:1 DS:2 TOTAL
35747 Sent 301 0 0 301
35748 Done 0 301 301 602
35749 Skipped 0 0 0 0
35750 Error 0 0 0 0
35751 Last Flush: Some(JobId(9753)) Some(JobId(9991)) Some(JobId(9991))
35752 Downstairs last five completed: 9753 9752 9751 9750 9749
35753 Upstairs last five completed: 10054 10053 10052 10051 10050
35754 CLIENT: Up:0 ds:301 act:2
357552025-03-17T22:24:15.600ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
357562025-03-17T22:24:16.343ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
357572025-03-17T22:24:17.345ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
357582025-03-17T22:24:18.345ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
35759 ----------------------------------------------------------------
35760 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
35761 States: Offline Active Active
35762 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
35763 9754 Acked Read 4 Sent Done Done false
35764 9755 Acked Read 2 Sent Done Done false
35765 9756 Acked Write 4 Sent Done Done false
35766 9757 Acked Write 10 Sent Done Done false
35767 9758 Acked Read 4 Sent Done Done false
35768 9759 Acked Write 4 Sent Done Done false
35769 9760 Acked Write 5 Sent Done Done false
35770 9761 Acked Read 7 Sent Done Done false
35771 9762 Acked Read 1 Sent Done Done false
35772 9763 Acked Write 10 Sent Done Done false
35773 9764 Acked Write 3 Sent Done Done false
35774 9765 Acked Write 3 Sent Done Done false
35775 9766 Acked Write 9 Sent Done Done false
35776 9767 Acked Write 8 Sent Done Done false
35777 9768 Acked Read 5 Sent Done Done false
35778 9769 Acked Read 6 Sent Done Done false
35779 9770 Acked Write 7 Sent Done Done false
35780 9771 Acked Read 3 Sent Done Done false
35781 9772 Acked Flush 0 Sent Done Done false
35782 9773 Acked Read 2 Sent Done Done false
35783 9774 Acked Read 4 Sent Done Done false
35784 9775 Acked Write 6 Sent Done Done false
35785 9776 Acked Read 2 Sent Done Done false
35786 9777 Acked Write 5 Sent Done Done false
35787 9778 Acked Write 7 Sent Done Done false
35788 9779 Acked Read 1 Sent Done Done false
35789 9780 Acked Read 4 Sent Done Done false
35790 9781 Acked Flush 0 Sent Done Done false
35791 9782 Acked Read 1 Sent Done Done false
35792 9783 Acked Read 1 Sent Done Done false
35793 9784 Acked Write 9 Sent Done Done false
35794 9785 Acked Write 3 Sent Done Done false
35795 9786 Acked Write 9 Sent Done Done false
35796 9787 Acked Read 2 Sent Done Done false
35797 9788 Acked Flush 0 Sent Done Done false
35798 9789 Acked Write 1 Sent Done Done false
35799 9790 Acked Read 9 Sent Done Done false
35800 9791 Acked Write 10 Sent Done Done false
35801 9792 Acked Read 5 Sent Done Done false
35802 9793 Acked Write 1 Sent Done Done false
35803 9794 Acked Read 8 Sent Done Done false
35804 9795 Acked Write 5 Sent Done Done false
35805 9796 Acked Write 9 Sent Done Done false
35806 9797 Acked Write 9 Sent Done Done false
35807 9798 Acked Read 7 Sent Done Done false
35808 9799 Acked Write 8 Sent Done Done false
35809 9800 Acked Read 4 Sent Done Done false
35810 9801 Acked Write 2 Sent Done Done false
35811 9802 Acked Read 8 Sent Done Done false
35812 9803 Acked Read 9 Sent Done Done false
35813 9804 Acked Read 7 Sent Done Done false
35814 9805 Acked Read 10 Sent Done Done false
35815 9806 Acked Read 7 Sent Done Done false
35816 9807 Acked Write 6 Sent Done Done false
35817 9808 Acked Read 8 Sent Done Done false
35818 9809 Acked Read 8 Sent Done Done false
35819 9810 Acked Read 6 Sent Done Done false
35820 9811 Acked Write 7 Sent Done Done false
35821 9812 Acked Read 9 Sent Done Done false
35822 9813 Acked Write 4 Sent Done Done false
35823 9814 Acked Write 8 Sent Done Done false
35824 9815 Acked Read 1 Sent Done Done false
35825 9816 Acked Read 9 Sent Done Done false
35826 9817 Acked Flush 0 Sent Done Done false
35827 9818 Acked Read 8 Sent Done Done false
35828 9819 Acked Read 5 Sent Done Done false
35829 9820 Acked Read 5 Sent Done Done false
35830 9821 Acked Read 3 Sent Done Done false
35831 9822 Acked Write 2 Sent Done Done false
35832 9823 Acked Write 4 Sent Done Done false
35833 9824 Acked Read 4 Sent Done Done false
35834 9825 Acked Flush 0 Sent Done Done false
35835 9826 Acked Read 10 Sent Done Done false
35836 9827 Acked Read 6 Sent Done Done false
35837 9828 Acked Read 7 Sent Done Done false
35838 9829 Acked Flush 0 Sent Done Done false
35839 9830 Acked Write 10 Sent Done Done false
35840 9831 Acked Read 10 Sent Done Done false
35841 9832 Acked Read 6 Sent Done Done false
35842 9833 Acked Read 8 Sent Done Done false
35843 9834 Acked Read 9 Sent Done Done false
35844 9835 Acked Write 4 Sent Done Done false
35845 9836 Acked Read 6 Sent Done Done false
35846 9837 Acked Read 10 Sent Done Done false
35847 9838 Acked Read 8 Sent Done Done false
35848 9839 Acked Read 6 Sent Done Done false
35849 9840 Acked Read 7 Sent Done Done false
35850 9841 Acked Read 2 Sent Done Done false
35851 9842 Acked Write 3 Sent Done Done false
35852 9843 Acked Read 4 Sent Done Done false
35853 9844 Acked Read 7 Sent Done Done false
35854 9845 Acked Read 5 Sent Done Done false
35855 9846 Acked Read 5 Sent Done Done false
35856 9847 Acked Flush 0 Sent Done Done false
35857 9848 Acked Read 2 Sent Done Done false
35858 9849 Acked Flush 0 Sent Done Done false
35859 9850 Acked Read 9 Sent Done Done false
35860 9851 Acked Flush 0 Sent Done Done false
35861 9852 Acked Write 4 Sent Done Done false
35862 9853 Acked Read 10 Sent Done Done false
35863 9854 Acked Flush 0 Sent Done Done false
35864 9855 Acked Write 1 Sent Done Done false
35865 9856 Acked Write 9 Sent Done Done false
35866 9857 Acked Read 8 Sent Done Done false
35867 9858 Acked Write 10 Sent Done Done false
35868 9859 Acked Write 9 Sent Done Done false
35869 9860 Acked Write 2 Sent Done Done false
35870 9861 Acked Read 7 Sent Done Done false
35871 9862 Acked Read 3 Sent Done Done false
35872 9863 Acked Read 8 Sent Done Done false
35873 9864 Acked Read 8 Sent Done Done false
35874 9865 Acked Flush 0 Sent Done Done false
35875 9866 Acked Read 1 Sent Done Done false
35876 9867 Acked Write 9 Sent Done Done false
35877 9868 Acked Write 6 Sent Done Done false
35878 9869 Acked Flush 0 Sent Done Done false
35879 9870 Acked Write 10 Sent Done Done false
35880 9871 Acked Read 1 Sent Done Done false
35881 9872 Acked Read 1 Sent Done Done false
35882 9873 Acked Read 1 Sent Done Done false
35883 9874 Acked Write 4 Sent Done Done false
35884 9875 Acked Read 1 Sent Done Done false
35885 9876 Acked Read 1 Sent Done Done false
35886 9877 Acked Write 2 Sent Done Done false
35887 9878 Acked Read 6 Sent Done Done false
35888 9879 Acked Write 8 Sent Done Done false
35889 9880 Acked Write 6 Sent Done Done false
35890 9881 Acked Flush 0 Sent Done Done false
35891 9882 Acked Read 1 Sent Done Done false
35892 9883 Acked Read 4 Sent Done Done false
35893 9884 Acked Read 9 Sent Done Done false
35894 9885 Acked Read 1 Sent Done Done false
35895 9886 Acked Flush 0 Sent Done Done false
35896 9887 Acked Write 7 Sent Done Done false
35897 9888 Acked Write 9 Sent Done Done false
35898 9889 Acked Read 1 Sent Done Done false
35899 9890 Acked Read 8 Sent Done Done false
35900 9891 Acked Read 6 Sent Done Done false
35901 9892 Acked Read 5 Sent Done Done false
35902 9893 Acked Read 2 Sent Done Done false
35903 9894 Acked Read 8 Sent Done Done false
35904 9895 Acked Read 1 Sent Done Done false
35905 9896 Acked Read 6 Sent Done Done false
35906 9897 Acked Read 8 Sent Done Done false
35907 9898 Acked Write 8 Sent Done Done false
35908 9899 Acked Write 9 Sent Done Done false
35909 9900 Acked Write 9 Sent Done Done false
35910 9901 Acked Write 6 Sent Done Done false
35911 9902 Acked Read 8 Sent Done Done false
35912 9903 Acked Write 5 Sent Done Done false
35913 9904 Acked Read 7 Sent Done Done false
35914 9905 Acked Read 2 Sent Done Done false
35915 9906 Acked Read 1 Sent Done Done false
35916 9907 Acked Read 8 Sent Done Done false
35917 9908 Acked Read 4 Sent Done Done false
35918 9909 Acked Read 9 Sent Done Done false
35919 9910 Acked Read 7 Sent Done Done false
35920 9911 Acked Write 1 Sent Done Done false
35921 9912 Acked Write 2 Sent Done Done false
35922 9913 Acked Read 2 Sent Done Done false
35923 9914 Acked Write 10 Sent Done Done false
35924 9915 Acked Write 8 Sent Done Done false
35925 9916 Acked Read 1 Sent Done Done false
35926 9917 Acked Read 3 Sent Done Done false
35927 9918 Acked Read 3 Sent Done Done false
35928 9919 Acked Write 3 Sent Done Done false
35929 9920 Acked Read 4 Sent Done Done false
35930 9921 Acked Read 6 Sent Done Done false
35931 9922 Acked Read 6 Sent Done Done false
35932 9923 Acked Read 6 Sent Done Done false
35933 9924 Acked Write 9 Sent Done Done false
35934 9925 Acked Write 5 Sent Done Done false
35935 9926 Acked Read 4 Sent Done Done false
35936 9927 Acked Read 5 Sent Done Done false
35937 9928 Acked Write 6 Sent Done Done false
35938 9929 Acked Read 1 Sent Done Done false
35939 9930 Acked Write 1 Sent Done Done false
35940 9931 Acked Write 5 Sent Done Done false
35941 9932 Acked Write 4 Sent Done Done false
35942 9933 Acked Read 1 Sent Done Done false
35943 9934 Acked Read 6 Sent Done Done false
35944 9935 Acked Read 3 Sent Done Done false
35945 9936 Acked Read 2 Sent Done Done false
35946 9937 Acked Write 6 Sent Done Done false
35947 9938 Acked Flush 0 Sent Done Done false
35948 9939 Acked Write 7 Sent Done Done false
35949 9940 Acked Write 1 Sent Done Done false
35950 9941 Acked Read 6 Sent Done Done false
35951 9942 Acked Read 1 Sent Done Done false
35952 9943 Acked Write 10 Sent Done Done false
35953 9944 Acked Write 9 Sent Done Done false
35954 9945 Acked Write 4 Sent Done Done false
35955 9946 Acked Write 7 Sent Done Done false
35956 9947 Acked Write 8 Sent Done Done false
35957 9948 Acked Read 9 Sent Done Done false
35958 9949 Acked Read 6 Sent Done Done false
35959 9950 Acked Read 8 Sent Done Done false
35960 9951 Acked Write 3 Sent Done Done false
35961 9952 Acked Read 8 Sent Done Done false
35962 9953 Acked Write 8 Sent Done Done false
35963 9954 Acked Write 5 Sent Done Done false
35964 9955 Acked Flush 0 Sent Done Done false
35965 9956 Acked Flush 0 Sent Done Done false
35966 9957 Acked Write 9 Sent Done Done false
35967 9958 Acked Write 4 Sent Done Done false
35968 9959 Acked Flush 0 Sent Done Done false
35969 9960 Acked Write 6 Sent Done Done false
35970 9961 Acked Write 5 Sent Done Done false
35971 9962 Acked Flush 0 Sent Done Done false
35972 9963 Acked Write 1 Sent Done Done false
35973 9964 Acked Read 1 Sent Done Done false
35974 9965 Acked Write 6 Sent Done Done false
35975 9966 Acked Write 7 Sent Done Done false
35976 9967 Acked Write 3 Sent Done Done false
35977 9968 Acked Write 8 Sent Done Done false
35978 9969 Acked Write 1 Sent Done Done false
35979 9970 Acked Read 7 Sent Done Done false
35980 9971 Acked Write 3 Sent Done Done false
35981 9972 Acked Read 10 Sent Done Done false
35982 9973 Acked Write 7 Sent Done Done false
35983 9974 Acked Read 9 Sent Done Done false
35984 9975 Acked Write 10 Sent Done Done false
35985 9976 Acked Flush 0 Sent Done Done false
35986 9977 Acked Read 1 Sent Done Done false
35987 9978 Acked Read 9 Sent Done Done false
35988 9979 Acked Flush 0 Sent Done Done false
35989 9980 Acked Write 10 Sent Done Done false
35990 9981 Acked Read 6 Sent Done Done false
35991 9982 Acked Read 3 Sent Done Done false
35992 9983 Acked Flush 0 Sent Done Done false
35993 9984 Acked Write 9 Sent Done Done false
35994 9985 Acked Write 9 Sent Done Done false
35995 9986 Acked Read 6 Sent Done Done false
35996 9987 Acked Flush 0 Sent Done Done false
35997 9988 Acked Write 6 Sent Done Done false
35998 9989 Acked Write 3 Sent Done Done false
35999 9990 Acked Read 7 Sent Done Done false
36000 9991 Acked Flush 0 Sent Done Done false
36001 9992 Acked Read 9 Sent Done Done false
36002 9993 Acked Read 5 Sent Done Done false
36003 9994 Acked Read 10 Sent Done Done false
36004 9995 Acked Read 6 Sent Done Done false
36005 9996 Acked Read 6 Sent Done Done false
36006 9997 Acked Write 6 Sent Done Done false
36007 9998 Acked Read 5 Sent Done Done false
36008 9999 Acked Read 6 Sent Done Done false
36009 10000 Acked Read 2 Sent Done Done false
36010 10001 Acked Read 5 Sent Done Done false
36011 10002 Acked Write 1 Sent Done Done false
36012 10003 Acked Read 9 Sent Done Done false
36013 10004 Acked Read 2 Sent Done Done false
36014 10005 Acked Write 4 Sent Done Done false
36015 10006 Acked Read 7 Sent Done Done false
36016 10007 Acked Read 7 Sent Done Done false
36017 10008 Acked Read 1 Sent Done Done false
36018 10009 Acked Read 7 Sent Done Done false
36019 10010 Acked Read 1 Sent Done Done false
36020 10011 Acked Read 8 Sent Done Done false
36021 10012 Acked Read 2 Sent Done Done false
36022 10013 Acked Write 7 Sent Done Done false
36023 10014 Acked Read 3 Sent Done Done false
36024 10015 Acked Write 8 Sent Done Done false
36025 10016 Acked Write 2 Sent Done Done false
36026 10017 Acked Read 3 Sent Done Done false
36027 10018 Acked Read 3 Sent Done Done false
36028 10019 Acked Write 7 Sent Done Done false
36029 10020 Acked Write 3 Sent Done Done false
36030 10021 Acked Write 4 Sent Done Done false
36031 10022 Acked Read 10 Sent Done Done false
36032 10023 Acked Read 6 Sent Done Done false
36033 10024 Acked Read 1 Sent Done Done false
36034 10025 Acked Write 9 Sent Done Done false
36035 10026 Acked Read 10 Sent Done Done false
36036 10027 Acked Read 5 Sent Done Done false
36037 10028 Acked Read 1 Sent Done Done false
36038 10029 Acked Write 8 Sent Done Done false
36039 10030 Acked Read 6 Sent Done Done false
36040 10031 Acked Write 9 Sent Done Done false
36041 10032 Acked Read 4 Sent Done Done false
36042 10033 Acked Write 6 Sent Done Done false
36043 10034 Acked Write 5 Sent Done Done false
36044 10035 Acked Write 9 Sent Done Done false
36045 10036 Acked Read 1 Sent Done Done false
36046 10037 Acked Write 2 Sent Done Done false
36047 10038 Acked Read 9 Sent Done Done false
36048 10039 Acked Write 2 Sent Done Done false
36049 10040 Acked Read 5 Sent Done Done false
36050 10041 Acked Read 4 Sent Done Done false
36051 10042 Acked Write 8 Sent Done Done false
36052 10043 Acked Write 9 Sent Done Done false
36053 10044 Acked Write 1 Sent Done Done false
36054 10045 Acked Read 9 Sent Done Done false
36055 10046 Acked Write 5 Sent Done Done false
36056 10047 Acked Write 7 Sent Done Done false
36057 10048 Acked Read 3 Sent Done Done false
36058 10049 Acked Read 5 Sent Done Done false
36059 10050 Acked Read 9 Sent Done Done false
36060 10051 Acked Write 10 Sent Done Done false
36061 10052 Acked Read 2 Sent Done Done false
36062 10053 Acked Read 5 Sent Done Done false
36063 10054 Acked Write 5 Sent Done Done false
36064 STATES DS:0 DS:1 DS:2 TOTAL
36065 Sent 301 0 0 301
36066 Done 0 301 301 602
36067 Skipped 0 0 0 0
36068 Error 0 0 0 0
36069 Last Flush: Some(JobId(9753)) Some(JobId(9991)) Some(JobId(9991))
36070 Downstairs last five completed: 9753 9752 9751 9750 9749
36071 Upstairs last five completed: 10054 10053 10052 10051 10050
36072 CLIENT: Up:0 ds:301 act:2
360732025-03-17T22:24:19.369ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
360742025-03-17T22:24:20.602ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
360752025-03-17T22:24:20.615ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
360762025-03-17T22:24:20.615ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
360772025-03-17T22:24:20.615ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
360782025-03-17T22:24:20.615ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
360792025-03-17T22:24:20.615ZINFOcrucible: client task is exiting = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
360802025-03-17T22:24:21.349ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
360812025-03-17T22:24:22.350ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
36082 ----------------------------------------------------------------
36083 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
36084 States: Offline Active Active
36085 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
36086 9754 Acked Read 4 Sent Done Done false
36087 9755 Acked Read 2 Sent Done Done false
36088 9756 Acked Write 4 Sent Done Done false
36089 9757 Acked Write 10 Sent Done Done false
36090 9758 Acked Read 4 Sent Done Done false
36091 9759 Acked Write 4 Sent Done Done false
36092 9760 Acked Write 5 Sent Done Done false
36093 9761 Acked Read 7 Sent Done Done false
36094 9762 Acked Read 1 Sent Done Done false
36095 9763 Acked Write 10 Sent Done Done false
36096 9764 Acked Write 3 Sent Done Done false
36097 9765 Acked Write 3 Sent Done Done false
36098 9766 Acked Write 9 Sent Done Done false
36099 9767 Acked Write 8 Sent Done Done false
36100 9768 Acked Read 5 Sent Done Done false
36101 9769 Acked Read 6 Sent Done Done false
36102 9770 Acked Write 7 Sent Done Done false
36103 9771 Acked Read 3 Sent Done Done false
36104 9772 Acked Flush 0 Sent Done Done false
36105 9773 Acked Read 2 Sent Done Done false
36106 9774 Acked Read 4 Sent Done Done false
36107 9775 Acked Write 6 Sent Done Done false
36108 9776 Acked Read 2 Sent Done Done false
36109 9777 Acked Write 5 Sent Done Done false
36110 9778 Acked Write 7 Sent Done Done false
36111 9779 Acked Read 1 Sent Done Done false
36112 9780 Acked Read 4 Sent Done Done false
36113 9781 Acked Flush 0 Sent Done Done false
36114 9782 Acked Read 1 Sent Done Done false
36115 9783 Acked Read 1 Sent Done Done false
36116 9784 Acked Write 9 Sent Done Done false
36117 9785 Acked Write 3 Sent Done Done false
36118 9786 Acked Write 9 Sent Done Done false
36119 9787 Acked Read 2 Sent Done Done false
36120 9788 Acked Flush 0 Sent Done Done false
36121 9789 Acked Write 1 Sent Done Done false
36122 9790 Acked Read 9 Sent Done Done false
36123 9791 Acked Write 10 Sent Done Done false
36124 9792 Acked Read 5 Sent Done Done false
36125 9793 Acked Write 1 Sent Done Done false
36126 9794 Acked Read 8 Sent Done Done false
36127 9795 Acked Write 5 Sent Done Done false
36128 9796 Acked Write 9 Sent Done Done false
36129 9797 Acked Write 9 Sent Done Done false
36130 9798 Acked Read 7 Sent Done Done false
36131 9799 Acked Write 8 Sent Done Done false
36132 9800 Acked Read 4 Sent Done Done false
36133 9801 Acked Write 2 Sent Done Done false
36134 9802 Acked Read 8 Sent Done Done false
36135 9803 Acked Read 9 Sent Done Done false
36136 9804 Acked Read 7 Sent Done Done false
36137 9805 Acked Read 10 Sent Done Done false
36138 9806 Acked Read 7 Sent Done Done false
36139 9807 Acked Write 6 Sent Done Done false
36140 9808 Acked Read 8 Sent Done Done false
36141 9809 Acked Read 8 Sent Done Done false
36142 9810 Acked Read 6 Sent Done Done false
36143 9811 Acked Write 7 Sent Done Done false
36144 9812 Acked Read 9 Sent Done Done false
36145 9813 Acked Write 4 Sent Done Done false
36146 9814 Acked Write 8 Sent Done Done false
36147 9815 Acked Read 1 Sent Done Done false
36148 9816 Acked Read 9 Sent Done Done false
36149 9817 Acked Flush 0 Sent Done Done false
36150 9818 Acked Read 8 Sent Done Done false
36151 9819 Acked Read 5 Sent Done Done false
36152 9820 Acked Read 5 Sent Done Done false
36153 9821 Acked Read 3 Sent Done Done false
36154 9822 Acked Write 2 Sent Done Done false
36155 9823 Acked Write 4 Sent Done Done false
36156 9824 Acked Read 4 Sent Done Done false
36157 9825 Acked Flush 0 Sent Done Done false
36158 9826 Acked Read 10 Sent Done Done false
36159 9827 Acked Read 6 Sent Done Done false
36160 9828 Acked Read 7 Sent Done Done false
36161 9829 Acked Flush 0 Sent Done Done false
36162 9830 Acked Write 10 Sent Done Done false
36163 9831 Acked Read 10 Sent Done Done false
36164 9832 Acked Read 6 Sent Done Done false
36165 9833 Acked Read 8 Sent Done Done false
36166 9834 Acked Read 9 Sent Done Done false
36167 9835 Acked Write 4 Sent Done Done false
36168 9836 Acked Read 6 Sent Done Done false
36169 9837 Acked Read 10 Sent Done Done false
36170 9838 Acked Read 8 Sent Done Done false
36171 9839 Acked Read 6 Sent Done Done false
36172 9840 Acked Read 7 Sent Done Done false
36173 9841 Acked Read 2 Sent Done Done false
36174 9842 Acked Write 3 Sent Done Done false
36175 9843 Acked Read 4 Sent Done Done false
36176 9844 Acked Read 7 Sent Done Done false
36177 9845 Acked Read 5 Sent Done Done false
36178 9846 Acked Read 5 Sent Done Done false
36179 9847 Acked Flush 0 Sent Done Done false
36180 9848 Acked Read 2 Sent Done Done false
36181 9849 Acked Flush 0 Sent Done Done false
36182 9850 Acked Read 9 Sent Done Done false
36183 9851 Acked Flush 0 Sent Done Done false
36184 9852 Acked Write 4 Sent Done Done false
36185 9853 Acked Read 10 Sent Done Done false
36186 9854 Acked Flush 0 Sent Done Done false
36187 9855 Acked Write 1 Sent Done Done false
36188 9856 Acked Write 9 Sent Done Done false
36189 9857 Acked Read 8 Sent Done Done false
36190 9858 Acked Write 10 Sent Done Done false
36191 9859 Acked Write 9 Sent Done Done false
36192 9860 Acked Write 2 Sent Done Done false
36193 9861 Acked Read 7 Sent Done Done false
36194 9862 Acked Read 3 Sent Done Done false
36195 9863 Acked Read 8 Sent Done Done false
36196 9864 Acked Read 8 Sent Done Done false
36197 9865 Acked Flush 0 Sent Done Done false
36198 9866 Acked Read 1 Sent Done Done false
36199 9867 Acked Write 9 Sent Done Done false
36200 9868 Acked Write 6 Sent Done Done false
36201 9869 Acked Flush 0 Sent Done Done false
36202 9870 Acked Write 10 Sent Done Done false
36203 9871 Acked Read 1 Sent Done Done false
36204 9872 Acked Read 1 Sent Done Done false
36205 9873 Acked Read 1 Sent Done Done false
36206 9874 Acked Write 4 Sent Done Done false
36207 9875 Acked Read 1 Sent Done Done false
36208 9876 Acked Read 1 Sent Done Done false
36209 9877 Acked Write 2 Sent Done Done false
36210 9878 Acked Read 6 Sent Done Done false
36211 9879 Acked Write 8 Sent Done Done false
36212 9880 Acked Write 6 Sent Done Done false
36213 9881 Acked Flush 0 Sent Done Done false
36214 9882 Acked Read 1 Sent Done Done false
36215 9883 Acked Read 4 Sent Done Done false
36216 9884 Acked Read 9 Sent Done Done false
36217 9885 Acked Read 1 Sent Done Done false
36218 9886 Acked Flush 0 Sent Done Done false
36219 9887 Acked Write 7 Sent Done Done false
36220 9888 Acked Write 9 Sent Done Done false
36221 9889 Acked Read 1 Sent Done Done false
36222 9890 Acked Read 8 Sent Done Done false
36223 9891 Acked Read 6 Sent Done Done false
36224 9892 Acked Read 5 Sent Done Done false
36225 9893 Acked Read 2 Sent Done Done false
36226 9894 Acked Read 8 Sent Done Done false
36227 9895 Acked Read 1 Sent Done Done false
36228 9896 Acked Read 6 Sent Done Done false
36229 9897 Acked Read 8 Sent Done Done false
36230 9898 Acked Write 8 Sent Done Done false
36231 9899 Acked Write 9 Sent Done Done false
36232 9900 Acked Write 9 Sent Done Done false
36233 9901 Acked Write 6 Sent Done Done false
36234 9902 Acked Read 8 Sent Done Done false
36235 9903 Acked Write 5 Sent Done Done false
36236 9904 Acked Read 7 Sent Done Done false
36237 9905 Acked Read 2 Sent Done Done false
36238 9906 Acked Read 1 Sent Done Done false
36239 9907 Acked Read 8 Sent Done Done false
36240 9908 Acked Read 4 Sent Done Done false
36241 9909 Acked Read 9 Sent Done Done false
36242 9910 Acked Read 7 Sent Done Done false
36243 9911 Acked Write 1 Sent Done Done false
36244 9912 Acked Write 2 Sent Done Done false
36245 9913 Acked Read 2 Sent Done Done false
36246 9914 Acked Write 10 Sent Done Done false
36247 9915 Acked Write 8 Sent Done Done false
36248 9916 Acked Read 1 Sent Done Done false
36249 9917 Acked Read 3 Sent Done Done false
36250 9918 Acked Read 3 Sent Done Done false
36251 9919 Acked Write 3 Sent Done Done false
36252 9920 Acked Read 4 Sent Done Done false
36253 9921 Acked Read 6 Sent Done Done false
36254 9922 Acked Read 6 Sent Done Done false
36255 9923 Acked Read 6 Sent Done Done false
36256 9924 Acked Write 9 Sent Done Done false
36257 9925 Acked Write 5 Sent Done Done false
36258 9926 Acked Read 4 Sent Done Done false
36259 9927 Acked Read 5 Sent Done Done false
36260 9928 Acked Write 6 Sent Done Done false
36261 9929 Acked Read 1 Sent Done Done false
36262 9930 Acked Write 1 Sent Done Done false
36263 9931 Acked Write 5 Sent Done Done false
36264 9932 Acked Write 4 Sent Done Done false
36265 9933 Acked Read 1 Sent Done Done false
36266 9934 Acked Read 6 Sent Done Done false
36267 9935 Acked Read 3 Sent Done Done false
36268 9936 Acked Read 2 Sent Done Done false
36269 9937 Acked Write 6 Sent Done Done false
36270 9938 Acked Flush 0 Sent Done Done false
36271 9939 Acked Write 7 Sent Done Done false
36272 9940 Acked Write 1 Sent Done Done false
36273 9941 Acked Read 6 Sent Done Done false
36274 9942 Acked Read 1 Sent Done Done false
36275 9943 Acked Write 10 Sent Done Done false
36276 9944 Acked Write 9 Sent Done Done false
36277 9945 Acked Write 4 Sent Done Done false
36278 9946 Acked Write 7 Sent Done Done false
36279 9947 Acked Write 8 Sent Done Done false
36280 9948 Acked Read 9 Sent Done Done false
36281 9949 Acked Read 6 Sent Done Done false
36282 9950 Acked Read 8 Sent Done Done false
36283 9951 Acked Write 3 Sent Done Done false
36284 9952 Acked Read 8 Sent Done Done false
36285 9953 Acked Write 8 Sent Done Done false
36286 9954 Acked Write 5 Sent Done Done false
36287 9955 Acked Flush 0 Sent Done Done false
36288 9956 Acked Flush 0 Sent Done Done false
36289 9957 Acked Write 9 Sent Done Done false
36290 9958 Acked Write 4 Sent Done Done false
36291 9959 Acked Flush 0 Sent Done Done false
36292 9960 Acked Write 6 Sent Done Done false
36293 9961 Acked Write 5 Sent Done Done false
36294 9962 Acked Flush 0 Sent Done Done false
36295 9963 Acked Write 1 Sent Done Done false
36296 9964 Acked Read 1 Sent Done Done false
36297 9965 Acked Write 6 Sent Done Done false
36298 9966 Acked Write 7 Sent Done Done false
36299 9967 Acked Write 3 Sent Done Done false
36300 9968 Acked Write 8 Sent Done Done false
36301 9969 Acked Write 1 Sent Done Done false
36302 9970 Acked Read 7 Sent Done Done false
36303 9971 Acked Write 3 Sent Done Done false
36304 9972 Acked Read 10 Sent Done Done false
36305 9973 Acked Write 7 Sent Done Done false
36306 9974 Acked Read 9 Sent Done Done false
36307 9975 Acked Write 10 Sent Done Done false
36308 9976 Acked Flush 0 Sent Done Done false
36309 9977 Acked Read 1 Sent Done Done false
36310 9978 Acked Read 9 Sent Done Done false
36311 9979 Acked Flush 0 Sent Done Done false
36312 9980 Acked Write 10 Sent Done Done false
36313 9981 Acked Read 6 Sent Done Done false
36314 9982 Acked Read 3 Sent Done Done false
36315 9983 Acked Flush 0 Sent Done Done false
36316 9984 Acked Write 9 Sent Done Done false
36317 9985 Acked Write 9 Sent Done Done false
36318 9986 Acked Read 6 Sent Done Done false
36319 9987 Acked Flush 0 Sent Done Done false
36320 9988 Acked Write 6 Sent Done Done false
36321 9989 Acked Write 3 Sent Done Done false
36322 9990 Acked Read 7 Sent Done Done false
36323 9991 Acked Flush 0 Sent Done Done false
36324 9992 Acked Read 9 Sent Done Done false
36325 9993 Acked Read 5 Sent Done Done false
36326 9994 Acked Read 10 Sent Done Done false
36327 9995 Acked Read 6 Sent Done Done false
36328 9996 Acked Read 6 Sent Done Done false
36329 9997 Acked Write 6 Sent Done Done false
36330 9998 Acked Read 5 Sent Done Done false
36331 9999 Acked Read 6 Sent Done Done false
36332 10000 Acked Read 2 Sent Done Done false
36333 10001 Acked Read 5 Sent Done Done false
36334 10002 Acked Write 1 Sent Done Done false
36335 10003 Acked Read 9 Sent Done Done false
36336 10004 Acked Read 2 Sent Done Done false
36337 10005 Acked Write 4 Sent Done Done false
36338 10006 Acked Read 7 Sent Done Done false
36339 10007 Acked Read 7 Sent Done Done false
36340 10008 Acked Read 1 Sent Done Done false
36341 10009 Acked Read 7 Sent Done Done false
36342 10010 Acked Read 1 Sent Done Done false
36343 10011 Acked Read 8 Sent Done Done false
36344 10012 Acked Read 2 Sent Done Done false
36345 10013 Acked Write 7 Sent Done Done false
36346 10014 Acked Read 3 Sent Done Done false
36347 10015 Acked Write 8 Sent Done Done false
36348 10016 Acked Write 2 Sent Done Done false
36349 10017 Acked Read 3 Sent Done Done false
36350 10018 Acked Read 3 Sent Done Done false
36351 10019 Acked Write 7 Sent Done Done false
36352 10020 Acked Write 3 Sent Done Done false
36353 10021 Acked Write 4 Sent Done Done false
36354 10022 Acked Read 10 Sent Done Done false
36355 10023 Acked Read 6 Sent Done Done false
36356 10024 Acked Read 1 Sent Done Done false
36357 10025 Acked Write 9 Sent Done Done false
36358 10026 Acked Read 10 Sent Done Done false
36359 10027 Acked Read 5 Sent Done Done false
36360 10028 Acked Read 1 Sent Done Done false
36361 10029 Acked Write 8 Sent Done Done false
36362 10030 Acked Read 6 Sent Done Done false
36363 10031 Acked Write 9 Sent Done Done false
36364 10032 Acked Read 4 Sent Done Done false
36365 10033 Acked Write 6 Sent Done Done false
36366 10034 Acked Write 5 Sent Done Done false
36367 10035 Acked Write 9 Sent Done Done false
36368 10036 Acked Read 1 Sent Done Done false
36369 10037 Acked Write 2 Sent Done Done false
36370 10038 Acked Read 9 Sent Done Done false
36371 10039 Acked Write 2 Sent Done Done false
36372 10040 Acked Read 5 Sent Done Done false
36373 10041 Acked Read 4 Sent Done Done false
36374 10042 Acked Write 8 Sent Done Done false
36375 10043 Acked Write 9 Sent Done Done false
36376 10044 Acked Write 1 Sent Done Done false
36377 10045 Acked Read 9 Sent Done Done false
36378 10046 Acked Write 5 Sent Done Done false
36379 10047 Acked Write 7 Sent Done Done false
36380 10048 Acked Read 3 Sent Done Done false
36381 10049 Acked Read 5 Sent Done Done false
36382 10050 Acked Read 9 Sent Done Done false
36383 10051 Acked Write 10 Sent Done Done false
36384 10052 Acked Read 2 Sent Done Done false
36385 10053 Acked Read 5 Sent Done Done false
36386 10054 Acked Write 5 Sent Done Done false
36387 STATES DS:0 DS:1 DS:2 TOTAL
36388 Sent 301 0 0 301
36389 Done 0 301 301 602
36390 Skipped 0 0 0 0
36391 Error 0 0 0 0
36392 Last Flush: Some(JobId(9753)) Some(JobId(9991)) Some(JobId(9991))
36393 Downstairs last five completed: 9753 9752 9751 9750 9749
36394 Upstairs last five completed: 10054 10053 10052 10051 10050
36395 CLIENT: Up:0 ds:301 act:2
363962025-03-17T22:24:24.353ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
36397 [8810] got start action
36398 [8810] I am going to restart
36399 Starting downstairs at port 8810
36400 Make output file at "/tmp/dsc/downstairs-8810.txt"
36401 [8810][0] reports Starting
36402 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1236)
36403 [8810][0] reports Running
364042025-03-17T22:24:25.602ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
364052025-03-17T22:24:26.356ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
36406 ----------------------------------------------------------------
36407 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
36408 States: Offline Active Active
36409 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
36410 9754 Acked Read 4 Sent Done Done false
36411 9755 Acked Read 2 Sent Done Done false
36412 9756 Acked Write 4 Sent Done Done false
36413 9757 Acked Write 10 Sent Done Done false
36414 9758 Acked Read 4 Sent Done Done false
36415 9759 Acked Write 4 Sent Done Done false
36416 9760 Acked Write 5 Sent Done Done false
36417 9761 Acked Read 7 Sent Done Done false
36418 9762 Acked Read 1 Sent Done Done false
36419 9763 Acked Write 10 Sent Done Done false
36420 9764 Acked Write 3 Sent Done Done false
36421 9765 Acked Write 3 Sent Done Done false
36422 9766 Acked Write 9 Sent Done Done false
36423 9767 Acked Write 8 Sent Done Done false
36424 9768 Acked Read 5 Sent Done Done false
36425 9769 Acked Read 6 Sent Done Done false
36426 9770 Acked Write 7 Sent Done Done false
36427 9771 Acked Read 3 Sent Done Done false
36428 9772 Acked Flush 0 Sent Done Done false
36429 9773 Acked Read 2 Sent Done Done false
36430 9774 Acked Read 4 Sent Done Done false
36431 9775 Acked Write 6 Sent Done Done false
36432 9776 Acked Read 2 Sent Done Done false
36433 9777 Acked Write 5 Sent Done Done false
36434 9778 Acked Write 7 Sent Done Done false
36435 9779 Acked Read 1 Sent Done Done false
36436 9780 Acked Read 4 Sent Done Done false
36437 9781 Acked Flush 0 Sent Done Done false
36438 9782 Acked Read 1 Sent Done Done false
36439 9783 Acked Read 1 Sent Done Done false
36440 9784 Acked Write 9 Sent Done Done false
36441 9785 Acked Write 3 Sent Done Done false
36442 9786 Acked Write 9 Sent Done Done false
36443 9787 Acked Read 2 Sent Done Done false
36444 9788 Acked Flush 0 Sent Done Done false
36445 9789 Acked Write 1 Sent Done Done false
36446 9790 Acked Read 9 Sent Done Done false
36447 9791 Acked Write 10 Sent Done Done false
36448 9792 Acked Read 5 Sent Done Done false
36449 9793 Acked Write 1 Sent Done Done false
36450 9794 Acked Read 8 Sent Done Done false
36451 9795 Acked Write 5 Sent Done Done false
36452 9796 Acked Write 9 Sent Done Done false
36453 9797 Acked Write 9 Sent Done Done false
36454 9798 Acked Read 7 Sent Done Done false
36455 9799 Acked Write 8 Sent Done Done false
36456 9800 Acked Read 4 Sent Done Done false
36457 9801 Acked Write 2 Sent Done Done false
36458 9802 Acked Read 8 Sent Done Done false
36459 9803 Acked Read 9 Sent Done Done false
36460 9804 Acked Read 7 Sent Done Done false
36461 9805 Acked Read 10 Sent Done Done false
36462 9806 Acked Read 7 Sent Done Done false
36463 9807 Acked Write 6 Sent Done Done false
36464 9808 Acked Read 8 Sent Done Done false
36465 9809 Acked Read 8 Sent Done Done false
36466 9810 Acked Read 6 Sent Done Done false
36467 9811 Acked Write 7 Sent Done Done false
36468 9812 Acked Read 9 Sent Done Done false
36469 9813 Acked Write 4 Sent Done Done false
36470 9814 Acked Write 8 Sent Done Done false
36471 9815 Acked Read 1 Sent Done Done false
36472 9816 Acked Read 9 Sent Done Done false
36473 9817 Acked Flush 0 Sent Done Done false
36474 9818 Acked Read 8 Sent Done Done false
36475 9819 Acked Read 5 Sent Done Done false
36476 9820 Acked Read 5 Sent Done Done false
36477 9821 Acked Read 3 Sent Done Done false
36478 9822 Acked Write 2 Sent Done Done false
36479 9823 Acked Write 4 Sent Done Done false
36480 9824 Acked Read 4 Sent Done Done false
36481 9825 Acked Flush 0 Sent Done Done false
36482 9826 Acked Read 10 Sent Done Done false
36483 9827 Acked Read 6 Sent Done Done false
36484 9828 Acked Read 7 Sent Done Done false
36485 9829 Acked Flush 0 Sent Done Done false
36486 9830 Acked Write 10 Sent Done Done false
36487 9831 Acked Read 10 Sent Done Done false
36488 9832 Acked Read 6 Sent Done Done false
36489 9833 Acked Read 8 Sent Done Done false
36490 9834 Acked Read 9 Sent Done Done false
36491 9835 Acked Write 4 Sent Done Done false
36492 9836 Acked Read 6 Sent Done Done false
36493 9837 Acked Read 10 Sent Done Done false
36494 9838 Acked Read 8 Sent Done Done false
36495 9839 Acked Read 6 Sent Done Done false
36496 9840 Acked Read 7 Sent Done Done false
36497 9841 Acked Read 2 Sent Done Done false
36498 9842 Acked Write 3 Sent Done Done false
36499 9843 Acked Read 4 Sent Done Done false
36500 9844 Acked Read 7 Sent Done Done false
36501 9845 Acked Read 5 Sent Done Done false
36502 9846 Acked Read 5 Sent Done Done false
36503 9847 Acked Flush 0 Sent Done Done false
36504 9848 Acked Read 2 Sent Done Done false
36505 9849 Acked Flush 0 Sent Done Done false
36506 9850 Acked Read 9 Sent Done Done false
36507 9851 Acked Flush 0 Sent Done Done false
36508 9852 Acked Write 4 Sent Done Done false
36509 9853 Acked Read 10 Sent Done Done false
36510 9854 Acked Flush 0 Sent Done Done false
36511 9855 Acked Write 1 Sent Done Done false
36512 9856 Acked Write 9 Sent Done Done false
36513 9857 Acked Read 8 Sent Done Done false
36514 9858 Acked Write 10 Sent Done Done false
36515 9859 Acked Write 9 Sent Done Done false
36516 9860 Acked Write 2 Sent Done Done false
36517 9861 Acked Read 7 Sent Done Done false
36518 9862 Acked Read 3 Sent Done Done false
36519 9863 Acked Read 8 Sent Done Done false
36520 9864 Acked Read 8 Sent Done Done false
36521 9865 Acked Flush 0 Sent Done Done false
36522 9866 Acked Read 1 Sent Done Done false
36523 9867 Acked Write 9 Sent Done Done false
36524 9868 Acked Write 6 Sent Done Done false
36525 9869 Acked Flush 0 Sent Done Done false
36526 9870 Acked Write 10 Sent Done Done false
36527 9871 Acked Read 1 Sent Done Done false
36528 9872 Acked Read 1 Sent Done Done false
36529 9873 Acked Read 1 Sent Done Done false
36530 9874 Acked Write 4 Sent Done Done false
36531 9875 Acked Read 1 Sent Done Done false
36532 9876 Acked Read 1 Sent Done Done false
36533 9877 Acked Write 2 Sent Done Done false
36534 9878 Acked Read 6 Sent Done Done false
36535 9879 Acked Write 8 Sent Done Done false
36536 9880 Acked Write 6 Sent Done Done false
36537 9881 Acked Flush 0 Sent Done Done false
36538 9882 Acked Read 1 Sent Done Done false
36539 9883 Acked Read 4 Sent Done Done false
36540 9884 Acked Read 9 Sent Done Done false
36541 9885 Acked Read 1 Sent Done Done false
36542 9886 Acked Flush 0 Sent Done Done false
36543 9887 Acked Write 7 Sent Done Done false
36544 9888 Acked Write 9 Sent Done Done false
36545 9889 Acked Read 1 Sent Done Done false
36546 9890 Acked Read 8 Sent Done Done false
36547 9891 Acked Read 6 Sent Done Done false
36548 9892 Acked Read 5 Sent Done Done false
36549 9893 Acked Read 2 Sent Done Done false
36550 9894 Acked Read 8 Sent Done Done false
36551 9895 Acked Read 1 Sent Done Done false
36552 9896 Acked Read 6 Sent Done Done false
36553 9897 Acked Read 8 Sent Done Done false
36554 9898 Acked Write 8 Sent Done Done false
36555 9899 Acked Write 9 Sent Done Done false
36556 9900 Acked Write 9 Sent Done Done false
36557 9901 Acked Write 6 Sent Done Done false
36558 9902 Acked Read 8 Sent Done Done false
36559 9903 Acked Write 5 Sent Done Done false
36560 9904 Acked Read 7 Sent Done Done false
36561 9905 Acked Read 2 Sent Done Done false
36562 9906 Acked Read 1 Sent Done Done false
36563 9907 Acked Read 8 Sent Done Done false
36564 9908 Acked Read 4 Sent Done Done false
36565 9909 Acked Read 9 Sent Done Done false
36566 9910 Acked Read 7 Sent Done Done false
36567 9911 Acked Write 1 Sent Done Done false
36568 9912 Acked Write 2 Sent Done Done false
36569 9913 Acked Read 2 Sent Done Done false
36570 9914 Acked Write 10 Sent Done Done false
36571 9915 Acked Write 8 Sent Done Done false
36572 9916 Acked Read 1 Sent Done Done false
36573 9917 Acked Read 3 Sent Done Done false
36574 9918 Acked Read 3 Sent Done Done false
36575 9919 Acked Write 3 Sent Done Done false
36576 9920 Acked Read 4 Sent Done Done false
36577 9921 Acked Read 6 Sent Done Done false
36578 9922 Acked Read 6 Sent Done Done false
36579 9923 Acked Read 6 Sent Done Done false
36580 9924 Acked Write 9 Sent Done Done false
36581 9925 Acked Write 5 Sent Done Done false
36582 9926 Acked Read 4 Sent Done Done false
36583 9927 Acked Read 5 Sent Done Done false
36584 9928 Acked Write 6 Sent Done Done false
36585 9929 Acked Read 1 Sent Done Done false
36586 9930 Acked Write 1 Sent Done Done false
36587 9931 Acked Write 5 Sent Done Done false
36588 9932 Acked Write 4 Sent Done Done false
36589 9933 Acked Read 1 Sent Done Done false
36590 9934 Acked Read 6 Sent Done Done false
36591 9935 Acked Read 3 Sent Done Done false
36592 9936 Acked Read 2 Sent Done Done false
36593 9937 Acked Write 6 Sent Done Done false
36594 9938 Acked Flush 0 Sent Done Done false
36595 9939 Acked Write 7 Sent Done Done false
36596 9940 Acked Write 1 Sent Done Done false
36597 9941 Acked Read 6 Sent Done Done false
36598 9942 Acked Read 1 Sent Done Done false
36599 9943 Acked Write 10 Sent Done Done false
36600 9944 Acked Write 9 Sent Done Done false
36601 9945 Acked Write 4 Sent Done Done false
36602 9946 Acked Write 7 Sent Done Done false
36603 9947 Acked Write 8 Sent Done Done false
36604 9948 Acked Read 9 Sent Done Done false
36605 9949 Acked Read 6 Sent Done Done false
36606 9950 Acked Read 8 Sent Done Done false
36607 9951 Acked Write 3 Sent Done Done false
36608 9952 Acked Read 8 Sent Done Done false
36609 9953 Acked Write 8 Sent Done Done false
36610 9954 Acked Write 5 Sent Done Done false
36611 9955 Acked Flush 0 Sent Done Done false
36612 9956 Acked Flush 0 Sent Done Done false
36613 9957 Acked Write 9 Sent Done Done false
36614 9958 Acked Write 4 Sent Done Done false
36615 9959 Acked Flush 0 Sent Done Done false
36616 9960 Acked Write 6 Sent Done Done false
36617 9961 Acked Write 5 Sent Done Done false
36618 9962 Acked Flush 0 Sent Done Done false
36619 9963 Acked Write 1 Sent Done Done false
36620 9964 Acked Read 1 Sent Done Done false
36621 9965 Acked Write 6 Sent Done Done false
36622 9966 Acked Write 7 Sent Done Done false
36623 9967 Acked Write 3 Sent Done Done false
36624 9968 Acked Write 8 Sent Done Done false
36625 9969 Acked Write 1 Sent Done Done false
36626 9970 Acked Read 7 Sent Done Done false
36627 9971 Acked Write 3 Sent Done Done false
36628 9972 Acked Read 10 Sent Done Done false
36629 9973 Acked Write 7 Sent Done Done false
36630 9974 Acked Read 9 Sent Done Done false
36631 9975 Acked Write 10 Sent Done Done false
36632 9976 Acked Flush 0 Sent Done Done false
36633 9977 Acked Read 1 Sent Done Done false
36634 9978 Acked Read 9 Sent Done Done false
36635 9979 Acked Flush 0 Sent Done Done false
36636 9980 Acked Write 10 Sent Done Done false
36637 9981 Acked Read 6 Sent Done Done false
36638 9982 Acked Read 3 Sent Done Done false
36639 9983 Acked Flush 0 Sent Done Done false
36640 9984 Acked Write 9 Sent Done Done false
36641 9985 Acked Write 9 Sent Done Done false
36642 9986 Acked Read 6 Sent Done Done false
36643 9987 Acked Flush 0 Sent Done Done false
36644 9988 Acked Write 6 Sent Done Done false
36645 9989 Acked Write 3 Sent Done Done false
36646 9990 Acked Read 7 Sent Done Done false
36647 9991 Acked Flush 0 Sent Done Done false
36648 9992 Acked Read 9 Sent Done Done false
36649 9993 Acked Read 5 Sent Done Done false
36650 9994 Acked Read 10 Sent Done Done false
36651 9995 Acked Read 6 Sent Done Done false
36652 9996 Acked Read 6 Sent Done Done false
36653 9997 Acked Write 6 Sent Done Done false
36654 9998 Acked Read 5 Sent Done Done false
36655 9999 Acked Read 6 Sent Done Done false
36656 10000 Acked Read 2 Sent Done Done false
36657 10001 Acked Read 5 Sent Done Done false
36658 10002 Acked Write 1 Sent Done Done false
36659 10003 Acked Read 9 Sent Done Done false
36660 10004 Acked Read 2 Sent Done Done false
36661 10005 Acked Write 4 Sent Done Done false
36662 10006 Acked Read 7 Sent Done Done false
36663 10007 Acked Read 7 Sent Done Done false
36664 10008 Acked Read 1 Sent Done Done false
36665 10009 Acked Read 7 Sent Done Done false
36666 10010 Acked Read 1 Sent Done Done false
36667 10011 Acked Read 8 Sent Done Done false
36668 10012 Acked Read 2 Sent Done Done false
36669 10013 Acked Write 7 Sent Done Done false
36670 10014 Acked Read 3 Sent Done Done false
36671 10015 Acked Write 8 Sent Done Done false
36672 10016 Acked Write 2 Sent Done Done false
36673 10017 Acked Read 3 Sent Done Done false
36674 10018 Acked Read 3 Sent Done Done false
36675 10019 Acked Write 7 Sent Done Done false
36676 10020 Acked Write 3 Sent Done Done false
36677 10021 Acked Write 4 Sent Done Done false
36678 10022 Acked Read 10 Sent Done Done false
36679 10023 Acked Read 6 Sent Done Done false
36680 10024 Acked Read 1 Sent Done Done false
36681 10025 Acked Write 9 Sent Done Done false
36682 10026 Acked Read 10 Sent Done Done false
36683 10027 Acked Read 5 Sent Done Done false
36684 10028 Acked Read 1 Sent Done Done false
36685 10029 Acked Write 8 Sent Done Done false
36686 10030 Acked Read 6 Sent Done Done false
36687 10031 Acked Write 9 Sent Done Done false
36688 10032 Acked Read 4 Sent Done Done false
36689 10033 Acked Write 6 Sent Done Done false
36690 10034 Acked Write 5 Sent Done Done false
36691 10035 Acked Write 9 Sent Done Done false
36692 10036 Acked Read 1 Sent Done Done false
36693 10037 Acked Write 2 Sent Done Done false
36694 10038 Acked Read 9 Sent Done Done false
36695 10039 Acked Write 2 Sent Done Done false
36696 10040 Acked Read 5 Sent Done Done false
36697 10041 Acked Read 4 Sent Done Done false
36698 10042 Acked Write 8 Sent Done Done false
36699 10043 Acked Write 9 Sent Done Done false
36700 10044 Acked Write 1 Sent Done Done false
36701 10045 Acked Read 9 Sent Done Done false
36702 10046 Acked Write 5 Sent Done Done false
36703 10047 Acked Write 7 Sent Done Done false
36704 10048 Acked Read 3 Sent Done Done false
36705 10049 Acked Read 5 Sent Done Done false
36706 10050 Acked Read 9 Sent Done Done false
36707 10051 Acked Write 10 Sent Done Done false
36708 10052 Acked Read 2 Sent Done Done false
36709 10053 Acked Read 5 Sent Done Done false
36710 10054 Acked Write 5 Sent Done Done false
36711 STATES DS:0 DS:1 DS:2 TOTAL
36712 Sent 301 0 0 301
36713 Done 0 301 301 602
36714 Skipped 0 0 0 0
36715 Error 0 0 0 0
36716 Last Flush: Some(JobId(9753)) Some(JobId(9991)) Some(JobId(9991))
36717 Downstairs last five completed: 9753 9752 9751 9750 9749
36718 Upstairs last five completed: 10054 10053 10052 10051 10050
36719 CLIENT: Up:0 ds:301 act:2
367202025-03-17T22:24:27.373ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
367212025-03-17T22:24:28.358ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
367222025-03-17T22:24:29.360ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
367232025-03-17T22:24:30.603ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
367242025-03-17T22:24:30.617ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
367252025-03-17T22:24:30.617ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:51976) = io task client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
367262025-03-17T22:24:30.618ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
367272025-03-17T22:24:30.619ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
367282025-03-17T22:24:30.619ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
367292025-03-17T22:24:30.619ZINFOcrucible: send last flush ID to this DS: Some(JobId(9753)) = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
367302025-03-17T22:24:30.619ZINFOcrucible: Transition from Offline to Active = downstairs client = 0 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
367312025-03-17T22:24:30.619ZINFOcrucible: [0] client re-new 301 jobs since flush Some(JobId(9753)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
367322025-03-17T22:24:30.619ZINFOcrucible: [0] Replayed 301 jobs since flush: Some(JobId(9753)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
36733 ----------------------------------------------------------------
36734 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:63
36735 States: Active Active Active
36736 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
36737 9992 Acked Read 9 Done Done Done true
36738 9993 Acked Read 5 Done Done Done true
36739 9994 Acked Read 10 Done Done Done true
36740 9995 Acked Read 6 Done Done Done true
36741 9996 Acked Read 6 Done Done Done true
36742 9997 Acked Write 6 Done Done Done true
36743 9998 Acked Read 5 Done Done Done true
36744 9999 Acked Read 6 Done Done Done true
36745 10000 Acked Read 2 Done Done Done true
36746 10001 Acked Read 5 Done Done Done true
36747 10002 Acked Write 1 Done Done Done true
36748 10003 Acked Read 9 Done Done Done true
36749 10004 Acked Read 2 Done Done Done true
36750 10005 Acked Write 4 Done Done Done true
36751 10006 Acked Read 7 Done Done Done true
36752 10007 Acked Read 7 Done Done Done true
36753 10008 Acked Read 1 Done Done Done true
36754 10009 Acked Read 7 Done Done Done true
36755 10010 Acked Read 1 Done Done Done true
36756 10011 Acked Read 8 Done Done Done true
36757 10012 Acked Read 2 Done Done Done true
36758 10013 Acked Write 7 Done Done Done true
36759 10014 Acked Read 3 Done Done Done true
36760 10015 Acked Write 8 Done Done Done true
36761 10016 Acked Write 2 Done Done Done true
36762 10017 Acked Read 3 Done Done Done true
36763 10018 Acked Read 3 Done Done Done true
36764 10019 Acked Write 7 Done Done Done true
36765 10020 Acked Write 3 Done Done Done true
36766 10021 Acked Write 4 Done Done Done true
36767 10022 Acked Read 10 Done Done Done true
36768 10023 Acked Read 6 Done Done Done true
36769 10024 Acked Read 1 Done Done Done true
36770 10025 Acked Write 9 Done Done Done true
36771 10026 Acked Read 10 Done Done Done true
36772 10027 Acked Read 5 Done Done Done true
36773 10028 Acked Read 1 Done Done Done true
36774 10029 Acked Write 8 Done Done Done true
36775 10030 Acked Read 6 Done Done Done true
36776 10031 Acked Write 9 Done Done Done true
36777 10032 Acked Read 4 Done Done Done true
36778 10033 Acked Write 6 Done Done Done true
36779 10034 Acked Write 5 Done Done Done true
36780 10035 Acked Write 9 Done Done Done true
36781 10036 Acked Read 1 Done Done Done true
36782 10037 Acked Write 2 Done Done Done true
36783 10038 Acked Read 9 Done Done Done true
36784 10039 Acked Write 2 Done Done Done true
36785 10040 Acked Read 5 Done Done Done true
36786 10041 Acked Read 4 Done Done Done true
36787 10042 Acked Write 8 Done Done Done true
36788 10043 Acked Write 9 Done Done Done true
36789 10044 Acked Write 1 Done Done Done true
36790 10045 Acked Read 9 Done Done Done true
36791 10046 Acked Write 5 Done Done Done true
36792 10047 Acked Write 7 Done Done Done true
36793 10048 Acked Read 3 Done Done Done true
36794 10049 Acked Read 5 Done Done Done true
36795 10050 Acked Read 9 Done Done Done true
36796 10051 Acked Write 10 Done Done Done true
36797 10052 Acked Read 2 Done Done Done true
36798 10053 Acked Read 5 Done Done Done true
36799 10054 Acked Write 5 Done Done Done true
36800 STATES DS:0 DS:1 DS:2 TOTAL
36801 Sent 0 0 0 0
36802 Done 63 63 63 189
36803 Skipped 0 0 0 0
36804 Error 0 0 0 0
36805 Last Flush: Some(JobId(9991)) Some(JobId(9991)) Some(JobId(9991))
36806 Downstairs last five completed: 9991 9990 9989 9988 9987
36807 Upstairs last five completed: 10054 10053 10052 10051 10050
36808 CLIENT: Up:0 ds:63 act:3
36809 ----------------------------------------------------------------
36810 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
36811 Downstairs last five completed: 10055 10054 10053 10052 10051
36812 Upstairs last five completed: 10055 10054 10053 10052 10051
36813 CLIENT: Up:0 ds:0 act:3
36814 Replay: All jobs finished
36815 Main task has work to do, go find it
36816 got dsc Stop(1)
36817 stop 1
36818 Mar 17 22:24:34.876 INFO request completed, latency_us: 366, response_code: 204, uri: /stop/cid/1[8820] Got stop action so:false kr:false
36819 , method: POST, req_id: 5687f70e-60e2-4e9d-b7fc-7b512f63c1c5, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
36820 Mar 17 22:24:34.877 INFO request completed{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2025-03-17T22:24:34.87773745Z","hostname":"w-01JPK2S810SFAA9787XEG8GFX5","pid":1206,"":"io task","client":"1","":"downstairs","session_id":"0f8310d7-099e-4ead-8484-91ddc162d307"}
36821 , latency_us: 178, response_code: 200, uri: /state/cid/1, method: GET, req_id: 7f43faf6-315f-483a-921a-2c8295634d24, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
368222025-03-17T22:24:34.877ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
368232025-03-17T22:24:34.877ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
368242025-03-17T22:24:34.877ZINFOcrucible: client task is exiting = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
36825 [8820] Exited with: ExitStatus(unix_wait_status(9))
36826 [8820][1] reports Exit
36827 Mar 17 22:24:38.879 INFO request completed, latency_us: 217, response_code: 200, uri: /state/cid/1, method: GET, req_id: 5c60152a-1b25-486e-8463-fd1c4c749dc0, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
36828 001/300 Read block 2476 len 36864
36829 002/300 Write block 1824 len 12288 data: 6 4 7
36830 003/300 Write block 246 len 12288 data: 3 4 4
36831 004/300 Flush
36832 005/300 Write block 3362 len 24576 data: 2 3 4 5 7 7
36833 006/300 Write block 3043 len 40960 data: 5 6 7 7 8 6 5 6 5 6
36834 007/300 Read block 444 len 20480
36835 008/300 Flush
36836 009/300 Read block 3627 len 24576
36837 010/300 Write block 2130 len 12288 data: 6 7 7
36838 011/300 Read block 1911 len 4096
36839 012/300 Flush
36840 013/300 Read block 4090 len 20480
36841 014/300 Write block 119 len 32768 data: 8 8 8 9 9 11 9 10
36842 015/300 Write block 26 len 4096 data: 10
36843 016/300 Read block 2499 len 32768
36844 017/300 Read block 4383 len 28672
36845 018/300 Read block 154 len 20480
36846 019/300 Write block 2357 len 28672 data: 9 10 10 8 7 7 7
36847 020/300 Write block 1859 len 24576 data: 7 8 5 5 6 7
36848 021/300 Write block 1310 len 40960 data: 8 9 8 9 7 6 5 6 6 4
36849 022/300 Write block 1569 len 8192 data: 6 7
36850 023/300 Read block 1791 len 12288
36851 024/300 Read block 3292 len 16384
36852 025/300 Write block 1525 len 40960 data: 10 12 12 11 10 10 10 8 8 7
36853 026/300 Read block 4903 len 20480
36854 027/300 Read block 2145 len 20480
36855 028/300 Read block 4621 len 20480
36856 029/300 Read block 1643 len 4096
36857 030/300 Write block 3129 len 4096 data: 7
36858 031/300 Read block 1446 len 28672
36859 032/300 Write block 1051 len 40960 data: 5 4 4 2 5 5 5 5 5 6
36860 033/300 Write block 2544 len 12288 data: 5 6 5
36861 034/300 Read block 3792 len 24576
36862 035/300 Write block 2648 len 16384 data: 4 4 3 3
36863 036/300 Write block 3668 len 16384 data: 4 5 3 4
36864 037/300 Write block 4625 len 8192 data: 7 7
36865 038/300 Read block 3725 len 12288
36866 039/300 Write block 1589 len 20480 data: 4 3 3 3 3
36867 040/300 Read block 1172 len 24576
36868 041/300 Write block 1676 len 24576 data: 7 6 5 6 8 8
36869 042/300 Write block 1658 len 32768 data: 6 6 6 4 4 4 4 3
36870 043/300 Flush
36871 044/300 Flush
36872 045/300 Write block 731 len 8192 data: 5 5
36873 046/300 Read block 390 len 20480
36874 047/300 Read block 582 len 8192
36875 048/300 Read block 2984 len 40960
36876 049/300 Flush
36877 050/300 Read block 4809 len 16384
36878 051/300 Write block 4355 len 16384 data: 2 2 2 3
36879 052/300 Flush
36880 053/300 Read block 3531 len 12288
36881 054/300 Flush
36882 055/300 Read block 1318 len 20480
36883 056/300 Read block 555 len 20480
36884 057/300 Write block 2928 len 8192 data: 8 9
36885 058/300 Read block 474 len 40960
36886 059/300 Read block 4867 len 20480
36887 060/300 Read block 4663 len 12288
36888 061/300 Read block 2630 len 28672
36889 062/300 Read block 3665 len 4096
36890 063/300 Write block 4891 len 16384 data: 5 5 4 3
36891 064/300 Read block 4164 len 32768
36892 065/300 Write block 1772 len 36864 data: 8 7 5 5 4 6 7 6 4
36893 066/300 Write block 800 len 4096 data: 9
36894 067/300 Read block 1280 len 8192
36895 068/300 Read block 4961 len 20480
36896 069/300 Read block 4571 len 36864
36897 070/300 Read block 1357 len 40960
36898 071/300 Write block 2011 len 4096 data: 5
36899 072/300 Write block 4371 len 28672 data: 9 9 8 7 7 7 8
36900 073/300 Read block 3518 len 8192
36901 074/300 Write block 3197 len 8192 data: 2 4
36902 075/300 Read block 3340 len 40960
36903 076/300 Write block 4399 len 16384 data: 6 7 5 5
36904 077/300 Write block 805 len 16384 data: 6 5 5 3
36905 078/300 Write block 2211 len 12288 data: 6 7 8
36906 079/300 Write block 2650 len 28672 data: 4 4 3 5 4 4 5
36907 080/300 Read block 1113 len 4096
36908 081/300 Write block 4865 len 36864 data: 12 11 11 7 6 8 5 5 7
36909 082/300 Flush
36910 083/300 Write block 2304 len 32768 data: 7 7 5 4 4 5 4 6
36911 084/300 Flush
36912 085/300 Read block 3287 len 8192
36913 086/300 Write block 3502 len 12288 data: 4 4 4
36914 087/300 Write block 3486 len 32768 data: 3 5 5 6 9 11 12 11
36915 088/300 Write block 356 len 36864 data: 7 6 5 6 6 6 7 6 5
36916 089/300 Write block 3695 len 36864 data: 3 3 4 4 6 4 7 6 6
36917 090/300 Write block 2517 len 8192 data: 5 4
36918 091/300 Read block 2277 len 20480
36919 092/300 Write block 2763 len 24576 data: 6 6 7 7 9 9
36920 093/300 Write block 785 len 40960 data: 7 7 7 5 4 4 6 6 5 4
36921 094/300 Write block 2889 len 8192 data: 9 8
36922 095/300 Read block 4104 len 8192
36923 096/300 Read block 4392 len 12288
36924 097/300 Read block 1676 len 40960
36925 098/300 Read block 4033 len 8192
36926 099/300 Read block 2430 len 36864
36927 100/300 Write block 2925 len 36864 data: 6 6 7 9 10 12 10 9 8
36928 101/300 Read block 224 len 8192
36929 102/300 Read block 2297 len 12288
36930 103/300 Write block 2329 len 28672 data: 6 6 7 7 5 6 7
36931 104/300 Write block 800 len 16384 data: 10 9 7 6
36932 105/300 Write block 3620 len 32768 data: 3 4 5 5 4 4 4 4
36933 106/300 Read block 2627 len 16384
36934 107/300 Write block 4061 len 24576 data: 12 11 9 9 7 6
36935 108/300 Flush
36936 109/300 Flush
36937 110/300 Write block 739 len 16384 data: 6 7 5 5
36938 111/300 Read block 1576 len 16384
36939 112/300 Read block 2807 len 8192
36940 113/300 Read block 1593 len 36864
36941 114/300 Read block 3467 len 12288
36942 115/300 Read block 1778 len 4096
36943 116/300 Write block 102 len 16384 data: 6 6 7 6
36944 117/300 Read block 1928 len 8192
36945 118/300 Write block 514 len 24576 data: 3 3 3 2 2 2
36946 119/300 Read block 2763 len 16384
36947 120/300 Write block 2429 len 20480 data: 7 6 5 5 6
36948 121/300 Write block 2097 len 16384 data: 4 5 5 4
36949 122/300 Read block 1306 len 36864
36950 123/300 Read block 1620 len 40960
36951 124/300 Read block 663 len 16384
36952 125/300 Write block 4697 len 24576 data: 4 3 2 2 4 6
36953 126/300 Flush
36954 127/300 Read block 1838 len 28672
36955 128/300 Write block 3077 len 20480 data: 5 2 2 3 4
36956 129/300 Write block 3969 len 20480 data: 6 6 6 5 5
36957 130/300 Write block 3137 len 24576 data: 7 6 7 7 6 7
36958 131/300 Read block 847 len 40960
36959 132/300 Flush
36960 133/300 Read block 3822 len 20480
36961 134/300 Read block 4748 len 4096
36962 135/300 Read block 473 len 32768
36963 136/300 Read block 1096 len 40960
36964 137/300 Write block 1835 len 12288 data: 6 5 6
36965 138/300 Read block 4585 len 28672
36966 139/300 Flush
36967 140/300 Flush
36968 141/300 Write block 1492 len 36864 data: 5 4 3 4 3 4 4 3 4
36969 142/300 Write block 2631 len 12288 data: 5 4 4
36970 143/300 Read block 3525 len 20480
36971 144/300 Write block 1165 len 8192 data: 9 9
36972 145/300 Read block 4169 len 16384
36973 146/300 Read block 3563 len 8192
36974 147/300 Write block 388 len 32768 data: 6 5 3 3 4 3 2 2
36975 148/300 Write block 2898 len 12288 data: 8 7 9
36976 149/300 Read block 2773 len 20480
36977 150/300 Write block 248 len 28672 data: 5 4 5 4 3 4 3
36978 151/300 Write block 2124 len 36864 data: 6 5 5 3 5 5 7 8 8
36979 152/300 Read block 1842 len 16384
36980 153/300 Write block 949 len 24576 data: 7 5 6 7 7 9
36981 154/300 Write block 4802 len 12288 data: 5 5 5
36982 155/300 Read block 4858 len 16384
36983 156/300 Flush
36984 157/300 Read block 1153 len 28672
36985 158/300 Read block 2044 len 36864
36986 159/300 Read block 725 len 36864
36987 160/300 Write block 3981 len 28672 data: 4 4 4 4 3 3 4
36988 161/300 Read block 1354 len 20480
36989 162/300 Read block 1811 len 8192
36990 163/300 Flush
36991 164/300 Flush
36992 165/300 Write block 4795 len 20480 data: 4 5 5 6 6
36993 166/300 Read block 2859 len 36864
36994 167/300 Read block 1786 len 36864
36995 168/300 Read block 4787 len 4096
36996 169/300 Write block 583 len 32768 data: 4 5 6 6 7 7 9 11
36997 170/300 Read block 67 len 36864
36998 171/300 Flush
36999 172/300 Read block 4337 len 28672
37000 173/300 Read block 1651 len 16384
37001 174/300 Flush
37002 175/300 Read block 265 len 36864
37003 176/300 Flush
37004 177/300 Write block 3928 len 4096 data: 7
37005 178/300 Read block 1560 len 28672
37006 179/300 Read block 3473 len 40960
37007 180/300 Write block 2712 len 36864 data: 2 3 4 4 5 5 6 6 7
37008 181/300 Read block 4546 len 36864
37009 182/300 Write block 3655 len 8192 data: 5 5
37010 183/300 Flush
37011 184/300 Read block 4381 len 24576
37012 185/300 Read block 70 len 28672
37013 186/300 Read block 4423 len 12288
37014 187/300 Read block 2534 len 40960
37015 188/300 Read block 4735 len 8192
37016 189/300 Read block 820 len 40960
37017 190/300 Write block 1078 len 24576 data: 9 10 9 8 7 6
37018 191/300 Write block 243 len 8192 data: 4 4
37019 192/300 Read block 4218 len 32768
37020 193/300 Write block 4716 len 4096 data: 4
37021 194/300 Read block 4223 len 4096
37022 195/300 Read block 1055 len 4096
37023 196/300 Write block 3224 len 20480 data: 5 5 7 8 8
37024 197/300 Write block 1842 len 32768 data: 7 6 7 8 6 6 6 5
37025 198/300 Read block 1374 len 4096
37026 199/300 Read block 1508 len 20480
37027 200/300 Read block 234 len 40960
37028 201/300 Flush
37029 202/300 Write block 2500 len 16384 data: 8 6 5 5
37030 203/300 Write block 611 len 28672 data: 6 7 9 9 9 7 9
37031 204/300 Write block 4392 len 40960 data: 4 5 5 4 5 4 7 7 8 6
37032 205/300 Read block 2662 len 8192
37033 206/300 Write block 2251 len 8192 data: 5 5
37034 207/300 Read block 284 len 4096
37035 208/300 Write block 3989 len 28672 data: 3 3 4 4 6 6 5
37036 209/300 Read block 3754 len 4096
37037 210/300 Write block 1062 len 36864 data: 6 7 8 6 8 7 5 6 4
37038 211/300 Flush
37039 212/300 Read block 112 len 36864
37040 213/300 Flush
37041 214/300 Read block 345 len 12288
37042 215/300 Read block 2820 len 24576
37043 216/300 Write block 1082 len 8192 data: 8 7
37044 217/300 Read block 2260 len 32768
37045 218/300 Read block 3612 len 4096
37046 219/300 Write block 976 len 28672 data: 2 2 2 3 4 4 5
37047 220/300 Read block 3544 len 32768
37048 221/300 Write block 1212 len 36864 data: 8 8 8 9 9 9 7 9 8
37049 222/300 Write block 942 len 4096 data: 4
37050 223/300 Write block 3553 len 24576 data: 4 4 4 4 6 7
37051 224/300 Read block 3823 len 16384
37052 225/300 Write block 2239 len 12288 data: 3 3 5
37053 226/300 Read block 4957 len 32768
37054 227/300 Write block 2665 len 36864 data: 7 8 8 9 6 7 10 13 13
37055 228/300 Read block 909 len 40960
37056 229/300 Read block 4498 len 28672
37057 230/300 Write block 67 len 28672 data: 7 9 8 7 7 7 6
37058 231/300 Write block 1379 len 36864 data: 6 5 6 6 4 5 6 6 5
37059 232/300 Write block 4945 len 32768 data: 4 5 5 4 6 8 7 6
37060 233/300 Write block 4107 len 28672 data: 10 10 8 6 5 6 6
37061 234/300 Read block 4548 len 36864
37062 235/300 Flush
37063 236/300 Read block 2250 len 32768
37064 237/300 Write block 4 len 36864 data: 5 5 6 8 8 7 9 8 9
37065 238/300 Write block 1450 len 4096 data: 2
37066 239/300 Flush
37067 240/300 Read block 1521 len 4096
37068 241/300 Read block 679 len 36864
37069 242/300 Read block 2467 len 36864
37070 243/300 Write block 4124 len 20480 data: 4 4 4 5 7
37071 244/300 Write block 3903 len 32768 data: 6 6 4 2 3 2 2 2
37072 245/300 Read block 236 len 24576
37073 246/300 Read block 4103 len 20480
37074 247/300 Write block 2899 len 40960 data: 8 10 8 7 7 7 7 6 4 5
37075 248/300 Flush
37076 249/300 Read block 1485 len 16384
37077 250/300 Read block 2404 len 40960
37078 251/300 Write block 2466 len 16384 data: 4 4 3 5
37079 252/300 Write block 454 len 4096 data: 4
37080 253/300 Write block 2111 len 36864 data: 8 7 6 6 6 7 9 10 10
37081 254/300 Write block 706 len 36864 data: 6 7 7 6 6 5 5 5 7
37082 255/300 Write block 2277 len 8192 data: 7 9
37083 256/300 Write block 3478 len 12288 data: 3 3 3
37084 257/300 Read block 321 len 4096
37085 258/300 Write block 2665 len 36864 data: 8 9 9 10 7 8 11 14 14
37086 259/300 Read block 647 len 20480
37087 260/300 Write block 4263 len 16384 data: 7 7 7 9
37088 261/300 Read block 1255 len 4096
37089 262/300 Write block 4498 len 36864 data: 5 6 9 7 8 8 6 8 8
37090 263/300 Flush
37091 264/300 Write block 2830 len 36864 data: 9 6 5 7 7 8 7 7 9
37092 265/300 Read block 2953 len 12288
37093 266/300 Write block 3485 len 20480 data: 4 4 6 6 7
37094 267/300 Write block 234 len 40960 data: 8 8 8 9 8 8 8 6 4 5
37095 268/300 Write block 4344 len 36864 data: 7 6 6 5 5 6 4 3 4
37096 269/300 Read block 2067 len 32768
37097 270/300 Read block 203 len 12288
37098 271/300 Read block 2601 len 32768
37099 272/300 Write block 1133 len 20480 data: 4 6 5 6 4
37100 273/300 Read block 1511 len 4096
37101 274/300 Write block 4051 len 36864 data: 5 5 8 9 8 11 11 11 10
37102 275/300 Flush
37103 276/300 Write block 469 len 40960 data: 6 5 5 6 4 3 2 2 2 3
37104 277/300 Write block 3055 len 12288 data: 4 4 4
37105 278/300 Flush
37106 279/300 Flush
37107 280/300 Write block 156 len 16384 data: 5 6 7 5
37108 281/300 Read block 364 len 24576
37109 282/300 Write block 3170 len 40960 data: 11 10 8 10 10 8 6 6 6 5
37110 283/300 Write block 2793 len 16384 data: 6 4 3 5
37111 284/300 Read block 2019 len 4096
37112 285/300 Write block 1603 len 12288 data: 4 4 4
37113 286/300 Write block 161 len 28672 data: 6 5 5 5 7 7 7
37114 287/300 Write block 4716 len 40960 data: 5 5 6 6 7 8 8 9 10 7
37115 288/300 Read block 3418 len 32768
37116 289/300 Write block 937 len 16384 data: 5 5 4 6
37117 290/300 Flush
37118 291/300 Read block 402 len 40960
37119 292/300 Write block 4341 len 28672 data: 7 9 7 8 7 7 6
37120 293/300 Read block 1575 len 20480
37121 294/300 Write block 1982 len 40960 data: 7 8 8 9 11 9 7 10 9 8
37122 295/300 Write block 684 len 8192 data: 7 6
37123 296/300 Read block 2528 len 12288
37124 297/300 Read block 4150 len 32768
37125 298/300 Read block 4755 len 36864
37126 299/300 Flush
37127 300/300 Write block 2448 len 28672 data: 6 6 7 6 5 5 6
37128 301/300 Write block 3217 len 28672 data: 5 5 4 4 5 4 4
37129 Main task has work to do, go find it
37130 got dsc Start(1)
37131 start 1
37132 Mar 17 22:24:39.121 INFO request completed, latency_us: 270, response_code: 204, uri: /start/cid/1, method: POST, req_id: cbbb7153-6ffb-4810-85f0-fad2ce059c6e, remote_addr: 127.0.0.1:60423, local_addr: 127.0.0.1:9998
37133 Replay: started 1, returned:Ok(())
37134 ----------------------------------------------------------------
37135 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
37136 States: Active Offline Active
37137 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
37138 10056 Acked Read 9 Done Sent Done false
37139 10057 Acked Write 3 Done Sent Done false
37140 10058 Acked Write 3 Done Sent Done false
37141 10059 Acked Flush 0 Done Sent Done false
37142 10060 Acked Write 6 Done Sent Done false
37143 10061 Acked Write 10 Done Sent Done false
37144 10062 Acked Read 5 Done Sent Done false
37145 10063 Acked Flush 0 Done Sent Done false
37146 10064 Acked Read 6 Done Sent Done false
37147 10065 Acked Write 3 Done Sent Done false
37148 10066 Acked Read 1 Done Sent Done false
37149 10067 Acked Flush 0 Done Sent Done false
37150 10068 Acked Read 5 Done Sent Done false
37151 10069 Acked Write 8 Done Sent Done false
37152 10070 Acked Write 1 Done Sent Done false
37153 10071 Acked Read 8 Done Sent Done false
37154 10072 Acked Read 7 Done Sent Done false
37155 10073 Acked Read 5 Done Sent Done false
37156 10074 Acked Write 7 Done Sent Done false
37157 10075 Acked Write 6 Done Sent Done false
37158 10076 Acked Write 10 Done Sent Done false
37159 10077 Acked Write 2 Done Sent Done false
37160 10078 Acked Read 3 Done Sent Done false
37161 10079 Acked Read 4 Done Sent Done false
37162 10080 Acked Write 10 Done Sent Done false
37163 10081 Acked Read 5 Done Sent Done false
37164 10082 Acked Read 5 Done Sent Done false
37165 10083 Acked Read 5 Done Sent Done false
37166 10084 Acked Read 1 Done Sent Done false
37167 10085 Acked Write 1 Done Sent Done false
37168 10086 Acked Read 7 Done Sent Done false
37169 10087 Acked Write 10 Done Sent Done false
37170 10088 Acked Write 3 Done Sent Done false
37171 10089 Acked Read 6 Done Sent Done false
37172 10090 Acked Write 4 Done Sent Done false
37173 10091 Acked Write 4 Done Sent Done false
37174 10092 Acked Write 2 Done Sent Done false
37175 10093 Acked Read 3 Done Sent Done false
37176 10094 Acked Write 5 Done Sent Done false
37177 10095 Acked Read 6 Done Sent Done false
37178 10096 Acked Write 6 Done Sent Done false
37179 10097 Acked Write 8 Done Sent Done false
37180 10098 Acked Flush 0 Done Sent Done false
37181 10099 Acked Flush 0 Done Sent Done false
37182 10100 Acked Write 2 Done Sent Done false
37183 10101 Acked Read 5 Done Sent Done false
37184 10102 Acked Read 2 Done Sent Done false
37185 10103 Acked Read 10 Done Sent Done false
37186 10104 Acked Flush 0 Done Sent Done false
37187 10105 Acked Read 4 Done Sent Done false
37188 10106 Acked Write 4 Done Sent Done false
37189 10107 Acked Flush 0 Done Sent Done false
37190 10108 Acked Read 3 Done Sent Done false
37191 10109 Acked Flush 0 Done Sent Done false
37192 10110 Acked Read 5 Done Sent Done false
37193 10111 Acked Read 5 Done Sent Done false
37194 10112 Acked Write 2 Done Sent Done false
37195 10113 Acked Read 10 Done Sent Done false
37196 10114 Acked Read 5 Done Sent Done false
37197 10115 Acked Read 3 Done Sent Done false
37198 10116 Acked Read 7 Done Sent Done false
37199 10117 Acked Read 1 Done Sent Done false
37200 10118 Acked Write 4 Done Sent Done false
37201 10119 Acked Read 8 Done Sent Done false
37202 10120 Acked Write 9 Done Sent Done false
37203 10121 Acked Write 1 Done Sent Done false
37204 10122 Acked Read 2 Done Sent Done false
37205 10123 Acked Read 5 Done Sent Done false
37206 10124 Acked Read 9 Done Sent Done false
37207 10125 Acked Read 10 Done Sent Done false
37208 10126 Acked Write 1 Done Sent Done false
37209 10127 Acked Write 7 Done Sent Done false
37210 10128 Acked Read 2 Done Sent Done false
37211 10129 Acked Write 2 Done Sent Done false
37212 10130 Acked Read 10 Done Sent Done false
37213 10131 Acked Write 4 Done Sent Done false
37214 10132 Acked Write 4 Done Sent Done false
37215 10133 Acked Write 3 Done Sent Done false
37216 10134 Acked Write 7 Done Sent Done false
37217 10135 Acked Read 1 Done Sent Done false
37218 10136 Acked Write 9 Done Sent Done false
37219 10137 Acked Flush 0 Done Sent Done false
37220 10138 Acked Write 8 Done Sent Done false
37221 10139 Acked Flush 0 Done Sent Done false
37222 10140 Acked Read 2 Done Sent Done false
37223 10141 Acked Write 3 Done Sent Done false
37224 10142 Acked Write 8 Done Sent Done false
37225 10143 Acked Write 9 Done Sent Done false
37226 10144 Acked Write 9 Done Sent Done false
37227 10145 Acked Write 2 Done Sent Done false
37228 10146 Acked Read 5 Done Sent Done false
37229 10147 Acked Write 6 Done Sent Done false
37230 10148 Acked Write 10 Done Sent Done false
37231 10149 Acked Write 2 Done Sent Done false
37232 10150 Acked Read 2 Done Sent Done false
37233 10151 Acked Read 3 Done Sent Done false
37234 10152 Acked Read 10 Done Sent Done false
37235 10153 Acked Read 2 Done Sent Done false
37236 10154 Acked Read 9 Done Sent Done false
37237 10155 Acked Write 9 Done Sent Done false
37238 10156 Acked Read 2 Done Sent Done false
37239 10157 Acked Read 3 Done Sent Done false
37240 10158 Acked Write 7 Done Sent Done false
37241 10159 Acked Write 4 Done Sent Done false
37242 10160 Acked Write 8 Done Sent Done false
37243 10161 Acked Read 4 Done Sent Done false
37244 10162 Acked Write 6 Done Sent Done false
37245 10163 Acked Flush 0 Done Sent Done false
37246 10164 Acked Flush 0 Done Sent Done false
37247 10165 Acked Write 4 Done Sent Done false
37248 10166 Acked Read 4 Done Sent Done false
37249 10167 Acked Read 2 Done Sent Done false
37250 10168 Acked Read 9 Done Sent Done false
37251 10169 Acked Read 3 Done Sent Done false
37252 10170 Acked Read 1 Done Sent Done false
37253 10171 Acked Write 4 Done Sent Done false
37254 10172 Acked Read 2 Done Sent Done false
37255 10173 Acked Write 6 Done Sent Done false
37256 10174 Acked Read 4 Done Sent Done false
37257 10175 Acked Write 5 Done Sent Done false
37258 10176 Acked Write 4 Done Sent Done false
37259 10177 Acked Read 9 Done Sent Done false
37260 10178 Acked Read 10 Done Sent Done false
37261 10179 Acked Read 4 Done Sent Done false
37262 10180 Acked Write 6 Done Sent Done false
37263 10181 Acked Flush 0 Done Sent Done false
37264 10182 Acked Read 7 Done Sent Done false
37265 10183 Acked Write 5 Done Sent Done false
37266 10184 Acked Write 5 Done Sent Done false
37267 10185 Acked Write 6 Done Sent Done false
37268 10186 Acked Read 10 Done Sent Done false
37269 10187 Acked Flush 0 Done Sent Done false
37270 10188 Acked Read 5 Done Sent Done false
37271 10189 Acked Read 1 Done Sent Done false
37272 10190 Acked Read 8 Done Sent Done false
37273 10191 Acked Read 10 Done Sent Done false
37274 10192 Acked Write 3 Done Sent Done false
37275 10193 Acked Read 7 Done Sent Done false
37276 10194 Acked Flush 0 Done Sent Done false
37277 10195 Acked Flush 0 Done Sent Done false
37278 10196 Acked Write 9 Done Sent Done false
37279 10197 Acked Write 3 Done Sent Done false
37280 10198 Acked Read 5 Done Sent Done false
37281 10199 Acked Write 2 Done Sent Done false
37282 10200 Acked Read 4 Done Sent Done false
37283 10201 Acked Read 2 Done Sent Done false
37284 10202 Acked Write 8 Done Sent Done false
37285 10203 Acked Write 3 Done Sent Done false
37286 10204 Acked Read 5 Done Sent Done false
37287 10205 Acked Write 7 Done Sent Done false
37288 10206 Acked Write 9 Done Sent Done false
37289 10207 Acked Read 4 Done Sent Done false
37290 10208 Acked Write 6 Done Sent Done false
37291 10209 Acked Write 3 Done Sent Done false
37292 10210 Acked Read 4 Done Sent Done false
37293 10211 Acked Flush 0 Done Sent Done false
37294 10212 Acked Read 7 Done Sent Done false
37295 10213 Acked Read 9 Done Sent Done false
37296 10214 Acked Read 9 Done Sent Done false
37297 10215 Acked Write 7 Done Sent Done false
37298 10216 Acked Read 5 Done Sent Done false
37299 10217 Acked Read 2 Done Sent Done false
37300 10218 Acked Flush 0 Done Sent Done false
37301 10219 Acked Flush 0 Done Sent Done false
37302 10220 Acked Write 5 Done Sent Done false
37303 10221 Acked Read 9 Done Sent Done false
37304 10222 Acked Read 9 Done Sent Done false
37305 10223 Acked Read 1 Done Sent Done false
37306 10224 Acked Write 8 Done Sent Done false
37307 10225 Acked Read 9 Done Sent Done false
37308 10226 Acked Flush 0 Done Sent Done false
37309 10227 Acked Read 7 Done Sent Done false
37310 10228 Acked Read 4 Done Sent Done false
37311 10229 Acked Flush 0 Done Sent Done false
37312 10230 Acked Read 9 Done Sent Done false
37313 10231 Acked Flush 0 Done Sent Done false
37314 10232 Acked Write 1 Done Sent Done false
37315 10233 Acked Read 7 Done Sent Done false
37316 10234 Acked Read 10 Done Sent Done false
37317 10235 Acked Write 9 Done Sent Done false
37318 10236 Acked Read 9 Done Sent Done false
37319 10237 Acked Write 2 Done Sent Done false
37320 10238 Acked Flush 0 Done Sent Done false
37321 10239 Acked Read 6 Done Sent Done false
37322 10240 Acked Read 7 Done Sent Done false
37323 10241 Acked Read 3 Done Sent Done false
37324 10242 Acked Read 10 Done Sent Done false
37325 10243 Acked Read 2 Done Sent Done false
37326 10244 Acked Read 10 Done Sent Done false
37327 10245 Acked Write 6 Done Sent Done false
37328 10246 Acked Write 2 Done Sent Done false
37329 10247 Acked Read 8 Done Sent Done false
37330 10248 Acked Write 1 Done Sent Done false
37331 10249 Acked Read 1 Done Sent Done false
37332 10250 Acked Read 1 Done Sent Done false
37333 10251 Acked Write 5 Done Sent Done false
37334 10252 Acked Write 8 Done Sent Done false
37335 10253 Acked Read 1 Done Sent Done false
37336 10254 Acked Read 5 Done Sent Done false
37337 10255 Acked Read 10 Done Sent Done false
37338 10256 Acked Flush 0 Done Sent Done false
37339 10257 Acked Write 4 Done Sent Done false
37340 10258 Acked Write 7 Done Sent Done false
37341 10259 Acked Write 10 Done Sent Done false
37342 10260 Acked Read 2 Done Sent Done false
37343 10261 Acked Write 2 Done Sent Done false
37344 10262 Acked Read 1 Done Sent Done false
37345 10263 Acked Write 7 Done Sent Done false
37346 10264 Acked Read 1 Done Sent Done false
37347 10265 Acked Write 9 Done Sent Done false
37348 10266 Acked Flush 0 Done Sent Done false
37349 10267 Acked Read 9 Done Sent Done false
37350 10268 Acked Flush 0 Done Sent Done false
37351 10269 Acked Read 3 Done Sent Done false
37352 10270 Acked Read 6 Done Sent Done false
37353 10271 Acked Write 2 Done Sent Done false
37354 10272 Acked Read 8 Done Sent Done false
37355 10273 Acked Read 1 Done Sent Done false
37356 10274 Acked Write 7 Done Sent Done false
37357 10275 Acked Read 8 Done Sent Done false
37358 10276 Acked Write 9 Done Sent Done false
37359 10277 Acked Write 1 Done Sent Done false
37360 10278 Acked Write 6 Done Sent Done false
37361 10279 Acked Read 4 Done Sent Done false
37362 10280 Acked Write 3 Done Sent Done false
37363 10281 Acked Read 8 Done Sent Done false
37364 10282 Acked Write 9 Done Sent Done false
37365 10283 Acked Read 10 Done Sent Done false
37366 10284 Acked Read 7 Done Sent Done false
37367 10285 Acked Write 7 Done Sent Done false
37368 10286 Acked Write 9 Done Sent Done false
37369 10287 Acked Write 8 Done Sent Done false
37370 10288 Acked Write 7 Done Sent Done false
37371 10289 Acked Read 9 Done Sent Done false
37372 10290 Acked Flush 0 Done Sent Done false
37373 10291 Acked Read 8 Done Sent Done false
37374 10292 Acked Write 9 Done Sent Done false
37375 10293 Acked Write 1 Done Sent Done false
37376 10294 Acked Flush 0 Done Sent Done false
37377 10295 Acked Read 1 Done Sent Done false
37378 10296 Acked Read 9 Done Sent Done false
37379 10297 Acked Read 9 Done Sent Done false
37380 10298 Acked Write 5 Done Sent Done false
37381 10299 Acked Write 8 Done Sent Done false
37382 10300 Acked Read 6 Done Sent Done false
37383 10301 Acked Read 5 Done Sent Done false
37384 10302 Acked Write 10 Done Sent Done false
37385 10303 Acked Flush 0 Done Sent Done false
37386 10304 Acked Read 4 Done Sent Done false
37387 10305 Acked Read 10 Done Sent Done false
37388 10306 Acked Write 4 Done Sent Done false
37389 10307 Acked Write 1 Done Sent Done false
37390 10308 Acked Write 9 Done Sent Done false
37391 10309 Acked Write 9 Done Sent Done false
37392 10310 Acked Write 2 Done Sent Done false
37393 10311 Acked Write 3 Done Sent Done false
37394 10312 Acked Read 1 Done Sent Done false
37395 10313 Acked Write 9 Done Sent Done false
37396 10314 Acked Read 5 Done Sent Done false
37397 10315 Acked Write 4 Done Sent Done false
37398 10316 Acked Read 1 Done Sent Done false
37399 10317 Acked Write 9 Done Sent Done false
37400 10318 Acked Flush 0 Done Sent Done false
37401 10319 Acked Write 9 Done Sent Done false
37402 10320 Acked Read 3 Done Sent Done false
37403 10321 Acked Write 5 Done Sent Done false
37404 10322 Acked Write 10 Done Sent Done false
37405 10323 Acked Write 9 Done Sent Done false
37406 10324 Acked Read 8 Done Sent Done false
37407 10325 Acked Read 3 Done Sent Done false
37408 10326 Acked Read 8 Done Sent Done false
37409 10327 Acked Write 5 Done Sent Done false
37410 10328 Acked Read 1 Done Sent Done false
37411 10329 Acked Write 9 Done Sent Done false
37412 10330 Acked Flush 0 Done Sent Done false
37413 10331 Acked Write 10 Done Sent Done false
37414 10332 Acked Write 3 Done Sent Done false
37415 10333 Acked Flush 0 Done Sent Done false
37416 10334 Acked Flush 0 Done Sent Done false
37417 10335 Acked Write 4 Done Sent Done false
37418 10336 Acked Read 6 Done Sent Done false
37419 10337 Acked Write 10 Done Sent Done false
37420 10338 Acked Write 4 Done Sent Done false
37421 10339 Acked Read 1 Done Sent Done false
37422 10340 Acked Write 3 Done Sent Done false
37423 10341 Acked Write 7 Done Sent Done false
37424 10342 Acked Write 10 Done Sent Done false
37425 10343 Acked Read 8 Done Sent Done false
37426 10344 Acked Write 4 Done Sent Done false
37427 10345 Acked Flush 0 Done Sent Done false
37428 10346 Acked Read 10 Done Sent Done false
37429 10347 Acked Write 7 Done Sent Done false
37430 10348 Acked Read 5 Done Sent Done false
37431 10349 Acked Write 10 Done Sent Done false
37432 10350 Acked Write 2 Done Sent Done false
37433 10351 Acked Read 3 Done Sent Done false
37434 10352 Acked Read 8 Done Sent Done false
37435 10353 Acked Read 9 Done Sent Done false
37436 10354 Acked Flush 0 Done Sent Done false
37437 10355 Acked Write 7 Done Sent Done false
37438 10356 Acked Write 7 Done Sent Done false
37439 STATES DS:0 DS:1 DS:2 TOTAL
37440 Sent 0 301 0 301
37441 Done 301 0 301 602
37442 Skipped 0 0 0 0
37443 Error 0 0 0 0
37444 Last Flush: Some(JobId(10354)) Some(JobId(10055)) Some(JobId(10354))
37445 Downstairs last five completed: 10055 10054 10053 10052 10051
37446 Upstairs last five completed: 10356 10355 10354 10353 10352
37447 CLIENT: Up:0 ds:301 act:2
374482025-03-17T22:24:39.624ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
374492025-03-17T22:24:40.606ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
374502025-03-17T22:24:41.380ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
374512025-03-17T22:24:42.382ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
37452 [8820] got start action
37453 [8820] I am going to restart
37454 Starting downstairs at port 8820
37455 Make output file at "/tmp/dsc/downstairs-8820.txt"
37456 [8820][1] reports Starting
37457 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1237)
37458 [8820][1] reports Running
37459 ----------------------------------------------------------------
37460 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
37461 States: Active Offline Active
37462 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
37463 10056 Acked Read 9 Done Sent Done false
37464 10057 Acked Write 3 Done Sent Done false
37465 10058 Acked Write 3 Done Sent Done false
37466 10059 Acked Flush 0 Done Sent Done false
37467 10060 Acked Write 6 Done Sent Done false
37468 10061 Acked Write 10 Done Sent Done false
37469 10062 Acked Read 5 Done Sent Done false
37470 10063 Acked Flush 0 Done Sent Done false
37471 10064 Acked Read 6 Done Sent Done false
37472 10065 Acked Write 3 Done Sent Done false
37473 10066 Acked Read 1 Done Sent Done false
37474 10067 Acked Flush 0 Done Sent Done false
37475 10068 Acked Read 5 Done Sent Done false
37476 10069 Acked Write 8 Done Sent Done false
37477 10070 Acked Write 1 Done Sent Done false
37478 10071 Acked Read 8 Done Sent Done false
37479 10072 Acked Read 7 Done Sent Done false
37480 10073 Acked Read 5 Done Sent Done false
37481 10074 Acked Write 7 Done Sent Done false
37482 10075 Acked Write 6 Done Sent Done false
37483 10076 Acked Write 10 Done Sent Done false
37484 10077 Acked Write 2 Done Sent Done false
37485 10078 Acked Read 3 Done Sent Done false
37486 10079 Acked Read 4 Done Sent Done false
37487 10080 Acked Write 10 Done Sent Done false
37488 10081 Acked Read 5 Done Sent Done false
37489 10082 Acked Read 5 Done Sent Done false
37490 10083 Acked Read 5 Done Sent Done false
37491 10084 Acked Read 1 Done Sent Done false
37492 10085 Acked Write 1 Done Sent Done false
37493 10086 Acked Read 7 Done Sent Done false
37494 10087 Acked Write 10 Done Sent Done false
37495 10088 Acked Write 3 Done Sent Done false
37496 10089 Acked Read 6 Done Sent Done false
37497 10090 Acked Write 4 Done Sent Done false
37498 10091 Acked Write 4 Done Sent Done false
37499 10092 Acked Write 2 Done Sent Done false
37500 10093 Acked Read 3 Done Sent Done false
37501 10094 Acked Write 5 Done Sent Done false
37502 10095 Acked Read 6 Done Sent Done false
37503 10096 Acked Write 6 Done Sent Done false
37504 10097 Acked Write 8 Done Sent Done false
37505 10098 Acked Flush 0 Done Sent Done false
37506 10099 Acked Flush 0 Done Sent Done false
37507 10100 Acked Write 2 Done Sent Done false
37508 10101 Acked Read 5 Done Sent Done false
37509 10102 Acked Read 2 Done Sent Done false
37510 10103 Acked Read 10 Done Sent Done false
37511 10104 Acked Flush 0 Done Sent Done false
37512 10105 Acked Read 4 Done Sent Done false
37513 10106 Acked Write 4 Done Sent Done false
37514 10107 Acked Flush 0 Done Sent Done false
37515 10108 Acked Read 3 Done Sent Done false
37516 10109 Acked Flush 0 Done Sent Done false
37517 10110 Acked Read 5 Done Sent Done false
37518 10111 Acked Read 5 Done Sent Done false
37519 10112 Acked Write 2 Done Sent Done false
37520 10113 Acked Read 10 Done Sent Done false
37521 10114 Acked Read 5 Done Sent Done false
37522 10115 Acked Read 3 Done Sent Done false
37523 10116 Acked Read 7 Done Sent Done false
37524 10117 Acked Read 1 Done Sent Done false
37525 10118 Acked Write 4 Done Sent Done false
37526 10119 Acked Read 8 Done Sent Done false
37527 10120 Acked Write 9 Done Sent Done false
37528 10121 Acked Write 1 Done Sent Done false
37529 10122 Acked Read 2 Done Sent Done false
37530 10123 Acked Read 5 Done Sent Done false
37531 10124 Acked Read 9 Done Sent Done false
37532 10125 Acked Read 10 Done Sent Done false
37533 10126 Acked Write 1 Done Sent Done false
37534 10127 Acked Write 7 Done Sent Done false
37535 10128 Acked Read 2 Done Sent Done false
37536 10129 Acked Write 2 Done Sent Done false
37537 10130 Acked Read 10 Done Sent Done false
37538 10131 Acked Write 4 Done Sent Done false
37539 10132 Acked Write 4 Done Sent Done false
37540 10133 Acked Write 3 Done Sent Done false
37541 10134 Acked Write 7 Done Sent Done false
37542 10135 Acked Read 1 Done Sent Done false
37543 10136 Acked Write 9 Done Sent Done false
37544 10137 Acked Flush 0 Done Sent Done false
37545 10138 Acked Write 8 Done Sent Done false
37546 10139 Acked Flush 0 Done Sent Done false
37547 10140 Acked Read 2 Done Sent Done false
37548 10141 Acked Write 3 Done Sent Done false
37549 10142 Acked Write 8 Done Sent Done false
37550 10143 Acked Write 9 Done Sent Done false
37551 10144 Acked Write 9 Done Sent Done false
37552 10145 Acked Write 2 Done Sent Done false
37553 10146 Acked Read 5 Done Sent Done false
37554 10147 Acked Write 6 Done Sent Done false
37555 10148 Acked Write 10 Done Sent Done false
37556 10149 Acked Write 2 Done Sent Done false
37557 10150 Acked Read 2 Done Sent Done false
37558 10151 Acked Read 3 Done Sent Done false
37559 10152 Acked Read 10 Done Sent Done false
37560 10153 Acked Read 2 Done Sent Done false
37561 10154 Acked Read 9 Done Sent Done false
37562 10155 Acked Write 9 Done Sent Done false
37563 10156 Acked Read 2 Done Sent Done false
37564 10157 Acked Read 3 Done Sent Done false
37565 10158 Acked Write 7 Done Sent Done false
37566 10159 Acked Write 4 Done Sent Done false
37567 10160 Acked Write 8 Done Sent Done false
37568 10161 Acked Read 4 Done Sent Done false
37569 10162 Acked Write 6 Done Sent Done false
37570 10163 Acked Flush 0 Done Sent Done false
37571 10164 Acked Flush 0 Done Sent Done false
37572 10165 Acked Write 4 Done Sent Done false
37573 10166 Acked Read 4 Done Sent Done false
37574 10167 Acked Read 2 Done Sent Done false
37575 10168 Acked Read 9 Done Sent Done false
37576 10169 Acked Read 3 Done Sent Done false
37577 10170 Acked Read 1 Done Sent Done false
37578 10171 Acked Write 4 Done Sent Done false
37579 10172 Acked Read 2 Done Sent Done false
37580 10173 Acked Write 6 Done Sent Done false
37581 10174 Acked Read 4 Done Sent Done false
37582 10175 Acked Write 5 Done Sent Done false
37583 10176 Acked Write 4 Done Sent Done false
37584 10177 Acked Read 9 Done Sent Done false
37585 10178 Acked Read 10 Done Sent Done false
37586 10179 Acked Read 4 Done Sent Done false
37587 10180 Acked Write 6 Done Sent Done false
37588 10181 Acked Flush 0 Done Sent Done false
37589 10182 Acked Read 7 Done Sent Done false
37590 10183 Acked Write 5 Done Sent Done false
37591 10184 Acked Write 5 Done Sent Done false
37592 10185 Acked Write 6 Done Sent Done false
37593 10186 Acked Read 10 Done Sent Done false
37594 10187 Acked Flush 0 Done Sent Done false
37595 10188 Acked Read 5 Done Sent Done false
37596 10189 Acked Read 1 Done Sent Done false
37597 10190 Acked Read 8 Done Sent Done false
37598 10191 Acked Read 10 Done Sent Done false
37599 10192 Acked Write 3 Done Sent Done false
37600 10193 Acked Read 7 Done Sent Done false
37601 10194 Acked Flush 0 Done Sent Done false
37602 10195 Acked Flush 0 Done Sent Done false
37603 10196 Acked Write 9 Done Sent Done false
37604 10197 Acked Write 3 Done Sent Done false
37605 10198 Acked Read 5 Done Sent Done false
37606 10199 Acked Write 2 Done Sent Done false
37607 10200 Acked Read 4 Done Sent Done false
37608 10201 Acked Read 2 Done Sent Done false
37609 10202 Acked Write 8 Done Sent Done false
37610 10203 Acked Write 3 Done Sent Done false
37611 10204 Acked Read 5 Done Sent Done false
37612 10205 Acked Write 7 Done Sent Done false
37613 10206 Acked Write 9 Done Sent Done false
37614 10207 Acked Read 4 Done Sent Done false
37615 10208 Acked Write 6 Done Sent Done false
37616 10209 Acked Write 3 Done Sent Done false
37617 10210 Acked Read 4 Done Sent Done false
37618 10211 Acked Flush 0 Done Sent Done false
37619 10212 Acked Read 7 Done Sent Done false
37620 10213 Acked Read 9 Done Sent Done false
37621 10214 Acked Read 9 Done Sent Done false
37622 10215 Acked Write 7 Done Sent Done false
37623 10216 Acked Read 5 Done Sent Done false
37624 10217 Acked Read 2 Done Sent Done false
37625 10218 Acked Flush 0 Done Sent Done false
37626 10219 Acked Flush 0 Done Sent Done false
37627 10220 Acked Write 5 Done Sent Done false
37628 10221 Acked Read 9 Done Sent Done false
37629 10222 Acked Read 9 Done Sent Done false
37630 10223 Acked Read 1 Done Sent Done false
37631 10224 Acked Write 8 Done Sent Done false
37632 10225 Acked Read 9 Done Sent Done false
37633 10226 Acked Flush 0 Done Sent Done false
37634 10227 Acked Read 7 Done Sent Done false
37635 10228 Acked Read 4 Done Sent Done false
37636 10229 Acked Flush 0 Done Sent Done false
37637 10230 Acked Read 9 Done Sent Done false
37638 10231 Acked Flush 0 Done Sent Done false
37639 10232 Acked Write 1 Done Sent Done false
37640 10233 Acked Read 7 Done Sent Done false
37641 10234 Acked Read 10 Done Sent Done false
37642 10235 Acked Write 9 Done Sent Done false
37643 10236 Acked Read 9 Done Sent Done false
37644 10237 Acked Write 2 Done Sent Done false
37645 10238 Acked Flush 0 Done Sent Done false
37646 10239 Acked Read 6 Done Sent Done false
37647 10240 Acked Read 7 Done Sent Done false
37648 10241 Acked Read 3 Done Sent Done false
37649 10242 Acked Read 10 Done Sent Done false
37650 10243 Acked Read 2 Done Sent Done false
37651 10244 Acked Read 10 Done Sent Done false
37652 10245 Acked Write 6 Done Sent Done false
37653 10246 Acked Write 2 Done Sent Done false
37654 10247 Acked Read 8 Done Sent Done false
37655 10248 Acked Write 1 Done Sent Done false
37656 10249 Acked Read 1 Done Sent Done false
37657 10250 Acked Read 1 Done Sent Done false
37658 10251 Acked Write 5 Done Sent Done false
37659 10252 Acked Write 8 Done Sent Done false
37660 10253 Acked Read 1 Done Sent Done false
37661 10254 Acked Read 5 Done Sent Done false
37662 10255 Acked Read 10 Done Sent Done false
37663 10256 Acked Flush 0 Done Sent Done false
37664 10257 Acked Write 4 Done Sent Done false
37665 10258 Acked Write 7 Done Sent Done false
37666 10259 Acked Write 10 Done Sent Done false
37667 10260 Acked Read 2 Done Sent Done false
37668 10261 Acked Write 2 Done Sent Done false
37669 10262 Acked Read 1 Done Sent Done false
37670 10263 Acked Write 7 Done Sent Done false
37671 10264 Acked Read 1 Done Sent Done false
37672 10265 Acked Write 9 Done Sent Done false
37673 10266 Acked Flush 0 Done Sent Done false
37674 10267 Acked Read 9 Done Sent Done false
37675 10268 Acked Flush 0 Done Sent Done false
37676 10269 Acked Read 3 Done Sent Done false
37677 10270 Acked Read 6 Done Sent Done false
37678 10271 Acked Write 2 Done Sent Done false
37679 10272 Acked Read 8 Done Sent Done false
37680 10273 Acked Read 1 Done Sent Done false
37681 10274 Acked Write 7 Done Sent Done false
37682 10275 Acked Read 8 Done Sent Done false
37683 10276 Acked Write 9 Done Sent Done false
37684 10277 Acked Write 1 Done Sent Done false
37685 10278 Acked Write 6 Done Sent Done false
37686 10279 Acked Read 4 Done Sent Done false
37687 10280 Acked Write 3 Done Sent Done false
37688 10281 Acked Read 8 Done Sent Done false
37689 10282 Acked Write 9 Done Sent Done false
37690 10283 Acked Read 10 Done Sent Done false
37691 10284 Acked Read 7 Done Sent Done false
37692 10285 Acked Write 7 Done Sent Done false
37693 10286 Acked Write 9 Done Sent Done false
37694 10287 Acked Write 8 Done Sent Done false
37695 10288 Acked Write 7 Done Sent Done false
37696 10289 Acked Read 9 Done Sent Done false
37697 10290 Acked Flush 0 Done Sent Done false
37698 10291 Acked Read 8 Done Sent Done false
37699 10292 Acked Write 9 Done Sent Done false
37700 10293 Acked Write 1 Done Sent Done false
37701 10294 Acked Flush 0 Done Sent Done false
37702 10295 Acked Read 1 Done Sent Done false
37703 10296 Acked Read 9 Done Sent Done false
37704 10297 Acked Read 9 Done Sent Done false
37705 10298 Acked Write 5 Done Sent Done false
37706 10299 Acked Write 8 Done Sent Done false
37707 10300 Acked Read 6 Done Sent Done false
37708 10301 Acked Read 5 Done Sent Done false
37709 10302 Acked Write 10 Done Sent Done false
37710 10303 Acked Flush 0 Done Sent Done false
37711 10304 Acked Read 4 Done Sent Done false
37712 10305 Acked Read 10 Done Sent Done false
37713 10306 Acked Write 4 Done Sent Done false
37714 10307 Acked Write 1 Done Sent Done false
37715 10308 Acked Write 9 Done Sent Done false
37716 10309 Acked Write 9 Done Sent Done false
37717 10310 Acked Write 2 Done Sent Done false
37718 10311 Acked Write 3 Done Sent Done false
37719 10312 Acked Read 1 Done Sent Done false
37720 10313 Acked Write 9 Done Sent Done false
37721 10314 Acked Read 5 Done Sent Done false
37722 10315 Acked Write 4 Done Sent Done false
37723 10316 Acked Read 1 Done Sent Done false
37724 10317 Acked Write 9 Done Sent Done false
37725 10318 Acked Flush 0 Done Sent Done false
37726 10319 Acked Write 9 Done Sent Done false
37727 10320 Acked Read 3 Done Sent Done false
37728 10321 Acked Write 5 Done Sent Done false
37729 10322 Acked Write 10 Done Sent Done false
37730 10323 Acked Write 9 Done Sent Done false
37731 10324 Acked Read 8 Done Sent Done false
37732 10325 Acked Read 3 Done Sent Done false
37733 10326 Acked Read 8 Done Sent Done false
37734 10327 Acked Write 5 Done Sent Done false
37735 10328 Acked Read 1 Done Sent Done false
37736 10329 Acked Write 9 Done Sent Done false
37737 10330 Acked Flush 0 Done Sent Done false
37738 10331 Acked Write 10 Done Sent Done false
37739 10332 Acked Write 3 Done Sent Done false
37740 10333 Acked Flush 0 Done Sent Done false
37741 10334 Acked Flush 0 Done Sent Done false
37742 10335 Acked Write 4 Done Sent Done false
37743 10336 Acked Read 6 Done Sent Done false
37744 10337 Acked Write 10 Done Sent Done false
37745 10338 Acked Write 4 Done Sent Done false
37746 10339 Acked Read 1 Done Sent Done false
37747 10340 Acked Write 3 Done Sent Done false
37748 10341 Acked Write 7 Done Sent Done false
37749 10342 Acked Write 10 Done Sent Done false
37750 10343 Acked Read 8 Done Sent Done false
37751 10344 Acked Write 4 Done Sent Done false
37752 10345 Acked Flush 0 Done Sent Done false
37753 10346 Acked Read 10 Done Sent Done false
37754 10347 Acked Write 7 Done Sent Done false
37755 10348 Acked Read 5 Done Sent Done false
37756 10349 Acked Write 10 Done Sent Done false
37757 10350 Acked Write 2 Done Sent Done false
37758 10351 Acked Read 3 Done Sent Done false
37759 10352 Acked Read 8 Done Sent Done false
37760 10353 Acked Read 9 Done Sent Done false
37761 10354 Acked Flush 0 Done Sent Done false
37762 10355 Acked Write 7 Done Sent Done false
37763 10356 Acked Write 7 Done Sent Done false
37764 STATES DS:0 DS:1 DS:2 TOTAL
37765 Sent 0 301 0 301
37766 Done 301 0 301 602
37767 Skipped 0 0 0 0
37768 Error 0 0 0 0
37769 Last Flush: Some(JobId(10354)) Some(JobId(10055)) Some(JobId(10354))
37770 Downstairs last five completed: 10055 10054 10053 10052 10051
37771 Upstairs last five completed: 10356 10355 10354 10353 10352
37772 CLIENT: Up:0 ds:301 act:2
377732025-03-17T22:24:43.626ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
377742025-03-17T22:24:44.384ZWARNcrucible: flush check fired despite having jobs; resetting it session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
377752025-03-17T22:24:44.879ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
377762025-03-17T22:24:44.879ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:47182) = io task client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
377772025-03-17T22:24:44.880ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: Offline } = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
377782025-03-17T22:24:44.881ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
377792025-03-17T22:24:44.881ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
377802025-03-17T22:24:44.881ZINFOcrucible: send last flush ID to this DS: Some(JobId(10055)) = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
377812025-03-17T22:24:44.881ZINFOcrucible: Transition from Offline to Active = downstairs client = 1 session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
377822025-03-17T22:24:44.881ZINFOcrucible: [1] client re-new 301 jobs since flush Some(JobId(10055)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
377832025-03-17T22:24:44.881ZINFOcrucible: [1] Replayed 301 jobs since flush: Some(JobId(10055)) = downstairs session_id = 0f8310d7-099e-4ead-8484-91ddc162d307
37784 ----------------------------------------------------------------
37785 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
37786 Downstairs last five completed: 10357 10356 10355 10354 10353
37787 Upstairs last five completed: 10357 10356 10355 10354 10353
37788 CLIENT: Up:0 ds:0 act:3
37789 Replay: All jobs finished
37790 Test replay has completed
37791 Wrote out file "/tmp/test_replay_verify.log"
37792 CLIENT: Tests done. All submitted work has been ACK'd
37793 ----------------------------------------------------------------
37794 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
37795 Downstairs last five completed: 10357 10356 10355 10354 10353
37796 Upstairs last five completed: 10357 10356 10355 10354 10353
37797 CLIENT: Up:0 ds:0 act:3
37798 CLIENT: All jobs finished, all DS active.
37799 Replays:30 time: 8:28
37800 Do final verify
37801 Mar 17 22:24:47.204 INFO accepted connection, remote_addr: 127.0.0.1:64058, local_addr: 127.0.0.1:9998
37802 Mar 17 22:24:47.205 INFO request completed, latency_us: 221, response_code: 200, uri: /regioninfo, method: GET, req_id: a04253fd-7319-4d7d-9cd2-1aeda0ad0b36, remote_addr: 127.0.0.1:64058, local_addr: 127.0.0.1:9998
378032025-03-17T22:24:47.206ZINFOcrucible: Use this region info from dsc: RegionExtentInfo { block_size: 4096, blocks_per_extent: 100, extent_count: 50 } task = crutest
37804 Mar 17 22:24:47.206 INFO request completed, latency_us: 100, response_code: 200, uri: /regioncount, method: GET, req_id: 27def632-7f84-4e38-a6e4-19218690a08a, remote_addr: 127.0.0.1:64058, local_addr: 127.0.0.1:9998
378052025-03-17T22:24:47.206ZINFOcrucible: dsc has 3 regions. This means 1 sub_volumes task = crutest
37806 Mar 17 22:24:47.207 INFO request completed, latency_us: 115, response_code: 200, uri: /port/cid/0, method: GET, req_id: 9b59e0d0-5ded-4c19-b340-caea3131babd, remote_addr: 127.0.0.1:64058, local_addr: 127.0.0.1:9998
37807 Mar 17 22:24:47.207 INFO request completed, latency_us: 105, response_code: 200, uri: /port/cid/1, method: GET, req_id: 824a4a40-3a9b-4808-9fd3-b35c419e1839, remote_addr: 127.0.0.1:64058, local_addr: 127.0.0.1:9998
37808 Mar 17 22:24:47.208 INFO request completed, latency_us: 103, response_code: 200, uri: /port/cid/2, method: GET, req_id: a1aaaaae-5e34-4463-b0d5-7700d98792f2, remote_addr: 127.0.0.1:64058, local_addr: 127.0.0.1:9998
378092025-03-17T22:24:47.208ZINFOcrucible: SV 0 has targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830] task = crutest
378102025-03-17T22:24:47.208ZINFOcrucible: Upstairs starts
378112025-03-17T22:24:47.208ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "f72bfdb36aa576722b377a737f26a9b5ef54faf2", git_commit_timestamp: "2025-03-17T21:58:36.000000000Z", git_branch: "renovate/tempfile-3.x-lockfile", rustc_semver: "1.84.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "9fc6b43126469e3858e2fe86cafb4f0fd5068869", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
378122025-03-17T22:24:47.208ZINFOcrucible: Upstairs <-> Downstairs Message Version: 13
378132025-03-17T22:24:47.208ZINFOcrucible: Using region definition RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 00000000-0000-0000-0000-000000000000, encrypted: false, database_read_version: 1, database_write_version: 1 }
378142025-03-17T22:24:47.209ZINFOcrucible: Crucible 8c6bdb29-cf61-4004-9370-5dc7a50ca2e4 has session id: 7da50dbc-5340-4767-b27f-30dfb7576759 session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378152025-03-17T22:24:47.209ZINFOcrucible: Upstairs opts: Upstairs UUID: 8c6bdb29-cf61-4004-9370-5dc7a50ca2e4, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378162025-03-17T22:24:47.209ZINFOcrucible: Crucible stats registered with UUID: 8c6bdb29-cf61-4004-9370-5dc7a50ca2e4 session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378172025-03-17T22:24:47.209ZWARNcrucible: could not find Downstairs address for Nexus session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378182025-03-17T22:24:47.210ZINFOcrucible: Downstairs in volume = 3 task = crutest
378192025-03-17T22:24:47.210ZINFOcrucible: The guest has requested activation with gen:3
378202025-03-17T22:24:47.210ZINFOcrucible: 8c6bdb29-cf61-4004-9370-5dc7a50ca2e4 active request set session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378212025-03-17T22:24:47.210ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378222025-03-17T22:24:47.210ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 0 session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378232025-03-17T22:24:47.210ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378242025-03-17T22:24:47.210ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 1 session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378252025-03-17T22:24:47.210ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378262025-03-17T22:24:47.210ZINFOcrucible: client set_active_request while in Connecting { state: Start { auto_promote: true }, mode: New }; waiting... = downstairs client = 2 session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378272025-03-17T22:24:47.210ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378282025-03-17T22:24:47.210ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378292025-03-17T22:24:47.210ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378302025-03-17T22:24:47.210ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:34952) = io task client = 0 session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378312025-03-17T22:24:47.210ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:62485) = io task client = 2 session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378322025-03-17T22:24:47.210ZINFOcrucible: ds_connection connected from Ok(127.0.0.1:49562) = io task client = 1 session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378332025-03-17T22:24:47.211ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 2 session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378342025-03-17T22:24:47.211ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 0 session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378352025-03-17T22:24:47.211ZINFOcrucible: version negotiation from state Connecting { state: WaitForPromote, mode: New } = downstairs client = 1 session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378362025-03-17T22:24:47.212ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378372025-03-17T22:24:47.212ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378382025-03-17T22:24:47.212ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378392025-03-17T22:24:47.213ZINFOcrucible: Waiting for more clients to be ready session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378402025-03-17T22:24:47.213ZINFOcrucible: Waiting for more clients to be ready session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378412025-03-17T22:24:47.213ZINFOcrucible: [0]R flush_numbers[0..12]: [941, 947, 944, 930, 945, 933, 948, 943, 924, 947, 940, 944] = downstairs session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378422025-03-17T22:24:47.213ZINFOcrucible: [0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378432025-03-17T22:24:47.213ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378442025-03-17T22:24:47.213ZINFOcrucible: [1]R flush_numbers[0..12]: [941, 947, 944, 930, 945, 933, 948, 943, 924, 947, 940, 944] = downstairs session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378452025-03-17T22:24:47.213ZINFOcrucible: [1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378462025-03-17T22:24:47.213ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378472025-03-17T22:24:47.213ZINFOcrucible: [2]R flush_numbers[0..12]: [941, 947, 944, 930, 945, 933, 948, 943, 924, 947, 940, 944] = downstairs session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378482025-03-17T22:24:47.213ZINFOcrucible: [2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378492025-03-17T22:24:47.213ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378502025-03-17T22:24:47.213ZINFOcrucible: Max found gen is 3 = downstairs session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378512025-03-17T22:24:47.213ZINFOcrucible: Generation requested: 3 >= found:3 = downstairs session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378522025-03-17T22:24:47.213ZINFOcrucible: Next flush: 950 = downstairs session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378532025-03-17T22:24:47.213ZINFOcrucible: All extents match = downstairs session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378542025-03-17T22:24:47.213ZINFOcrucible: No downstairs reconciliation required session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
37855 {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2025-03-17T22:24:47.213642161ZWait for a query_work_queue command to finish before sending IO
37856 ","hostname":"w-01JPK2S810SFAA9787XEG8GFX5","pid":1242,"client":"0","":"downstairs","session_id":"7da50dbc-5340-4767-b27f-30dfb7576759"}
378572025-03-17T22:24:47.213ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378582025-03-17T22:24:47.213ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378592025-03-17T22:24:47.213ZINFOcrucible: All required reconciliation work is completed session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378602025-03-17T22:24:47.213ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378612025-03-17T22:24:47.213ZINFOcrucible: 8c6bdb29-cf61-4004-9370-5dc7a50ca2e4 is now active with session: 7da50dbc-5340-4767-b27f-30dfb7576759 session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378622025-03-17T22:24:47.213ZINFOcrucible: Set Active after no reconciliation session_id = 7da50dbc-5340-4767-b27f-30dfb7576759
378632025-03-17T22:24:47.213ZINFOcrucible: The guest has finished waiting for activation with:3
378642025-03-17T22:24:47.213ZINFOcrucible: Activated sub_volume 8c6bdb29-cf61-4004-9370-5dc7a50ca2e4
37865 Disk: sv:1 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
37866 Loading write count information from file "/tmp/test_replay_verify.log"
37867 Read and Verify all blocks (0..5000 range:false)
37868 Wrote out file "/tmp/test_replay_verify.log"
37869 Verify test completed
37870 Wrote out file "/tmp/test_replay_verify.log"
37871 CLIENT: Tests done. All submitted work has been ACK'd
37872 ----------------------------------------------------------------
37873 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:20
37874 States: Active Active Active
37875 JOBID ACK TYPE BKS/EXT DS:0 DS:1 DS:2 REPLAY
37876 1000 Acked Read 256 Done Done Done false
37877 1001 Acked Read 256 Done Done Done false
37878 1002 Acked Read 256 Done Done Done false
37879 1003 Acked Read 256 Done Done Done false
37880 1004 Acked Read 256 Done Done Done false
37881 1005 Acked Read 256 Done Done Done false
37882 1006 Acked Read 256 Done Done Done false
37883 1007 Acked Read 256 Done Done Done false
37884 1008 Acked Read 256 Done Done Done false
37885 1009 Acked Read 256 Done Done Done false
37886 1010 Acked Read 256 Done Done Done false
37887 1011 Acked Read 256 Done Done Done false
37888 1012 Acked Read 256 Done Done Done false
37889 1013 Acked Read 256 Done Done Done false
37890 1014 Acked Read 256 Done Done Done false
37891 1015 Acked Read 256 Done Done Done false
37892 1016 Acked Read 256 Done Done Done false
37893 1017 Acked Read 256 Done Done Done false
37894 1018 Acked Read 256 Done Done Done false
37895 1019 Acked Read 136 Done Done Done false
37896 STATES DS:0 DS:1 DS:2 TOTAL
37897 Sent 0 0 0 0
37898 Done 20 20 20 60
37899 Skipped 0 0 0 0
37900 Error 0 0 0 0
37901 Last Flush: None None None
37902 Downstairs last five completed:
37903 Upstairs last five completed: 1019 1018 1017 1016 1015
37904 CLIENT: Up:0 ds:20 act:3
37905 ----------------------------------------------------------------
37906 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0
37907 Downstairs last five completed: 1020 1019 1018 1017 1016
37908 Upstairs last five completed: 1020 1019 1018 1017 1016
37909 CLIENT: Up:0 ds:0 act:3
37910 CLIENT: All crucible jobs finished, exiting program
37911 Mar 17 22:24:51.681 INFO accepted connection, remote_addr: 127.0.0.1:50573, local_addr: 127.0.0.1:9998
37912 Main task has work to do, go find it
37913 got dsc Shutdown
37914 Shutdown
37915 Shut it down
37916 Mar 17 22:24:51.682 INFO request completed, latency_us: 189, response_code: 204, uri: /shutdown, method: POST, req_id: a77e8ee6-1db0-49a2-b8f6-f4a3a933c826, remote_addr: 127.0.0.1:50573, local_addr: 127.0.0.1:9998[8820] Disable keep_running
37917 
37918 [8820] Got stop action so:false kr:false
37919 [8810] Disable keep_running
37920 [8810] Got stop action so:false kr:false
37921 [8830] Disable keep_running
37922 [8830] Got stop action so:false kr:false
37923 [8820] Exited with: ExitStatus(unix_wait_status(9))
37924 [8810] Exited with: ExitStatus(unix_wait_status(9))
37925 [8820][1] reports Exit
37926 [8830] Exited with: ExitStatus(unix_wait_status(9))
37927 [8810][0] reports Exit
37928 [8830][2] reports Exit
37929 March 17, 2025 at 10:24:56 PM UTC Test ends with 0